Karman580 opened a new pull request, #19209:
URL: https://github.com/apache/iotdb/pull/19209

   Fixes #19208
   
   ## Summary
   
   When `import-data` infers the type of an untyped CSV column (no type in the 
header, and the series or column does not exist yet), two things go wrong with 
numeric values:
   
   - Some values make inference throw `NumberFormatException`, which aborts the 
file.
   - Other values are stored silently with a different value than the CSV 
contains.
   
   This PR fixes both in `AbstractDataTool.typeInfer`, which serves the tree 
and table import paths. The CLI key mapping (`int` -> FLOAT, `long` -> DOUBLE, 
`float` -> FLOAT, `double` -> DOUBLE) and `-ti` overrides are unchanged.
   
   ## Root cause
   
   - Every numeric value without a `.` went to `isConvertFloatPrecisionLack`, 
which calls `Long.parseLong`. That throws for `1e5`, `1E10`, `Infinity`, 
`-Infinity`, `12f`, `3d` and `99999999999999999999`, all of which `isNumber` 
accepts. The exception reached `AbstractImportData.run()`, so the file failed, 
rows already flushed in earlier batches stayed written, and no `.failed` file 
was produced.
   - The bound was `Long.parseLong(s) > (2 << 24)`, which is `2^25` instead of 
`2^24`, and it is one-sided. As a result `16777217` was stored as `16777216`, 
and every negative integer became FLOAT: `-16777217` was stored as `-16777216` 
and `-123456789` as `-123456792`. With `-ti int=int`, `3000000001` became 
DOUBLE but `-3000000001` became INT32 and was rejected.
   - Decimals always took the `float` key, so `1.0e39` was stored as `Infinity` 
and `1.0e-50` as `0.0`.
   
   ## Changes
   
   - **Plain integers (`[+-]?\d+`):** take the `int` key only when `-2^24 <= v 
<= 2^24`. Every other integer, including those outside the long range, takes 
the `long` key. The bound is on magnitude rather than per value, because the 
inferred type is used for every later value in the column: `16777218` is exact 
as FLOAT but `16777219` is not.
   - **Other numerics** (decimals, exponents, `Infinity`, hex floats, and 
`f`/`d` suffixes, which `1.5f` and `1.5d` already imported with as FLOAT): take 
the `float` key. They take the `double` key instead when the value is finite 
but outside FLOAT's normal range.
   - Removed `isConvertFloatPrecisionLack` and the now-unused `TsFileConstant` 
import.
   
   Before and after, default mapping, tree model (`Time,root.fix.d3.s2,...`):
   
   | CSV value | Before | After |
   |---|---|---|
   | `1e5` | file aborted: `Unexpected error occurred: For input string: 
"1e5"`, exit status 1 | FLOAT `100000.0`, exit status 0 |
   | `Infinity`, `12f` | file aborted | FLOAT `Infinity`, `12.0` |
   | `16777217` | FLOAT, stored `16777216` | DOUBLE `1.6777217E7` |
   | `-16777217` | FLOAT, stored `-16777216` | DOUBLE `-1.6777217E7` |
   | `-123456789` | FLOAT, stored `-123456792` | DOUBLE `-1.23456789E8` |
   | `33554431` | FLOAT, stored `33554432` | DOUBLE `3.3554431E7` |
   | `1.0e39` | FLOAT, stored `Infinity` | DOUBLE `1.0E39` |
   | `-3000000001` with `-ti int=int` | INT32, line rejected | DOUBLE 
`-3.000000001E9` |
   
   All "After" values come from real imports. The "Before" values for `12f`, 
`33554431` and `1.0e39` come from calling `typeInfer` and `typeTrans` directly 
on the old code; the other "Before" values come from real imports.
   
   ## Regression tests
   
   Added to `DataToolRegressionTest`:
   
   - `testTypeInferKeepsIntegersExact`: inferred type and stored value around 
`±2^24` and `±2^25`. Covers `16777218` and `33554432` (exact, but in a range 
where neighbours are not), `-123456789`, `1099511627776`, the long bounds and 
an integer outside the long range. Also checks that the stored value is exactly 
the CSV value wherever the type can hold it.
   - `testTypeInferAcceptsEveryNumberThatIsNumberAccepts`: exponents (both 
cases), infinities, `NaN`, `-NaN`, suffixes, hex, the FLOAT range limits, 
values beyond DOUBLE, and non-numbers.
   - `testTypeInferWithInt32MappingTreatsBothSignsAlike`: `-ti int=int`. The 
shared mapping is restored in `finally`.
   - `testTreeCsvImportInfersNumbersWithoutLosingRows` and 
`testTableCsvImportInfersNumbersWithoutLosingRows`: full imports through mocked 
sessions with `batchPointSize = 2` (more than one batch). They mix values in 
the same column, include valid rows after `1e5` and `Infinity`, and include an 
explicitly typed `INT64` column. They check the inferred types, every stored 
value, success status, and that no failed file was written.
   
   Without the change to `AbstractDataTool`, all five new tests fail and the 39 
existing ones pass:
   
   ```
   [ERROR] Tests run: 44, Failures: 4, Errors: 1, Skipped: 0 <<< FAILURE! -- in 
org.apache.iotdb.tool.data.DataToolRegressionTest
   [ERROR]   
DataToolRegressionTest.testTypeInferKeepsIntegersExact:957->assertInferred:1349 
16777217 expected:<DOUBLE> but was:<FLOAT>
   [ERROR]   
DataToolRegressionTest.testTypeInferWithInt32MappingTreatsBothSignsAlike:1011->assertInferred:1349
 -3000000001 expected:<DOUBLE> but was:<INT32>
   java.lang.NumberFormatException: For input string: "12f"
   ```
   
   ## Test commands and results
   
   JDK 25 (google-java-format in Spotless fails on JDK 27 with 
`NoSuchFieldError` in `JCCompilationUnit`, so formatting was run on 25).
   
   ```
   $ ./mvnw test -pl iotdb-client/cli -am -Dtest=DataToolRegressionTest 
-Dsurefire.failIfNoSpecifiedTests=false
   [INFO] Tests run: 44, Failures: 0, Errors: 0, Skipped: 0 -- in 
org.apache.iotdb.tool.data.DataToolRegressionTest
   [INFO] BUILD SUCCESS
   
   $ ./mvnw clean test -pl iotdb-client/cli
   [INFO] Tests run: 85, Failures: 0, Errors: 0, Skipped: 0
   [INFO] BUILD SUCCESS          (checkstyle and spotless:check ran in this 
build)
   
   $ ./mvnw spotless:check -pl iotdb-client/cli
   [INFO] BUILD SUCCESS
   
   $ ./mvnw test-compile -pl iotdb-client/cli -P with-zh-locale 
-Dspotless.skip=true
   exit 0
   ```
   
   I also imported CSVs with the CLI rebuilt from this branch into a local 
standalone server (built at `cd31721c14`; nothing under `iotdb-client/cli` 
changed between that commit and this branch's base). All tree imports and three 
of the four table imports finished with `Import completely!` and exit status 0. 
The fourth (`time,dev,s1` with `1e5`) failed with the separate table issue 
listed under known limitations; the same file with the column named `s3` 
imported correctly. The stored values match the "After" column above, for 
example:
   
   ```
   $ select s1,s9 from root.fix.d2          (-batch 2; s9 = 1e5, Infinity, 12f 
on rows 5-7)
   |1970-01-01T05:30:00.005+05:30|          11.0|      100000.0|
   |1970-01-01T05:30:00.006+05:30|          12.0|      Infinity|
   |1970-01-01T05:30:00.007+05:30|          13.0|          12.0|
   Total line number = 7
   
   $ select * from f4                        (table model, time,dev,s3 with 1e5 
then 7)
   |1970-01-01T05:30:00.001+05:30|  a|100000.0|
   |1970-01-01T05:30:00.002+05:30|  a|     7.0|
   Total line number = 2
   ```
   
   `ImportDataTestIT` only checks the missing-source-directory message and does 
not reach type inference, so I did not run it for this change.
   
   ## Behaviour changes and known limitations
   
   - New columns get DOUBLE instead of FLOAT for integers with `2^24 < |v| <= 
2^25`, including the even ones that happen to be exact, and for all negative 
integers below `-2^24`. Existing series and columns are not affected, since 
their type is queried rather than inferred.
   - `12f` and `3d` stay numeric (FLOAT), as `1.5f` and `1.5d` already were. 
Server-side inference treats suffixed values as text. I kept the CLI behaviour 
so inputs that import today do not change type.
   - Values beyond DOUBLE (`1e309`, `1e-400`) are stored as `Infinity` and 
`0.0`. They cannot be kept by either type.
   - Separate from this change: the table import path can lose a field value, 
or fail with `Missing or invalid column categories for table insertion`, when a 
new column is inferred in the middle of a file. The outcome depends on header 
iteration order. I reproduced both with plain integers on the unmodified CLI, 
so this PR does not change that behaviour, and I'd rather handle it separately.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to