Karman580 opened a new pull request, #19209: URL: https://github.com/apache/iotdb/pull/19209
Fixes #19208 ## Summary When `import-data` infers the type of an untyped CSV column (no type in the header, and the series or column does not exist yet), two things go wrong with numeric values: - Some values make inference throw `NumberFormatException`, which aborts the file. - Other values are stored silently with a different value than the CSV contains. This PR fixes both in `AbstractDataTool.typeInfer`, which serves the tree and table import paths. The CLI key mapping (`int` -> FLOAT, `long` -> DOUBLE, `float` -> FLOAT, `double` -> DOUBLE) and `-ti` overrides are unchanged. ## Root cause - Every numeric value without a `.` went to `isConvertFloatPrecisionLack`, which calls `Long.parseLong`. That throws for `1e5`, `1E10`, `Infinity`, `-Infinity`, `12f`, `3d` and `99999999999999999999`, all of which `isNumber` accepts. The exception reached `AbstractImportData.run()`, so the file failed, rows already flushed in earlier batches stayed written, and no `.failed` file was produced. - The bound was `Long.parseLong(s) > (2 << 24)`, which is `2^25` instead of `2^24`, and it is one-sided. As a result `16777217` was stored as `16777216`, and every negative integer became FLOAT: `-16777217` was stored as `-16777216` and `-123456789` as `-123456792`. With `-ti int=int`, `3000000001` became DOUBLE but `-3000000001` became INT32 and was rejected. - Decimals always took the `float` key, so `1.0e39` was stored as `Infinity` and `1.0e-50` as `0.0`. ## Changes - **Plain integers (`[+-]?\d+`):** take the `int` key only when `-2^24 <= v <= 2^24`. Every other integer, including those outside the long range, takes the `long` key. The bound is on magnitude rather than per value, because the inferred type is used for every later value in the column: `16777218` is exact as FLOAT but `16777219` is not. - **Other numerics** (decimals, exponents, `Infinity`, hex floats, and `f`/`d` suffixes, which `1.5f` and `1.5d` already imported with as FLOAT): take the `float` key. They take the `double` key instead when the value is finite but outside FLOAT's normal range. - Removed `isConvertFloatPrecisionLack` and the now-unused `TsFileConstant` import. Before and after, default mapping, tree model (`Time,root.fix.d3.s2,...`): | CSV value | Before | After | |---|---|---| | `1e5` | file aborted: `Unexpected error occurred: For input string: "1e5"`, exit status 1 | FLOAT `100000.0`, exit status 0 | | `Infinity`, `12f` | file aborted | FLOAT `Infinity`, `12.0` | | `16777217` | FLOAT, stored `16777216` | DOUBLE `1.6777217E7` | | `-16777217` | FLOAT, stored `-16777216` | DOUBLE `-1.6777217E7` | | `-123456789` | FLOAT, stored `-123456792` | DOUBLE `-1.23456789E8` | | `33554431` | FLOAT, stored `33554432` | DOUBLE `3.3554431E7` | | `1.0e39` | FLOAT, stored `Infinity` | DOUBLE `1.0E39` | | `-3000000001` with `-ti int=int` | INT32, line rejected | DOUBLE `-3.000000001E9` | All "After" values come from real imports. The "Before" values for `12f`, `33554431` and `1.0e39` come from calling `typeInfer` and `typeTrans` directly on the old code; the other "Before" values come from real imports. ## Regression tests Added to `DataToolRegressionTest`: - `testTypeInferKeepsIntegersExact`: inferred type and stored value around `±2^24` and `±2^25`. Covers `16777218` and `33554432` (exact, but in a range where neighbours are not), `-123456789`, `1099511627776`, the long bounds and an integer outside the long range. Also checks that the stored value is exactly the CSV value wherever the type can hold it. - `testTypeInferAcceptsEveryNumberThatIsNumberAccepts`: exponents (both cases), infinities, `NaN`, `-NaN`, suffixes, hex, the FLOAT range limits, values beyond DOUBLE, and non-numbers. - `testTypeInferWithInt32MappingTreatsBothSignsAlike`: `-ti int=int`. The shared mapping is restored in `finally`. - `testTreeCsvImportInfersNumbersWithoutLosingRows` and `testTableCsvImportInfersNumbersWithoutLosingRows`: full imports through mocked sessions with `batchPointSize = 2` (more than one batch). They mix values in the same column, include valid rows after `1e5` and `Infinity`, and include an explicitly typed `INT64` column. They check the inferred types, every stored value, success status, and that no failed file was written. Without the change to `AbstractDataTool`, all five new tests fail and the 39 existing ones pass: ``` [ERROR] Tests run: 44, Failures: 4, Errors: 1, Skipped: 0 <<< FAILURE! -- in org.apache.iotdb.tool.data.DataToolRegressionTest [ERROR] DataToolRegressionTest.testTypeInferKeepsIntegersExact:957->assertInferred:1349 16777217 expected:<DOUBLE> but was:<FLOAT> [ERROR] DataToolRegressionTest.testTypeInferWithInt32MappingTreatsBothSignsAlike:1011->assertInferred:1349 -3000000001 expected:<DOUBLE> but was:<INT32> java.lang.NumberFormatException: For input string: "12f" ``` ## Test commands and results JDK 25 (google-java-format in Spotless fails on JDK 27 with `NoSuchFieldError` in `JCCompilationUnit`, so formatting was run on 25). ``` $ ./mvnw test -pl iotdb-client/cli -am -Dtest=DataToolRegressionTest -Dsurefire.failIfNoSpecifiedTests=false [INFO] Tests run: 44, Failures: 0, Errors: 0, Skipped: 0 -- in org.apache.iotdb.tool.data.DataToolRegressionTest [INFO] BUILD SUCCESS $ ./mvnw clean test -pl iotdb-client/cli [INFO] Tests run: 85, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS (checkstyle and spotless:check ran in this build) $ ./mvnw spotless:check -pl iotdb-client/cli [INFO] BUILD SUCCESS $ ./mvnw test-compile -pl iotdb-client/cli -P with-zh-locale -Dspotless.skip=true exit 0 ``` I also imported CSVs with the CLI rebuilt from this branch into a local standalone server (built at `cd31721c14`; nothing under `iotdb-client/cli` changed between that commit and this branch's base). All tree imports and three of the four table imports finished with `Import completely!` and exit status 0. The fourth (`time,dev,s1` with `1e5`) failed with the separate table issue listed under known limitations; the same file with the column named `s3` imported correctly. The stored values match the "After" column above, for example: ``` $ select s1,s9 from root.fix.d2 (-batch 2; s9 = 1e5, Infinity, 12f on rows 5-7) |1970-01-01T05:30:00.005+05:30| 11.0| 100000.0| |1970-01-01T05:30:00.006+05:30| 12.0| Infinity| |1970-01-01T05:30:00.007+05:30| 13.0| 12.0| Total line number = 7 $ select * from f4 (table model, time,dev,s3 with 1e5 then 7) |1970-01-01T05:30:00.001+05:30| a|100000.0| |1970-01-01T05:30:00.002+05:30| a| 7.0| Total line number = 2 ``` `ImportDataTestIT` only checks the missing-source-directory message and does not reach type inference, so I did not run it for this change. ## Behaviour changes and known limitations - New columns get DOUBLE instead of FLOAT for integers with `2^24 < |v| <= 2^25`, including the even ones that happen to be exact, and for all negative integers below `-2^24`. Existing series and columns are not affected, since their type is queried rather than inferred. - `12f` and `3d` stay numeric (FLOAT), as `1.5f` and `1.5d` already were. Server-side inference treats suffixed values as text. I kept the CLI behaviour so inputs that import today do not change type. - Values beyond DOUBLE (`1e309`, `1e-400`) are stored as `Infinity` and `0.0`. They cannot be kept by either type. - Separate from this change: the table import path can lose a field value, or fail with `Missing or invalid column categories for table insertion`, when a new column is inferred in the middle of a file. The outcome depends on header iteration order. I reproduced both with plain integers on the unmodified CLI, so this PR does not change that behaviour, and I'd rather handle it separately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
