Karman580 opened a new issue, #19208:
URL: https://github.com/apache/iotdb/issues/19208

   ### Problem statement
   
   When `import-data` imports a CSV column whose type is not in the header and 
does not exist on the server yet, the CLI infers the type with 
`AbstractDataTool.typeInfer`. For numeric values this inference has two defects:
   
   - **A.** Integers that FLOAT cannot hold exactly are inferred as FLOAT, so 
they are silently rounded. The same happens to decimals outside the FLOAT 
range, which become `Infinity` or `0.0`.
   - **B.** Numbers such as `1e5` or `Infinity` make `typeInfer` throw 
`NumberFormatException`. The whole file is aborted after earlier batches may 
already have been written.
   
   ### Affected component
   
   `iotdb-client/cli`, file 
`iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractDataTool.java`:
   
   - `typeInfer(String)`
   - `isNumber(String)`
   - `isConvertFloatPrecisionLack(String)`
   
   These are called from `ImportDataTree` (tree model, aligned by time and by 
device) and `ImportDataTable` (table model). The default key mapping is in 
`Constants.TYPE_INFER_KEY_DICT` (`int` -> FLOAT, `long` -> DOUBLE, `float` -> 
FLOAT).
   
   ```java
   } else if (isNumber(strValue)) {
     if (!strValue.contains(TsFileConstant.PATH_SEPARATOR)) {
       if (isConvertFloatPrecisionLack(StringUtils.trim(strValue))) {
         return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_LONG);
       }
       return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_INT);
     } else {
       return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_FLOAT);
     }
   }
   ...
   private static boolean isConvertFloatPrecisionLack(String s) {
     return Long.parseLong(s) > (2 << 24);
   }
   ```
   
   ### Environment and commit
   
   - master @ `cd31721c1476b8edd07c4f205acd2fb4e046bc6d` (2.0.11-SNAPSHOT). 
Current master (`a4568b6e90`) has no changes under `iotdb-client/cli` since 
then. `dev/1.3` contains the same code.
   - macOS, JDK 27, Maven wrapper 3.9.12, standalone server and 
`tools/import-data.sh` from `distribution` built at that commit.
   
   ### Defect A — FLOAT precision inference
   
   FLOAT has a 24-bit significand, so integers with `|v| <= 2^24` (16777216) 
are exact. Above that, only some are. The check has two problems:
   
   - `2 << 24` is `2^25` (33554432), not `2^24`.
   - The comparison only has an upper bound, so every negative integer is 
inferred as FLOAT, however large its magnitude.
   
   The server had the same `2 << 24` bug and fixed it in IOTDB-4245 (#7321). It 
now uses a guarded `isLong` check and a configurable integer type. The CLI copy 
(added in #13930) still has the old logic.
   
   Decimals take the `float` key without a range check, so values beyond the 
FLOAT range lose their value entirely.
   
   ### Defect B — NumberFormatException
   
   `isNumber` accepts everything `Double.parseDouble` accepts. Any such value 
without a `.` is then passed to `Long.parseLong`, which throws for:
   
   - `1e5`, `1E10`
   - `Infinity`, `-Infinity`, `-NaN`
   - `12f`, `3d`
   - `99999999999999999999` (outside the long range)
   - `1e309`, `1e-400`, `0x1p3`
   
   The exception propagates to `AbstractImportData.run()`, which marks the file 
as failed.
   
   ### Steps to reproduce
   
   Tree model:
   
   ```
   $ cat a3.csv
   Time,root.test.d3.s1
   1,1e5
   2,7
   $ tools/import-data.sh -h 127.0.0.1 -p 6667 -u root -pw root -ft csv 
-sql_dialect tree -s a3.csv
   
   $ cat b.csv            # likewise c.csv with root.test.d1.s3 = -16777217
   Time,root.test.d1.s2   # and f.csv with root.test.d1.s4 = -123456789
   1,16777217
   $ tools/import-data.sh ... -ft csv -sql_dialect tree -s b.csv
   ```
   
   Table model (target table created first as `create table t3(dev STRING 
TAG)`):
   
   ```
   $ cat tp.csv
   time,dev,s4,s5
   1,a,7,
   2,a,8,
   3,a,9,
   4,a,10,
   5,a,11,Infinity
   6,a,12,1
   $ tools/import-data.sh ... -ft csv -sql_dialect table -db testdb -table t3 
-batch 2 -s tp.csv
   ```
   
   ### Actual behaviour
   
   Defect B, tree model:
   
   ```
   [a3.csv] - Unexpected error occurred: For input string: "1e5"
   exit status = 1
   files in dir: a3.csv
   ```
   
   No series was created, and row 2 (`7`) was not imported either.
   
   With `-batch 2`, a 7-row file that has `1e5` on row 5 kept rows 1-4 and lost 
rows 5-7. The new column was not created and no `.failed` file was written.
   
   Defect B, table model:
   
   ```
   [tp.csv] - Unexpected error occurred: For input string: "Infinity"
   exit status = 1
   files in dir: tp.csv
   ```
   
   ```
   $ select * from t3
   +-----------------------------+---+----+
   |                         time|dev|  s4|
   +-----------------------------+---+----+
   |1970-01-01T05:30:00.001+05:30|  a| 7.0|
   |1970-01-01T05:30:00.002+05:30|  a| 8.0|
   |1970-01-01T05:30:00.003+05:30|  a| 9.0|
   |1970-01-01T05:30:00.004+05:30|  a|10.0|
   +-----------------------------+---+----+
   ```
   
   Defect A, tree model: the import printed `Import completely!` with exit 
status 0. All columns were created as FLOAT.
   
   ```
   $ select cast(s2 as INT64) as s2_i64, cast(s3 as INT64) as s3_i64, cast(s4 
as INT64) as s4_i64 from root.test.d1
   +-----------------------------+--------+---------+----------+
   |                         Time|  s2_i64|   s3_i64|    s4_i64|
   +-----------------------------+--------+---------+----------+
   |1970-01-01T05:30:00.001+05:30|16777216|-16777216|-123456792|
   +-----------------------------+--------+---------+----------+
   ```
   
   Defect A, table model, `-16777217` into a new column:
   
   ```
   |                         time|dev|          s3|      s3i|
   |1970-01-01T05:30:00.001+05:30|  a|-1.6777216E7|-16777216|
   ```
   
   With `-ti int=int` (the `int` key mapped to INT32), `3000000001` is inferred 
as DOUBLE. `-3000000001` is inferred as INT32 and rejected: `Line '1', column 
'root.test.d5.neg': '-3000000001' can't convert to 'INT32'`.
   
   Columns whose type is given in the header (for example `s1(DOUBLE)`), or 
that already exist, skip inference and are not affected.
   
   ### Expected behaviour
   
   - An inferred numeric type should hold the value exactly whenever the 
configured type allows it. For integers, the `int` key should only be used when 
FLOAT holds every integer of that magnitude, for both signs.
   - A value that `isNumber` accepts should never make type inference throw. It 
should be inferred as a numeric type, and the import should continue.
   
   ### Real test results
   
   Method-level probe that calls the compiled `typeInfer`, 
`isConvertFloatPrecisionLack` and `typeTrans` from 
`iotdb-cli-2.0.11-SNAPSHOT.jar` by reflection:
   
   ```
   input                  lack()   infer  stored   storedValue              
exact?
   16777215               false    FLOAT  Float    16777215                 true
   16777216               false    FLOAT  Float    16777216                 true
   16777217               false    FLOAT  Float    16777216                 
false
   16777218               false    FLOAT  Float    16777218                 true
   33554431               false    FLOAT  Float    33554432                 
false
   33554432               false    FLOAT  Float    33554432                 true
   33554433               true     DOUBLE Double   33554433                 true
   -16777216              false    FLOAT  Float    -16777216                true
   -16777217              false    FLOAT  Float    -16777216                
false
   -33554433              false    FLOAT  Float    -33554432                
false
   -100000001             false    FLOAT  Float    -100000000               
false
   ```
   
   ```
   1.0e39 -> FLOAT -> stored Infinity
   3.5e38 -> FLOAT -> stored Infinity
   1.0e-50 -> FLOAT -> stored 0.0
   ```
   
   ```
   input                    isNumber  typeInfer
   [1e5]                    true      THROWS NumberFormatException
   [1E10]                   true      THROWS NumberFormatException
   [1.5e3]                  true      FLOAT
   [Infinity]               true      THROWS NumberFormatException
   [-Infinity]              true      THROWS NumberFormatException
   [NaN]                    false     DOUBLE
   [12f]                    true      THROWS NumberFormatException
   [3d]                     true      THROWS NumberFormatException
   [99999999999999999999]   true      THROWS NumberFormatException
   [+5]                     true      FLOAT
   [-5]                     true      FLOAT
   [1.0]                    true      FLOAT
   
   java.lang.NumberFormatException: For input string: "1e5"
        at java.base/java.lang.Long.parseLong(Long.java:660)
        at 
org.apache.iotdb.tool.data.AbstractDataTool.isConvertFloatPrecisionLack(AbstractDataTool.java:509)
        at 
org.apache.iotdb.tool.data.AbstractDataTool.typeInfer(AbstractDataTool.java:455)
   ```
   
   The existing regression tests pass, but none of them cover numeric type 
inference:
   
   ```
   $ ./mvnw test -pl iotdb-client/cli -am -Dtest=DataToolRegressionTest 
-Dsurefire.failIfNoSpecifiedTests=false -Dspotless.skip=true -Drat.skip=true 
-Dcheckstyle.skip=true
   [INFO] Tests run: 39, Failures: 0, Errors: 0, Skipped: 0 -- in 
org.apache.iotdb.tool.data.DataToolRegressionTest
   [INFO] BUILD SUCCESS
   ```
   
   ### Root cause
   
   - `isConvertFloatPrecisionLack` uses a wrong, one-sided bound (`> 2^25`), 
and its unguarded `Long.parseLong` assumes that "numeric without a `.`" means 
"fits in a long".
   - The decimal branch picks the `float` key without checking the FLOAT range.
   
   ### Proposed solution
   
   Keep the CLI's own key mapping (`int`/`long`/`float`/`double` keys, 
configurable with `-ti`). Do not copy the server's type defaults. Change only 
how a numeric string selects a key:
   
   - **Plain integers (`[+-]?\d+`):** use the `int` key only when `-2^24 <= v 
<= 2^24`. Otherwise use the `long` key, which also covers integers outside the 
long range. This is a bound on magnitude rather than a per-value 
representability check, because the inferred type is applied to every later 
value in the column.
   - **Other numerics** (decimals, exponents, `Infinity`, suffixed values such 
as `12f`, which already import as FLOAT for `1.5f`): use the `float` key. If 
the value is finite but outside FLOAT's normal range, use the `double` key 
instead.
   
   To decide: should `12f`/`3d` stay numeric, as the CLI already treats `1.5f`, 
or become text, as the server does? The proposal keeps the current CLI 
behaviour for suffixed values.
   
   ### Regression testing
   
   - Table-driven tests in `DataToolRegressionTest` for the inferred type and 
stored value: values around `±2^24` and `±2^25`, the long bounds, an 
out-of-range integer, scientific notation, infinities, NaN, suffixes and the 
FLOAT range limits.
   - Tree and table import tests with mocked sessions: a valid row after one of 
these values, multiple batches, and explicitly typed columns.
   
   ### Additional context
   
   - Related server fix: IOTDB-4245 / #7321 (`b0129297c5`).
   - I found no existing issue or PR covering this. #18238 touches the same 
import classes but only fixes the `CSVParser` file-descriptor leak.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to