Karman580 opened a new issue, #19208:
URL: https://github.com/apache/iotdb/issues/19208
### Problem statement
When `import-data` imports a CSV column whose type is not in the header and
does not exist on the server yet, the CLI infers the type with
`AbstractDataTool.typeInfer`. For numeric values this inference has two defects:
- **A.** Integers that FLOAT cannot hold exactly are inferred as FLOAT, so
they are silently rounded. The same happens to decimals outside the FLOAT
range, which become `Infinity` or `0.0`.
- **B.** Numbers such as `1e5` or `Infinity` make `typeInfer` throw
`NumberFormatException`. The whole file is aborted after earlier batches may
already have been written.
### Affected component
`iotdb-client/cli`, file
`iotdb-client/cli/src/main/java/org/apache/iotdb/tool/data/AbstractDataTool.java`:
- `typeInfer(String)`
- `isNumber(String)`
- `isConvertFloatPrecisionLack(String)`
These are called from `ImportDataTree` (tree model, aligned by time and by
device) and `ImportDataTable` (table model). The default key mapping is in
`Constants.TYPE_INFER_KEY_DICT` (`int` -> FLOAT, `long` -> DOUBLE, `float` ->
FLOAT).
```java
} else if (isNumber(strValue)) {
if (!strValue.contains(TsFileConstant.PATH_SEPARATOR)) {
if (isConvertFloatPrecisionLack(StringUtils.trim(strValue))) {
return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_LONG);
}
return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_INT);
} else {
return Constants.TYPE_INFER_KEY_DICT.get(Constants.DATATYPE_FLOAT);
}
}
...
private static boolean isConvertFloatPrecisionLack(String s) {
return Long.parseLong(s) > (2 << 24);
}
```
### Environment and commit
- master @ `cd31721c1476b8edd07c4f205acd2fb4e046bc6d` (2.0.11-SNAPSHOT).
Current master (`a4568b6e90`) has no changes under `iotdb-client/cli` since
then. `dev/1.3` contains the same code.
- macOS, JDK 27, Maven wrapper 3.9.12, standalone server and
`tools/import-data.sh` from `distribution` built at that commit.
### Defect A — FLOAT precision inference
FLOAT has a 24-bit significand, so integers with `|v| <= 2^24` (16777216)
are exact. Above that, only some are. The check has two problems:
- `2 << 24` is `2^25` (33554432), not `2^24`.
- The comparison only has an upper bound, so every negative integer is
inferred as FLOAT, however large its magnitude.
The server had the same `2 << 24` bug and fixed it in IOTDB-4245 (#7321). It
now uses a guarded `isLong` check and a configurable integer type. The CLI copy
(added in #13930) still has the old logic.
Decimals take the `float` key without a range check, so values beyond the
FLOAT range lose their value entirely.
### Defect B — NumberFormatException
`isNumber` accepts everything `Double.parseDouble` accepts. Any such value
without a `.` is then passed to `Long.parseLong`, which throws for:
- `1e5`, `1E10`
- `Infinity`, `-Infinity`, `-NaN`
- `12f`, `3d`
- `99999999999999999999` (outside the long range)
- `1e309`, `1e-400`, `0x1p3`
The exception propagates to `AbstractImportData.run()`, which marks the file
as failed.
### Steps to reproduce
Tree model:
```
$ cat a3.csv
Time,root.test.d3.s1
1,1e5
2,7
$ tools/import-data.sh -h 127.0.0.1 -p 6667 -u root -pw root -ft csv
-sql_dialect tree -s a3.csv
$ cat b.csv # likewise c.csv with root.test.d1.s3 = -16777217
Time,root.test.d1.s2 # and f.csv with root.test.d1.s4 = -123456789
1,16777217
$ tools/import-data.sh ... -ft csv -sql_dialect tree -s b.csv
```
Table model (target table created first as `create table t3(dev STRING
TAG)`):
```
$ cat tp.csv
time,dev,s4,s5
1,a,7,
2,a,8,
3,a,9,
4,a,10,
5,a,11,Infinity
6,a,12,1
$ tools/import-data.sh ... -ft csv -sql_dialect table -db testdb -table t3
-batch 2 -s tp.csv
```
### Actual behaviour
Defect B, tree model:
```
[a3.csv] - Unexpected error occurred: For input string: "1e5"
exit status = 1
files in dir: a3.csv
```
No series was created, and row 2 (`7`) was not imported either.
With `-batch 2`, a 7-row file that has `1e5` on row 5 kept rows 1-4 and lost
rows 5-7. The new column was not created and no `.failed` file was written.
Defect B, table model:
```
[tp.csv] - Unexpected error occurred: For input string: "Infinity"
exit status = 1
files in dir: tp.csv
```
```
$ select * from t3
+-----------------------------+---+----+
| time|dev| s4|
+-----------------------------+---+----+
|1970-01-01T05:30:00.001+05:30| a| 7.0|
|1970-01-01T05:30:00.002+05:30| a| 8.0|
|1970-01-01T05:30:00.003+05:30| a| 9.0|
|1970-01-01T05:30:00.004+05:30| a|10.0|
+-----------------------------+---+----+
```
Defect A, tree model: the import printed `Import completely!` with exit
status 0. All columns were created as FLOAT.
```
$ select cast(s2 as INT64) as s2_i64, cast(s3 as INT64) as s3_i64, cast(s4
as INT64) as s4_i64 from root.test.d1
+-----------------------------+--------+---------+----------+
| Time| s2_i64| s3_i64| s4_i64|
+-----------------------------+--------+---------+----------+
|1970-01-01T05:30:00.001+05:30|16777216|-16777216|-123456792|
+-----------------------------+--------+---------+----------+
```
Defect A, table model, `-16777217` into a new column:
```
| time|dev| s3| s3i|
|1970-01-01T05:30:00.001+05:30| a|-1.6777216E7|-16777216|
```
With `-ti int=int` (the `int` key mapped to INT32), `3000000001` is inferred
as DOUBLE. `-3000000001` is inferred as INT32 and rejected: `Line '1', column
'root.test.d5.neg': '-3000000001' can't convert to 'INT32'`.
Columns whose type is given in the header (for example `s1(DOUBLE)`), or
that already exist, skip inference and are not affected.
### Expected behaviour
- An inferred numeric type should hold the value exactly whenever the
configured type allows it. For integers, the `int` key should only be used when
FLOAT holds every integer of that magnitude, for both signs.
- A value that `isNumber` accepts should never make type inference throw. It
should be inferred as a numeric type, and the import should continue.
### Real test results
Method-level probe that calls the compiled `typeInfer`,
`isConvertFloatPrecisionLack` and `typeTrans` from
`iotdb-cli-2.0.11-SNAPSHOT.jar` by reflection:
```
input lack() infer stored storedValue
exact?
16777215 false FLOAT Float 16777215 true
16777216 false FLOAT Float 16777216 true
16777217 false FLOAT Float 16777216
false
16777218 false FLOAT Float 16777218 true
33554431 false FLOAT Float 33554432
false
33554432 false FLOAT Float 33554432 true
33554433 true DOUBLE Double 33554433 true
-16777216 false FLOAT Float -16777216 true
-16777217 false FLOAT Float -16777216
false
-33554433 false FLOAT Float -33554432
false
-100000001 false FLOAT Float -100000000
false
```
```
1.0e39 -> FLOAT -> stored Infinity
3.5e38 -> FLOAT -> stored Infinity
1.0e-50 -> FLOAT -> stored 0.0
```
```
input isNumber typeInfer
[1e5] true THROWS NumberFormatException
[1E10] true THROWS NumberFormatException
[1.5e3] true FLOAT
[Infinity] true THROWS NumberFormatException
[-Infinity] true THROWS NumberFormatException
[NaN] false DOUBLE
[12f] true THROWS NumberFormatException
[3d] true THROWS NumberFormatException
[99999999999999999999] true THROWS NumberFormatException
[+5] true FLOAT
[-5] true FLOAT
[1.0] true FLOAT
java.lang.NumberFormatException: For input string: "1e5"
at java.base/java.lang.Long.parseLong(Long.java:660)
at
org.apache.iotdb.tool.data.AbstractDataTool.isConvertFloatPrecisionLack(AbstractDataTool.java:509)
at
org.apache.iotdb.tool.data.AbstractDataTool.typeInfer(AbstractDataTool.java:455)
```
The existing regression tests pass, but none of them cover numeric type
inference:
```
$ ./mvnw test -pl iotdb-client/cli -am -Dtest=DataToolRegressionTest
-Dsurefire.failIfNoSpecifiedTests=false -Dspotless.skip=true -Drat.skip=true
-Dcheckstyle.skip=true
[INFO] Tests run: 39, Failures: 0, Errors: 0, Skipped: 0 -- in
org.apache.iotdb.tool.data.DataToolRegressionTest
[INFO] BUILD SUCCESS
```
### Root cause
- `isConvertFloatPrecisionLack` uses a wrong, one-sided bound (`> 2^25`),
and its unguarded `Long.parseLong` assumes that "numeric without a `.`" means
"fits in a long".
- The decimal branch picks the `float` key without checking the FLOAT range.
### Proposed solution
Keep the CLI's own key mapping (`int`/`long`/`float`/`double` keys,
configurable with `-ti`). Do not copy the server's type defaults. Change only
how a numeric string selects a key:
- **Plain integers (`[+-]?\d+`):** use the `int` key only when `-2^24 <= v
<= 2^24`. Otherwise use the `long` key, which also covers integers outside the
long range. This is a bound on magnitude rather than a per-value
representability check, because the inferred type is applied to every later
value in the column.
- **Other numerics** (decimals, exponents, `Infinity`, suffixed values such
as `12f`, which already import as FLOAT for `1.5f`): use the `float` key. If
the value is finite but outside FLOAT's normal range, use the `double` key
instead.
To decide: should `12f`/`3d` stay numeric, as the CLI already treats `1.5f`,
or become text, as the server does? The proposal keeps the current CLI
behaviour for suffixed values.
### Regression testing
- Table-driven tests in `DataToolRegressionTest` for the inferred type and
stored value: values around `±2^24` and `±2^25`, the long bounds, an
out-of-range integer, scientific notation, infinities, NaN, suffixes and the
FLOAT range limits.
- Tree and table import tests with mocked sessions: a valid row after one of
these values, multiple batches, and explicitly typed columns.
### Additional context
- Related server fix: IOTDB-4245 / #7321 (`b0129297c5`).
- I found no existing issue or PR covering this. #18238 touches the same
import classes but only fixes the `CSVParser` file-descriptor leak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]