wombatu-kun opened a new issue, #16605:
URL: https://github.com/apache/iceberg/issues/16605
### Problem
`SchemaUtils.inferIcebergType` (used to infer a column type for schemaless
records when schema evolution is enabled) builds a decimal type directly from a
`BigDecimal`'s precision and scale:
```java
return DecimalType.of(bigDecimal.precision(), bigDecimal.scale());
```
Iceberg's decimal requires `0 <= scale <= precision <= 38`, but a
`BigDecimal` does not guarantee that:
- A value with magnitude < 1 can have fewer significant digits than its
scale. `new BigDecimal("0.001")` has `precision() == 1` and `scale() == 3`,
producing `decimal(1, 3)` (scale > precision).
- A value can have a negative scale (BigDecimal preserves the parsed form).
`new BigDecimal("1E+2")` has `precision() == 1` and `scale() == -2`, producing
`decimal(1, -2)`.
`DecimalType.of` only validates `precision <= 38`, so these malformed types
are constructed silently.
### When this is reached
`inferIcebergType` runs when (1) schema evolution is enabled, (2) the record
is schemaless (a `Map`, no Connect schema), (3) the field is a new column, and
(4) the field value is a `java.math.BigDecimal`. The stock `JsonConverter`
(`schemas.enable=false`) deserializes fractional JSON numbers to `Double`, so
this is reached when a `BigDecimal` enters a schemaless map by another route.
### Reproduction
End-to-end: with schema evolution enabled, a schemaless record carrying a
fractional decimal for a new column makes the write fail. On current code:
```
org.apache.kafka.connect.errors.DataException: An error occurred converting
record, topic: topic, partition, 1, offset: 100
Caused by: java.lang.IllegalArgumentException: Invalid DECIMAL scale: 3
cannot be greater than precision: 1
```
At the unit level, `inferIcebergType` returns the malformed type directly
(it does not throw, since `DecimalType.of` only validates `precision <= 38`):
```java
assertThat(SchemaUtils.inferIcebergType(new BigDecimal("0.001"), config))
.isEqualTo(DecimalType.of(3, 3)); // current code infers decimal(1, 3)
assertThat(SchemaUtils.inferIcebergType(new BigDecimal("1E+2"), config))
.isEqualTo(DecimalType.of(3, 0)); // current code infers decimal(1, -2)
```
### Fix
Normalize precision and scale to satisfy Iceberg's invariant: rescale a
negative scale to 0, and widen precision to at least the scale.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
