rahulsmahadev opened a new pull request, #4348: URL: https://github.com/apache/iceberg-python/pull/4348
# Rationale for this change A decimal filter literal with more digits than the column's precision makes the scan fail, even though the filter is valid. Here `amt` is `decimal(5, 2)`, so its largest value is 999.99: ```python tbl.scan(row_filter="amt < 1000.00").to_arrow() # ArrowInvalid: Decimal type with precision 6 does not fit into precision inferred from first array element: 5 ``` `!=`, `IN` and `NOT IN` with such a literal fail the same way. PyIceberg builds the Arrow literal with the column's type, `decimal128(5, 2)`, which cannot hold `1000.00`. Java compares by value here: its `DecimalLiteral.to` keeps the literal as is. This change gives the Arrow literal of a decimal column a precision wide enough for its digits (`decimal128(6, 2)` here) and keeps the column's scale, for comparisons and for `IN` / `NOT IN`. When the literal fits, the Arrow type is unchanged. Related to #4203 / #4210, which handle a column widened by schema evolution (old files keep the narrower type). This PR covers a literal wider than the table's own column type, which #4210 does not change. The two do not conflict: #4210 changes how the file-level filter is bound, this PR changes the Arrow type of decimal literals. ## Are these changes tested? Yes. Both new tests in `tests/io/test_pyarrow.py` fail with the `ArrowInvalid` above without the fix: - `test_decimal_literal_wider_than_column_to_pyarrow` converts `<`, `!=`, `IN` and `NOT IN` on a `decimal(5, 2)` column and evaluates them on an Arrow table. - `test_row_filter_decimal_literal_wider_than_column` scans an in-memory catalog table with `amt < 1000.00`, `amt > -1000.00`, `amt IN (1.25, 1000.00)` and `amt < 50.00` (a literal that fits). A NULL row in the table matches none of them. ## Are there any user-facing changes? Bug fix only: a decimal literal wider than the column's precision no longer makes the scan raise `ArrowInvalid`. No API changes. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
