erikbogado-nstech commented on issue #7442: URL: https://github.com/apache/datafusion-comet/issues/7442#issuecomment-6750656693
I'm taking this one, following the expected behavior described above. Plan: - Expression: `CometIcebergTruncate` declines an integer `truncate[W]` with `W > 2^30`, so Spark evaluates it with the user's Iceberg jar. Long, string and binary truncation stay native, because `truncateLong` uses long arithmetic and `W` is an `int`, so it cannot overflow at these widths. - Native write: a new trigger inspects the output partition spec and declines when a `truncate[W]` field with `W > 2^30` has an `int` or `date` source type. That keeps native writes off tables where iceberg-rust would write the spec value and the user's jar would write the overflowed one. - Docs and test comments: the Rust test comment and the user guide now cite the spec and attribute the overflow to Java (apache/iceberg#18111, fix in apache/iceberg#18112), keeping `i32::MIN` / `i64::MIN` as the iceberg-rust cases. Tests: support-level assertions at `2^30` and above for int and long, an expression fallback test against Spark's answer, and write-detection tests for a `truncate(2000000000, id)` spec and for the maximum safe width. PR to follow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
