andygrove opened a new issue, #7500: URL: https://github.com/apache/datafusion-comet/issues/7500
Part of #7267. ## Problem Several native kernels produce more bytes than they read. Spark evaluates one row at a time, so each row's result only has to fit in a value. In Comet, the batch's result is one Arrow array, so it fails once the batch total passes 2 GiB: | Expression | Kernel | Output size | Failure | | --- | --- | --- | --- | | `replace(col, 'a', 'bbb')`, native by default for literal arguments (`spark/src/main/scala/org/apache/comet/serde/strings.scala:187-215`) | DataFusion `replace` | grows with replacement length / search length per match | error `byte array offset overflow: output size exceeds 2147483647 bytes` | | `repeat(col, n)` | DataFusion `repeat` (`string/repeat.rs:224-230`) | n times the input | panic `byte array offset overflow` | | `rpad(col, len, pad)`, `lpad(...)` | Comet `static_invoke/char_varchar_utils/read_side_padding.rs:205-262` | up to `len` per row | panic `byte array offset overflow` | | `base64(col)` | Comet `string_funcs/base64.rs:142-155` | about 1.37x the input | panic `byte array offset overflow` | | `CAST(binary AS STRING)` of invalid UTF-8 | Comet `conversion_funcs/cast.rs:984-1054` | up to 3x (U+FFFD) | panic `byte array offset overflow` | | `hex(binary)` | datafusion-spark `math/hex.rs:128,164` | 2x the input | error `hex output exceeds i32 offset range` | `space(n)` and `randstr(n)` panic the same way. `repeat`, `rpad`/`lpad`, `base64` and `randstr` also reserve the whole batch's output before writing anything (`with_capacity` on the total). When the allocator refuses that reservation, arrow aborts the process (`MutableBuffer::with_capacity` calls `handle_alloc_error`, arrow-buffer 59.3.2 `src/buffer/mutable.rs:131-140`). So the executor dies instead of the task failing. `repeat(s, 1000000)` over 8192 rows of 100-byte strings asks for about 819 GB, and `rpad(s, 10000000, ' ')` for about 82 GB. Line references are for main at 4e6b12290f, DataFusion 55.2.0 and arrow 59.3.2. ## Workaround Lower `spark.comet.batchSize`, which helps when the growth is moderate. ## Proposed fix - Check the computed total against `i32::MAX` before allocating, and return an error that suggests a smaller batch size, in Comet's kernels and upstream in `repeat`. That removes the abort and the panics. - Matching Spark needs these kernels to return several arrays, or `LargeUtf8`, for such batches. That is the same problem as #7247 and #7498. ## Tests Call each kernel with a few rows whose total output passes 2 GiB and assert an error instead of a panic. Test the up-front size check with a request far beyond available memory, which today aborts the test process. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
