andygrove opened a new issue, #7497: URL: https://github.com/apache/datafusion-comet/issues/7497
Part of #7267. ## Problem The JVM columnar shuffle hands sorted UnsafeRows to native code, which converts them to Arrow in batches of `spark.comet.shuffle.jvm.batchSize` rows, capped by `spark.comet.batchSize`. Both entry points cut batches by row count only: `process_sorted_partitions` splits each partition's rows with `run.chunks(batch_size)` (`native/shuffle/src/spark_unsafe/row.rs:1686`), and `process_sorted_row_partition` does the same at `:1579-1588`. Strings and binaries are appended with arrow's `append_value` (`:1173-1203`), which panics with `byte array offset overflow` once a column passes 2 GiB. Strings inside arrays, maps and structs go through the same builders. So a batch fails when one string or binary column holds more than 2 GiB, which is more than about 256 KiB per value at 8192 rows. When the column is dictionary-encoded, the writer succeeds and the reader fails instead (#7496). When the dictionary doesn't pay off, the cast back to plain values returns `OffsetOverflowError` (`row.rs:1777`). Spark shuffles rows and has no batch to overflow. #7269 covers the native shuffle writer, which is separate code. Line references are for main at 4e6b12290f. ## Workaround Lower `spark.comet.shuffle.jvm.batchSize`. ## Proposed fix Cut batches by bytes as well as rows. The row sizes are known before conversion (`row_sizes`), and a row's size bounds the bytes it adds to any column, so ending a batch before the sum of its row sizes passes `i32::MAX` keeps every column valid without looking at the values. ## Tests Inject a small byte budget and check that `process_sorted_row_partition` writes several blocks for rows that would otherwise share one, with the same rows read back. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
