andygrove opened a new issue, #7494: URL: https://github.com/apache/datafusion-comet/issues/7494
Part of #7267. ## Problem With `spark.comet.exec.columnarToRow.native.enabled=true` (default false), `ColumnarToRowContext::convert` writes every row of a batch into one buffer and returns each row's start and length as `i32`: - `native/core/src/execution/columnar_to_row.rs:1003` pushes `row_start as i32`, and `:1009` pushes `(row_end - row_start) as i32`. - The fixed-width path does the same at `:1112`, `(row_idx * row_size) as i32`. Nothing checks the buffer size. Once a batch's rows take more than `i32::MAX` bytes, the offsets wrap negative and `convert` still returns `Ok`. The JVM adds each offset to the buffer address (`NativeColumnarToRowConverter.scala:137`) and copies the row from there, so every row past the 2 GiB mark is read from memory before the buffer. That gives garbage rows, or a SIGSEGV if the memory is unmapped. Each column can stay well under Arrow's limit. Two string columns averaging 150 KB per value hold about 1.2 GB each in an 8192-row batch, and the batch's rows take 2.4 GB. Spark converts one row at a time and returns correct rows. A unit test with two `Utf8` columns of five 300 MiB rows (1.5 GiB per column, so valid Arrow) shows the wrap: ``` offsets=[0, 629145624, 1258291248, 1887436872, -1778384800] ``` Line references are for main at 4e6b12290f. ## Workaround Keep `spark.comet.exec.columnarToRow.native.enabled=false`, the default. ## Proposed fix Return the row offsets as `i64` (a `long[]` on the JVM side), so a batch's rows can take more than 2 GiB. Convert each row length with a checked conversion and fail a row larger than `ByteArrayMethods.MAX_ROUNDED_ARRAY_LENGTH`, as `DirectColumnarToRowConverter` already does. A single row that large fails in Spark's `UnsafeRowWriter` too. ## Tests A batch past 2 GiB needs about 3 GB of memory, too much for CI, so that test can be `#[ignore]`d and run by hand. The row length check can be tested directly with sizes past the limit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
