andygrove opened a new issue, #7503: URL: https://github.com/apache/datafusion-comet/issues/7503
Part of #7267. These came out of the same audit as the other new sub-issues. Each needs an opt-in feature, sizes that are unlikely in practice, or a fix upstream, so they are collected here instead of filed one by one. Line references are for main at 4e6b12290f, DataFusion 55.2.0 and arrow 59.3.2. ## Comet - [ ] The native Iceberg writer (opt-in) concatenates pending pieces into 1000-row units (`native/core/src/execution/operators/iceberg_write.rs:1897-1909`, `ROWS_DIVISOR` at `:727`). Values averaging more than about 2.1 MB in one column overflow a unit with `OffsetOverflowError`. Iceberg's Java writer writes one row at a time. - [ ] `FixedWidthArrowFieldWriter.writeArrayElements` calls `ensureCapacity(count + numElements)` with Int arithmetic (`ArrowWriters.scala:980-1005`). Past 2^31 elements in one batch the sum goes negative, the buffer is not grown, and `copyMemory` writes past its end. Comet sets `arrow.enable_unsafe_memory_access=true` (`NativeBase.java:228`), so Arrow's own bounds checks don't catch it. `Math.addExact` would. - [ ] Importing JVM strings with invalid UTF-8 decodes them with U+FFFD, up to 3x the bytes, through a builder that panics on overflow (`native/common/src/utf8.rs:236-245`). - [ ] DataFusion UDF results of type `Utf8View`/`BinaryView` are cast back with arrow's `cast_view_to_byte` (`native/core/src/execution/planner.rs:2227-2244`), which panics on overflow instead of returning an error. - [ ] Past 2^31 list elements in one batch, these panic instead of returning an error: `array_insert` (`array_insert.rs:309`), `arrays_zip` (`arrays_zip.rs:295`), `percentile` (`percentile.rs:205-206`), `pivot_first` (`pivot_first.rs:536-537`). - [ ] `CometScalaUDFCodegen.estimatedOutputBytes` sums input sizes as an Int (`CometScalaUDFCodegen.scala:294-308`), so inputs over 2 GiB give a wrong preallocation size. - [ ] `Utils.narrowOffsets` (`Utils.scala:791-805`) fails broadcast and cache serialization for a large-offset column over 2 GiB, for example from `mapInArrow` with large variable-width types. - [ ] `ConstantColumnVectors.materialize` (`ArrowWriters.scala:129-143`) writes a constant once per row with no size limit, the JVM side of #7247. In practice these constants are small partition and default values. ## Upstream - [ ] arrow's `MapBuilder::append` pushes `key_builder.len() as i32` without a check (arrow-array `src/builder/map_builder.rs:176`), and `finish` builds the array unchecked (`:277`). Past 2^31 entries in one batch the offsets wrap silently. Comet reaches it through `str_to_map` (datafusion-spark) and the JVM shuffle's map conversion (`native/shuffle/src/spark_unsafe/row.rs:1234`). - [ ] parquet's DELTA_LENGTH_BYTE_ARRAY decoder panics on offset overflow (`src/arrow/array_reader/byte_array.rs:514`), where the other decoders return an error. - [ ] arrow-csv builds each string column with `collect::<StringArray>()` (`src/reader/mod.rs:872-879`), which panics past 2 GiB. Comet's native CSV scan is opt-in. - [ ] DataFusion's sort-merge join pushes to its `BatchCoalescer` with `.expect(...)` (`src/joins/sort_merge_join/materializing_stream.rs:465`, `:505`, `:516`), so a coalescer overflow (#7271) panics instead of failing with an error. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
