andygrove opened a new issue, #6596:
URL: https://github.com/apache/datafusion-comet/issues/6596
### What is the problem the feature request solves?
Comet's JVM columnar shuffle buffers Spark `UnsafeRow`s and converts them to
Arrow natively, in `native/shuffle/src/spark_unsafe/`. I compared that
converter with the JVM `ArrowWriter` row path that `CometSparkToColumnarExec`
uses (#6566), on identical rows. Spark's `UnsafeProjection` generated the rows
from the column shapes in `CometArrowWriterBenchmark`: one column, 8192 rows,
every tenth row null.
Per-row cost in ns:
| Type | Native converter | JVM `ArrowWriter` |
| ----------------------------- | ---------------- | ----------------- |
| int, boolean | 2.3 | 4.0-5.2 |
| decimal(18,2), decimal(38,10) | 2.4, 3.8 | 7.6, 13.4 |
| struct<int,long,double,date> | 11.4 | 22.3-22.7 |
| array<int>, array<string> | 9.1, 38 | 11.0-11.8, 32.9-35.7 |
| string | 20-23 | 9.1-13 |
| map<string,string> | 149 | 65-86 |
The native converter is two to three times faster on fixed-width types,
decimals and structs, but it loses on strings and maps. Three per-value costs
account for that.
- **Map field names.** `append_map_elements` calls
`get_map_key_value_fields` for every map value. That builds a `MapFieldNames`,
allocating and freeing six strings, to read two field references. Resolving the
fields once per batch takes `map<string,string>` from 149 to 73 ns per row.
- **UTF-8 validation.** `get_string` checks each value with
`decode_utf8_spark_lossy`, and its `str::from_utf8` call is about 7.5 of the 12
ns that a short string takes. Checking `is_ascii()` first, and decoding only
the values that are not ASCII, gives 5.1 ns with identical output. Validating a
whole batch once, as `StringArray::try_new` does, gives 5.2. The same cost
repeats for every key and value of a map: without it, `map<string,string>`
drops to about 31 ns.
- **Element arrays.** The `append_slice` bulk path only runs for map keys.
Array elements and map values are appended one at a time, even when an array
has no nulls. For arrays of 50 ints, a bulk copy when the array's null bitset
is empty takes 51.7 ns per row down to 9.6. For arrays of zero to five elements
the bulk copy is slightly slower than the loop, so it needs a length threshold.
### Describe the potential solution
- Read a map's key and value fields without building `MapFieldNames`, which
only `make_builders` needs.
- Add an ASCII fast path to `decode_utf8_spark_lossy`.
- Bulk-copy the elements of an array that has no nulls once it holds more
than a few elements.
### Additional context
Measured on an M3 Max with a release build (`-Ctarget-cpu=native`), timing
`append_columns` plus `builder_to_array` per 8192-row batch and taking the best
of 2000 batches. The JVM numbers come from the same rows through
`ArrowWriter.write`. Other jobs were running on the machine, so treat the
numbers as indicative.
For fixed-width columns, compression costs more than conversion. The whole
`process_sorted_row_partition` call for a `decimal(18,2)` column takes 2.4 ns
per row to convert, and 14.6 with LZ4 encoding and the write.
The default dictionary trial for top-level strings is a separate cost,
tracked in {{B}}. Converting row input with `CometSparkToColumnarExec` and
using native shuffle, instead of the JVM columnar shuffle, is tracked in {{C}}.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
