sgg opened a new pull request, #2660: URL: https://github.com/apache/iceberg-go/pull/2660
This diff adds support for writing to tables that define sort keys from nested struct fields. Previously attempting to write to a table with such a sort order defined would error. With this diff: * Tables whose sort keys are all top-level fields use arrow-go directly, same as before. * Tables with any sort key on a nested field fall back to the row comparator used by sort compaction (introduced with #2345). * This implementation is entirely internal to the table package so we should be able to swap it out later. * Sources under lists or maps are rejected w/ `ErrInvalidArgument` as they are not allowed by the Iceberg spec. * We now set `FloatTotalOrder` on arrow's sort keys so top-level float keys match the Iceberg spec's float order, same as the comparator. Previously, arrow placed NaN next to nulls and treated -0 as equal to 0, so tables w/ a top-level float sort key now write rows w/ NaN or -0 keys in a different order. This is also faster: a 64K-row float64 batch w/o nulls sorts in ~2.0 ms instead of ~18.5 ms. Other Notes: * Long term we should add support to arrow-go. I am working on that (see apache/arrow-go#1337) and plan to follow up w/ a change removing this logic if/when I get that landed. * Bonus: sort compaction now supports nested sort keys. * I've added `BenchmarkSortRecordBatch` and `BenchmarkAppendSorted` to validate perf. * I benchmarked appending 1M rows against main: top-level keys run as fast as before, and a nested float64 key takes ~699 ms vs ~402 ms unsorted. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
