sgg opened a new pull request, #2660:
URL: https://github.com/apache/iceberg-go/pull/2660

   This diff adds support for writing to tables that define sort keys from 
nested struct fields.
   
   Previously attempting to write to a table with such a sort order defined 
would error. With this diff:
   
   * Tables whose sort keys are all top-level fields use arrow-go directly, 
same as before.
   * Tables with any sort key on a nested field fall back to the row comparator 
used by sort compaction (introduced with #2345).
   * This implementation is entirely internal to the table package so we should 
be able to swap it out later.
   * Sources under lists or maps are rejected w/ `ErrInvalidArgument` as they 
are not allowed by the Iceberg spec.
   * We now set `FloatTotalOrder` on arrow's sort keys so top-level float keys 
match the Iceberg spec's float order, same as the comparator. Previously, arrow 
placed NaN next to nulls and treated -0 as equal to 0, so tables w/ a top-level 
float sort key now write rows w/ NaN or -0 keys in a different order. This is 
also faster: a 64K-row float64 batch w/o nulls sorts in ~2.0 ms instead of 
~18.5 ms.
   
   Other Notes:
   
   * Long term we should add support to arrow-go. I am working on that (see 
apache/arrow-go#1337) and plan to follow up w/ a change removing this logic 
if/when I get that landed.
   * Bonus: sort compaction now supports nested sort keys.
   * I've added `BenchmarkSortRecordBatch` and `BenchmarkAppendSorted` to 
validate perf.
   * I benchmarked appending 1M rows against main: top-level keys run as fast 
as before, and a nested float64 key takes ~699 ms vs ~402 ms unsorted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to