Anurag-S1ngh opened a new pull request, #4334: URL: https://github.com/apache/iceberg-python/pull/4334
# Rationale for this change `get_rows_to_update` compared every matched row one cell at a time, slicing both tables and calling `as_py()` for each column. Cost grew with rows times columns: on a table with 200 columns and 20k matched rows the comparison alone took around 20 seconds, before anything was written. The matched rows are now gathered once and compared column by column with PyArrow kernels. Two nulls count as equal, a null against a value counts as a change, and every row is still returned only once. Columns whose types have no comparison kernel, structs and lists today, fall back to the previous value-by-value comparison, so nested types behave as before. The reproduction from the issue now takes about 0.1 seconds instead of about 20. # Are these changes tested? Yes. New tests in `tests/table/test_upsert.py` cover `get_rows_to_update` directly: - only the changed rows are returned - a row is returned once even when several columns changed - two nulls compare equal while a null against a value is a change - struct columns are still compared Full unit suite: `4570 passed, 5 skipped`. `prek run -a` passes. # Are there any user-facing changes? Yes, only in speed: `upsert` no longer slows down sharply with the number of columns. The returned rows are unchanged, except that two NaN values in the same column now compare equal, which matches Iceberg's comparator semantics. Closes #3860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
