GaspardMerten opened a new pull request, #3862: URL: https://github.com/apache/iceberg-python/pull/3862
<!-- Closes #3860 --> # Rationale for this change `upsert` compares the matched rows one cell at a time. For every matched row, it takes a one-row slice of the source and of the target, then calls `.as_py()` on each non-key column until it finds a difference. That is one PyArrow call per row and per column, so the cost grows with the size of the table and not with the amount of data that actually changed. On a table with 200 columns, comparing 20k matched rows takes around 20 seconds before anything is written on a standard computer (mine), while it takes 0.15s if we push this comparison logic to PyArrow Compute (PC). Two other things come out of that: - PyArrow cannot compare struct columns (apache/arrow#35785), so I implemented nested comparison, still pushing to PC. - The previous system was casting the source table to the target schema, which in itself is expensive. In addition, when some types such as `timestamp[us]` were not cast to `timestamp[us, UTC]` as PyArrow refuses, it reverted back to Python, which made every row reported as changed. This is a small thing we gain. ! Lists and maps have no fields to compare and still go to Python, a slice at a time so the objects of a whole column are never held at once. The result is the same: two nulls still count as equal, a null and a value still count as a change, and rows whose non-key columns did not change are still skipped. # Are these changes tested? Yes, 10 tests in `tests/table/test_upsert.py`. Eight cover the comparison itself: nulls, a struct that is null, a nested struct, a list column, a column whose type differs from the target, a cast PyArrow refuses, a source missing one of the target columns, and no match at all. Two cover the change in the way rows are compared, which no assertion on the result can see, because both ways return the same rows. One counts the comparisons and checks there is one per column, whatever the number of rows. The other checks a struct never reaches the Python comparison, with the types the upsert path really produces (a scan reads a `string` as a `large_string`). The existing `tests/table/test_upsert.py` and `tests/table` suites pass unchanged. # Are there any user-facing changes? No API or behaviour change. `get_rows_to_update` returns the same rows. One error message changes. A source that does not have every column of the target was rejected by `Table.cast` with `Target schema's field names are not matching the table's field names`. It is now rejected by an explicit check, with a message naming the source. Without that check, the missing columns would never be compared and would be written as null. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
