minhduc29 opened a new pull request, #3611: URL: https://github.com/apache/iceberg-python/pull/3611
Closes #3598 # Rationale for this change `upsert` compares every non-key column of each matched row in Python to decide whether it needs updating, which is expensive for wide tables or complex types (structs, lists). This adds an optional `difference_cols` parameter to `Table.upsert` / `Transaction.upsert` / `get_rows_to_update` that limits the comparison to a user-provided subset of columns (e.g. a hash column that reflects any change to the row), as proposed in #3598. Design notes: - `difference_cols` only affects change *detection*: rows detected as changed are still written with all of their columns. - Columns missing from the source table, join columns, or an empty list raise `ValueError`, validated up front in `Transaction.upsert` to fail fast. The issue proposed either raising or falling back to comparing all columns — raising seemed safer, but happy to change if the fallback is preferred. - Behavior without the parameter is unchanged. ## Are these changes tested? Yes. Unit tests cover detection limited to the listed columns (a matched row changed only outside `difference_cols` is skipped), updated rows being written with all columns, the three validation errors, and an end-to-end upsert via the in-memory catalog. ## Are there any user-facing changes? Yes, a new optional `difference_cols` parameter on `Table.upsert` and `Transaction.upsert`, documented in the API docs. No behavior change when it is not provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
