zn-danielbariudin commented on issue #3860: URL: https://github.com/apache/iceberg-python/issues/3860#issuecomment-6742446620
I can reproduce this (pyiceberg main at e9f7f6b4, Python 3.14.7, pyarrow 26.0.1, Apple M5 Pro): 20,000 rows × 200 columns takes ~5.1 s in get_rows_to_update. Timing the steps separately, cast + join takes 0.03 s and the per-row Python comparison loop takes 4.6 s, so the loop is ~99% of the time. I’d like to take this. Plan: compare matched rows column by column with Arrow compute (take both sides by the join indices, then compare), keep the current per-cell path as a fallback for types Arrow can’t compare (list/struct/map), and keep today’s behavior for nulls (null == null) and NaN (NaN != NaN). I’ll add tests for those cases first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
