XiaoHongbo-Hope opened a new pull request, #9314: URL: https://github.com/apache/paimon/pull/9314
## Purpose Ray self-merge already avoids the source-target join, but SET values are limited to literals and column references. This prevents row-local transformations such as updating existing VARIANT paths from being expressed in the distributed merge API. ## Changes - add Paimon predicate pushdown to the `_ROW_ID` self-merge path - add explicit `read_columns` projection for callable SET values - require callables to return one Arrow Array or ChunkedArray value per input row - preserve ChunkedArray boundaries while casting callable results - keep predicate, read columns, and callables restricted to self-merge; normal merge behavior is unchanged - document the callback contract and self-merge example This removes the source-target join for this workflow. Writer-side `_FIRST_ROW_ID` routing still uses the existing Ray shuffle and can be optimized independently with file-group affinity. ## Tests - `ray_data_evolution_merge_into_test.py`: 67 passed, 26 skipped - predicate pushdown and projection verification - end-to-end VARIANT path update - no-match, validation, and ChunkedArray coverage - flake8 and `git diff --check` passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
