wangzhigang1999 opened a new issue, #10465: URL: https://github.com/apache/paimon/issues/10465
### Search before asking - [x] I searched in the issues and found nothing similar. ### Motivation In `_AsOfJoinRight.plan()`, we build a dictionary mapping each group key to its row range in the sorted right-side metadata. We currently convert Arrow scalars to Python values and construct a tuple for every row, even when many rows belong to the same group. As-of joins, linear interpolation and window joins share this planning code. For example, with one million right-side rows across 100 groups, we perform one million Python iterations to produce 100 dictionary entries. In a local warm-cache benchmark, replacing this loop reduced end-to-end as-of latency from 1220.7 ms to 406.8 ms. The same query over OSS went from 1788.8 ms to 979.4 ms. These measurements include planning, reading and result materialization. ### Solution Use Arrow `run_end_encode` to find group boundaries in the already sorted metadata when grouping by a single supported column. Convert the run values and ends to Python in bulk, then populate the existing dictionary once per group. Limit the fast path to integer, boolean, string/large-string and binary/large-binary keys. Keep the existing loop for other types, composite keys, missing APIs and unsupported kernels. Floating-point keys need the existing equality semantics: Arrow run-end encoding distinguishes `-0.0` and `+0.0`, while Python dictionary keys treat them as equal. ### Anything else? The measurements compare commit `9c3c5187a4` with the proposed optimization, using Python 3.11.13 and PyArrow 19.0.1. Each query had one warm-up and three timed runs; the figures above are medians. Small tables showed no stable benefit. In a 200,000-row case with one group per row, peak RSS increased by 6.6–8.7 MiB across the three temporal operations. Cold storage and concurrent queries remain unmeasured. This proposal concerns the existing PyPaimon temporal APIs. It requires no change to the table format or the public API. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
