JingsongLi commented on PR #1390: URL: https://github.com/apache/paimon-rust/pull/1390#issuecomment-6905966771
Requirement fit: PIVOT. Implementation: CLEAN in the inspected remapping paths. Closing this implementation because it does not demonstrate a useful reduction on the default input-batching path for this production review. The retention problem in #1386 remains valid. I reviewed the cursor/output-index remapping and ran all 42 sort-merge tests successfully. I also reproduced the multi-row case raised in the earlier review, using the exact same lazy ownership probe in separate worktrees at base f91fdb8a and head 70bac76d: - Two streams of 1024-row input batches (the default read batch size). - Keys 0..1000 survive; keys 1000..204800 have a later DELETE. - Output batch size 1024, so 1000 surviving rows remain pending. - Correct output is preserved, but **both base and PR peak at 400 live source arrays out of 400 generated batches**. These are Weak-reference observations; the probe does not retain the input itself. The trigger uses the number of pending rows, so its bound is about 2004 buffer entries even though those 1000 rows reference very few source batches. The reader therefore keeps all the unrelated decoded data in this ordinary-sized example. The one-row-batch regression demonstrates a narrow improvement but does not establish the intended production memory benefit; later reclamation at a much larger entry count can still exceed a practical read budget. Please revise the trigger around actual live/consumed source batches (while accounting separately for materialized merge rows), and demonstrate lower peak ownership with multi-row batches plus unchanged values/order. Verify the deduplicate, aggregation, partial-update and lookup paths without introducing the substantial read regressions discussed above. A focused revision with that evidence can be reopened/reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
