wangzhigang1999 opened a new issue, #10142: URL: https://github.com/apache/paimon/issues/10142
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation PyPaimon reconstructs MAP values when reading ordinary MAP columns and shared-shredding MAP columns. These paths convert Arrow arrays to Python objects and process mapping entries or keys row by row. For large batches, this introduces CPU overhead even after unrelated columns have been pruned. The same module contains three reconstruction paths with opportunities to reduce this overhead. This umbrella issue tracks them as separate changes so that each can be reviewed and benchmarked independently. ### Solution - [ ] Optimize `assemble_shared_shredding_selected_keys`: use Arrow operations to resolve requested keys and select values, with scalar fallback for small batches and expensive candidate mappings. - [ ] Optimize `assemble_normal_map_selected_keys`: use Arrow key lookup to avoid converting and searching each row's full MAP in Python where supported. - [ ] Optimize `assemble_shared_shredding_map`: batch the integer field-mapping conversion while preserving MAP entry order and value handling. Preserve null handling, missing-key behavior, sliced-array offsets, and the existing selection and ordering semantics. Keep fallback paths where Arrow operations are unavailable or do not improve performance. ### Anything else? Submit a separate PR for each item, with correctness coverage and before/after benchmarks. These changes concern read-side reconstruction and require no file-format or public API change. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
