JingsongLi commented on PR #10152: URL: https://github.com/apache/paimon/pull/10152#issuecomment-5805488241
Requirement fit: SUPPORTED. Forwarding Paimon's immutable data-file size avoids redundant object-store metadata work on the Parquet/Jindo read path, with a bounded handler cache and no format change. Implementation: CLEAN in the changed path. I traced DataFileMeta.file_size through split read, FormatPyArrowReader, PyArrow fragment creation, and Jindo's seek/read handling. Unknown sizes and PyArrow < 7 retain the prior behavior; the metadata cache key includes the known size. I found no actionable regression. Verification on an isolated checkout (Python 3.13, PyArrow 24): the two focused test modules passed (31 passed, 19 Jindo-dependent skips), Parquet row-range/page-index tests passed (89), and selected append-only/primary-key/reader tests passed (45, with optional Lance/Vortex and ORC paths excluded). A broader run hit a PyArrow ORC sysctlbyname error and an optional Vortex crash in this macOS environment; I did not reproduce the reported Jindo OSS trace. The local source base is older than the PR base, so current-head CI remains the final integration check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
