nikhilred2619 commented on issue #55325: URL: https://github.com/apache/arrow/issues/55325#issuecomment-6558011839
Hi @bhavya-sl, I went through your proposal and the performance measurements. I can see the benefit of allowing Parquet fragments to reuse metadata that's already available, especially for large datasets where repeated footer reads add unnecessary overhead. I'd be happy to work on this, including the Python API changes and regression tests. Before starting, I'd like to confirm the preferred approach. Would you recommend accepting metadata when creating a fragment, or adding a method to attach metadata to an existing fragment? If the issue is available, I'm happy to implement the agreed approach and submit a PR for review. Thanks, Nikhil -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
