swaminathanmanish opened a new pull request, #12255: URL: https://github.com/apache/pinot/pull/12255
**Description**: This PR makes generateSegmentMetadataFile accessible so as to generate metadata file. This enables us to create metadata file, push it to deep store, so that we only need to download the metadata file during segment metadata creation. With this, we dont need to download the entire segment to extract metadata, if we already have segment metadata in deep store. **Why the above is needed:** This is useful for usecases where we want to decouple segment push and metadata push, in order to use segment replacement protocol ([consistent push](https://docs.pinot.apache.org/operators/operating-pinot/consistent-push-and-rollback)). Segment generation phase creates segments, uploads the segment and metadata to deep store. When all segments have been generated, segment replacement protocol is initiated which is when we also kick off metadata push to create segment metadata (also when segments are downloaded in servers). At this time, we want to download only the metadata file instead of the entire segment to extract metadata. This [PR](https://github.com/apache/pinot/pull/10034) already enabled metdata push logic to download only the segment metadata is job spec has **_preferMetadataTarGz** set to true. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
