swaminathanmanish opened a new pull request, #10630: URL: https://github.com/apache/pinot/pull/10630
**Problem** Segment replacement has 2 phases (start, end). The start phase takes in the "**from**" and "**to**" list of segments, to create a lineage record that's used in the the end phase. The end phase expects all segments in the "**to**" to be available. However it can happen that only a subset of segments get created between the start and end phase. **Usecase** - While ingesting batch of files via minion tasks, we use segment replacement to do a global atomic switch to avoid inconsistencies during ingestion. We start with a list of "**to**" segments, however can end up with a subset of the original "to" segments list (eg: files can have empty records). Today the code has validations when segments in the original "to" list are not available, during the end phase of the protocol. **Solution** Allow the endReplaceSegments API take in the list of "**to**" segments that actually gets generated/created. Validate that this list is a subset of the original to List. **Tests** 1) Tested for backwards compatibility (MergeRollupMinionClusterIntegrationTest) 2) Unit tests 3) Used Swagger to verify that the new parameter is taken by the API -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
