npawar commented on pull request #7605: URL: https://github.com/apache/pinot/pull/7605#issuecomment-975952506
> @npawar If a segment is `CONSUMING` or `DONE`, this check is already being performed [here](https://github.com/apache/pinot/blob/1bd899c9ba45676d1ac25979274391431bdf5ce9/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java#L132), i.e., at the beginning of `RealtimeToOfflineSegment` task generation. > > [This check](https://github.com/apache/pinot/blob/1bd899c9ba45676d1ac25979274391431bdf5ce9/pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/realtimetoofflinesegments/RealtimeToOfflineSegmentsTaskGenerator.java#L187) also ensures that a `CONSUMING` segment is not processed in `RealtimeToOfflineSegmentTask`. > > This PR only progresses the watermark if no segment is found for those segments marked as `DONE` in our processing window. > > The solution you are proposing will not work if the segments in the next window are still in `CONSUMING` state at time of progressing the watermark. > > Please let me know your thoughts on all of the above. I agree. Your logic seems to work. Please add a test case, I think it will not actually update zk so it may of be working as of yet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
