swaminathanmanish commented on PR #10630:
URL: https://github.com/apache/pinot/pull/10630#issuecomment-1513576438

   > @swaminathanmanish For me to understand the use case, is it possible to 
avoid generating empty files in your offline data source? I feel like it would 
be cleaner to avoid generating empty files instead of changing the protocol.
   
   Sure, for our ingestion usecase, we want to do a global atomic switch 
(fromSegments to toSegments), for data consistency reasons. We plan to do this 
in the Minion task generator, where we have the old & new segment names, since 
we use fixed name generator. We initiate the "start phase" when tasks are 
created and "end phase" after all tasks have been executed, in the generator. 
However the files are read only in the executor and can be empty (empty rows). 
Hence we need to be able to provide a subset of toSegments for the end phase. 
Note that this is a subset from the original toSegment list.
   
   Could you describe your concerns about this change ? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to