jihoonson commented on issue #8249: ability to let user configure segment version in indexing task URL: https://github.com/apache/incubator-druid/issues/8249#issuecomment-518942700 Hmm, I understand it could be useful for your use case if custom segment version is allowed, but I still think it would be better if the functionality supported by Druid is more organized and safe. I'm wondering if it's possible to achieve the same goal with a new supervisor (not supervisor task). This supervisor coordinates the completion order of incoming tasks. I guess the completion order might be defined in a various way in the future, but we can start with a simple one like FIFO. So, you may want to submit all indexing tasks to this supervisor instead of directly submitting them to the overlord. The supervisor guarantees first-issued tasks are finished first, which means the segments created by later tasks will have higher versions. I think this could be implemented quite easily with a simple restriction, i.e., the `intervals` parameter should be specified in the `granularitySpec`. If `intervals` of a new task are overlapped with those of any other tasks, then that new task will wait for those tasks to be finished first. If a task fails, the supervisor will retry it until it succeeds or the number of failures exceeds a configured limit. With this supervisor, your tasks can still be run in parallel if their intervals are not overlapped. Tasks of overlapping intervals can't be run at the same time, but I think this should be same in the custom segment version approach. What do you think?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
