jihoonson commented on issue #8249: ability to let user configure segment version in indexing task URL: https://github.com/apache/incubator-druid/issues/8249#issuecomment-519262740 Yes, I don't see a simple and better alternative. For this kind of feature, I personally prefer to document with a scary warning with details. > @jihoonson I see that there has been a lot of code changes in locking etc and I haven't gone through them yet, do you think this would be doable in a central place or needs to be handled per task ? So, you don't have to roll up and it will make things a bit simpler. I think the part you may want to modify will depend on what lock type you want to use. If a task is using the time chunk lock, it can assign the segment version by itself ([LocalSegmentAllocator for local indexing](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/LocalSegmentAllocator.java) and [ParallelSupervisorTask for parallel indexing](https://github.com/apache/incubator-druid/blob/master/indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ParallelIndexSupervisorTask.java#L462)). If it's using the segment lock, the segment version is assigned by [the overlord in a centralized manner](https://github.com/apache/incubator-druid/blob/master/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L344). This is because the major version of the new segment is determined based on the existing segments. With the segment lock, all your tasks can run at the same time unless they are overwriting existing segments. For segment lock, you may want to set custom minor versions. With the time chunk lock, tasks can run at the same time only if their intervals are not overlapped. For time chunk lock, you may want to set custom major versions.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
