clintropolis commented on a change in pull request #8173: Add a cluster-wide configuration to force timeChunk lock and add a doc for segment locking URL: https://github.com/apache/incubator-druid/pull/8173#discussion_r308908873
########## File path: docs/content/ingestion/locking-and-priority.md ########## @@ -24,30 +24,73 @@ title: "Task Locking & Priority" # Task Locking & Priority +This document explains the task locking system in Druid. Druid's locking system +and versioning system are tighly coupled with each other to guarantee the correctness of ingested data. + +## Overshadow Relation between Segments + +You can run a task to overwrite existing data. The segments created by an overwriting task _overshadows_ existing segments. +Note that the overshadow relation holds only for the same time chunk and the same data source. +These overshadowed segments are not considered in query processing to filter out stale data. + +A segment `s1` can overshadow another `s2` if + +- `s1` has a higher major version than `s2`. +- `s1` has the same major version and a higher minor version than `s2`. + +Here are some examples. + +- A segment of the major version of `2019-01-01T00:00:00.000Z` and the minor version of `0` overshadows + another of the major version of `2018-01-01T00:00:00.000Z` and the minor version of `1`. +- A segment of the major version of `2019-01-01T00:00:00.000Z` and the minor version of `1` overshadows + another of the major version of `2019-01-01T00:00:00.000Z` and the minor version of `0`. + ## Locking -Once an Overlord process accepts a task, the task acquires locks for the data source and intervals specified in the task. +If you are running two or more [druid tasks](./tasks.html) which generate segments for the same data source and the same time chunk, +the generated segments could potentially overshadow each other which could lead to incorrect query results. -There are two lock types, i.e., _shared lock_ and _exclusive lock_. +To avoid this problem, tasks should get locks first before creating any segment in Druid. Review comment: suggest: suggest rewording to avoid making it sound like getting the locks is something the user/operator has to do > To avoid this problem, tasks will attempt to get locks prior to creating any segment in Druid. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
