writer-jill commented on code in PR #13503: URL: https://github.com/apache/druid/pull/13503#discussion_r1041186267
########## docs/ingestion/tasks.md: ########## @@ -343,6 +343,27 @@ You can override the task priority by setting your priority in the task context "priority" : 100 } ``` +<a name="actions"></a> + +## Task actions + +These are various overlord actions performed by tasks during their lifecycle. Some typical actions are as follows: +- `lockAcquire`: acquires a time-chunk lock on an interval for the task +- `lockRelease`: releases a lock acquired by the task on an interval +- `segmentTransactionalInsert`: publishes new segments created by a task and optionally overwrites and/or drops existing segments in a single transaction +- `segmentAllocate`: allocates pending segments to a task to write rows +- etc. + +### Batching `segmentAllocate` actions + +In a cluster with several concurrent tasks, `segmentAllocate` actions on the overlord may take very long intervals of time to finish thus causing spikes in the `task/action/run/time`. This may result in ingestion lag building up while a task waits for a segment to get allocated. +The root causes of such spikes are: +- several concurrent tasks trying to allocate segments for the same datasource and interval +- large number of metadata calls made to the segments and pending segments tables +- concurrency limitations while acquiring a task lock required for allocating a segment + +Since the contention typically arises from tasks allocating segments for the same datasource and interval, the run times can be improved by batching the actions together. +Batched segment allocation can be enabled on the overlord by setting `druid.indexer.tasklock.batchSegmentAllocation=true`.See [overlord configuration](../configuration/index.md#overlord-operations) for more details. Review Comment: ```suggestion To enable batched segment allocation on the overlord, set `druid.indexer.tasklock.batchSegmentAllocation` to `true`. See [overlord configuration](../configuration/index.md#overlord-operations) for more details. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
