amaechler opened a new issue, #20773: URL: https://github.com/apache/druid/issues/20773
### Affected Version `master` since #20740 (63e3c24). Before #20740, the task could launch instead; see #20298. ### Description Follow-up to #20740. If `TaskQueue` kills a task while `HttpRemoteTaskRunner` is assigning it to a worker, and the assignment then fails, the task never launches but stays pending: - `getPendingTasks()` and `getPendingTaskPayloads()` still list it, so `PendingTaskBasedWorkerProvisioningStrategy` counts it as demand. - Its result future stays open. This lasts until `TaskQueue` calls `shutdown()` again in the management pass after its next storage sync, 1 to 2 `druid.indexer.queue.storageSyncRate` periods (1 minute by default) later. That call completes the task as FAILED. The assignment fails this way only when the chosen worker drops out right after it is picked, so this is rare. HTTP errors don't cause it, because `assignTask()` [returns true when unsure](https://github.com/apache/druid/blob/63e3c2422df8a67435de1cd47d2651f2434a35b4/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/WorkerHolder.java#L226-L229). If the assignment succeeds instead, the killed task runs on the worker until that same `shutdown()` call kills it. The reason for this: 1. For a `PENDING_WORKER_ASSIGN` work item, `shutdown()` only [removes it from `pendingTaskIds`](https://github.com/apache/druid/blob/63e3c2422df8a67435de1cd47d2651f2434a35b4/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L1263-L1266), because the assigning thread owns the item. 2. When the worker turns out to be [removed, lazy or blacklisted](https://github.com/apache/druid/blob/63e3c2422df8a67435de1cd47d2651f2434a35b4/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L316-L319), or [disabled](https://github.com/apache/druid/blob/63e3c2422df8a67435de1cd47d2651f2434a35b4/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/WorkerHolder.java#L177-L184), the assigning thread [reverts the item to PENDING](https://github.com/apache/druid/blob/63e3c2422df8a67435de1cd47d2651f2434a35b4/indexing-service/src/main/java/org/apache/druid/indexing/overlord/hrtr/HttpRemoteTaskRunner.java#L1190-L1193). Nothing records the kill, so the item is neither assigned again nor completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
