wangyum opened a new pull request, #57274:
URL: https://github.com/apache/spark/pull/57274
## What changes were proposed in this pull request?
When speculation is enabled, `dequeueTaskFromList` iterates through the
entire pending speculative task list and calls `hasAttemptOnHost` for each
entry — all while holding the taskSetLock. The current implementation does an
O(m) linear scan:
```scala
taskAttempts(taskIndex).exists(_.host == host)
```
This PR adds a lazily-allocated `HashSet[String]` per task index
(`taskAttemptHosts`) to track which hosts have attempted each task, enabling
O(1) lookup instead of O(m) linear scan.
## Why are the changes needed?
With 100K+ speculatable tasks and many attempts per task (due to failures or
prior speculation), the lock hold time grows to tens of seconds, blocking all
other make-offer threads and causing:
```
WARN [make-offer-thread-pool-4294] cluster.YarnScheduler:72 : Unable to lock
taskSet TaskSet_178603.1 after 5000ms, details:
org.apache.spark.scheduler.TaskSetLock@1a6daa67[Locked by thread
make-offer-thread-pool-4295]
at scala.collection.LinearSeqOptimized.exists 94
at scala.collection.LinearSeqOptimized.exists$ 92
at scala.collection.immutable.List.exists 91
at org.apache.spark.scheduler.TaskSetManager.hasAttemptOnHost 376
at org.apache.spark.scheduler.TaskSetManager.dequeueTaskFromList 357
```
## Benchmark results
(100K tasks, 1000 offers, while holding lock)
| Attempts/Task | Linear Scan (ms/offer) | HashSet (ms/offer) | Speedup |
|---|---|---|---|
| 1 | 352 | 466 | 0.8X |
| 10 | 2,655 | 2,246 | 1.2X |
| 50 | 17,114 | 3,455 | 5.0X |
| 100 | 30,640 | 3,758 | 8.2X |
At 100 attempts/task, lock hold time drops from ~30.6s to ~3.8s.
## Does this PR introduce any user-facing change?
No.
## How was this patch tested?
Added 4 unit tests in `TaskSetManagerSuite`:
- `hasAttemptOnHost uses taskAttemptHosts for O(1) host lookup` — verifies
speculation is blocked on hosts with prior attempts
- `hasAttemptOnHost tracks hosts across task retries` — verifies host
tracking persists across task failures
- `hasAttemptOnHost isolates host tracking per task` — verifies per-task
isolation with multiple speculatable tasks
- `hasAttemptOnHost works after dropTaskInfoAccumulablesOnTaskCompletion` —
verifies host tracking survives TaskInfo cloning
Closes #SPARK-58137
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
