laxmanmamididd opened a new issue, #3836:
URL: https://github.com/apache/celeborn/issues/3836
## Environment
- Celeborn 0.6.3 (server), Spark 4.0.1 client
(`celeborn-client-spark-4-shaded_2.13-0.6.3`)
- Worker config: `celeborn.storage.availableTypes=SSD,S3`,
`celeborn.worker.storage.storagePolicy.createFilePolicy=SSD,S3`,
`celeborn.worker.storage.storagePolicy.evictPolicy=SSD,S3`
- Only one local disk configured per worker (`disktype=SSD`), no HDD
directories configured anywhere.
- 4 workers, `celeborn.client.shuffle.replication.factor=2`
## What's happening
Under a real ~200GB shuffle load (`org.apache.spark.examples.GroupByTest`),
every worker eventually throws:
```
ERROR StorageManager: Create partition data writer failed
java.lang.IndexOutOfBoundsException: -1
at
scala.collection.LinearSeqOptimized.apply(LinearSeqOptimized.scala:67)
at scala.collection.immutable.List.apply(List.scala:91)
at
org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.$anonfun$createFileWriter$6(StoragePolicy.scala:197)
at
org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.createFileWriter(StoragePolicy.scala:196)
at
org.apache.celeborn.service.deploy.worker.storage.StoragePolicy.createFileWriter(StoragePolicy.scala:60)
at
org.apache.celeborn.service.deploy.worker.storage.PartitionDataWriter.<init>(PartitionDataWriter.java:91)
at
org.apache.celeborn.service.deploy.worker.storage.StorageManager.createPartitionDataWriter(StorageManager.scala:477)
at
org.apache.celeborn.service.deploy.worker.Controller.createPartitionDataWriter(Controller.scala:344)
at
org.apache.celeborn.service.deploy.worker.Controller.createWriter$1(Controller.scala:290)
at
org.apache.celeborn.service.deploy.worker.Controller.$anonfun$createWriters$1(Controller.scala:293)
...
at
org.apache.celeborn.service.deploy.worker.Controller.org$apache$celeborn$service$deploy$worker$Controller$$handleReserveSlots(Controller.scala:210)
```
The full log line for the failing `PartitionLocation` shows:
```
Controller: Create FileWriter for PartitionLocation[
id-epoch:5-1
host-rpcPort-pushPort-fetchPort-replicatePort:[...]
mode:PRIMARY
...
storage hint:StorageInfo{type=HDD, mountPoint='', finalResult=false,
filePath=, fileSize=0, chunkOffsets=[]}
...] celeborn-data03-coldtier-fill5-<id>-0-0 failed.
```
Note the `id-epoch:5-1` — this is epoch 1, i.e. a **revived** partition
location (not the initial epoch-0 allocation), and its `storage hint` reports
`type=HDD`.
## Why this crashes
`StoragePolicy.createFileWriter` (the non-`evict` overload, called via
`PartitionDataWriter`'s constructor →
`StorageManager.createPartitionDataWriter`) computes:
```scala
val tryCreateFileTypeIndex = order.get.indexOf(
partitionDataWriterContext.getPartitionLocation.getStorageInfo.getType.name())
```
Since `order` is our configured create-file order (`["SSD", "S3"]`) and the
assigned type is `"HDD"`, `indexOf` returns `-1`, and the subsequent
`order.get(tryCreateFileTypeIndex)` (`StoragePolicy.scala:197`) throws
`IndexOutOfBoundsException: -1`.
## The actual question
**Why does a revived `PartitionLocation` get assigned storage type `HDD`**
when nothing in the cluster's config uses HDD anywhere? `StorageInfo`'s own
no-arg constructor defaults `type` to `MEMORY`, not `HDD`, so this isn't a
generic uninitialized-field default — something in the revive/reallocation path
(looks like it should be in `SlotsAllocator.getStorageInfo` or the master's
revive handler, but I wasn't able to pin down the exact assignment) appears to
be defaulting to `HDD` specifically for revived locations, independent of what
disk types the target worker(s) actually have configured.
This reproduces reliably (100% of large-shuffle runs so far) once enough
map/reduce tasks trigger at least one push-retry/revive cycle. Small jobs (a
handful of reducers, little data) never hit it.
Happy to provide more repro detail, full worker/master logs, or test against
a patch if pointed at the right code path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
