eolivelli commented on code in PR #3608:
URL: https://github.com/apache/celeborn/pull/3608#discussion_r2838135502
##########
master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala:
##########
@@ -1019,9 +1019,12 @@ private[celeborn] class Master(
Utils.getSlotsPerDisk(slots.asInstanceOf[WorkerResource])
.asScala.map { case (worker, slots) => worker.toUniqueId -> slots
}.asJava,
requestSlots.requestId)
-
+ val primaryLocationsByType = slots.values.asScala
+ .flatMap(entry => entry._1.asScala) // ._1 extracts the primary location
+ .groupBy(l => l.getStorageInfo.getType)
+ .mapValues(locations => locations.size)
var offerSlotsMsg = s"Successfully offered slots for $numReducers reducers
of $shuffleKey" +
- s" on ${slots.size()} workers"
+ s" on ${slots.size()} workers, primary types: $primaryLocationsByType"
Review Comment:
when you have different spark jobs that select different storageTypes this
log line is very useful, to quickly check that the client is doing what it is
expected to do
##########
master/src/main/java/org/apache/celeborn/service/deploy/master/SlotsAllocator.java:
##########
@@ -260,8 +260,12 @@ private static StorageInfo getStorageInfo(
storageInfo = new StorageInfo("", StorageInfo.Type.S3,
availableStorageTypes);
} else if (StorageInfo.OSSAvailable(availableStorageTypes)) {
storageInfo = new StorageInfo("", StorageInfo.Type.OSS,
availableStorageTypes);
- } else {
+ } else if (StorageInfo.HDFSAvailable(availableStorageTypes)) {
storageInfo = new StorageInfo("", StorageInfo.Type.HDFS,
availableStorageTypes);
+ } else if (StorageInfo.memoryAvailable(availableStorageTypes)) {
Review Comment:
this change is not strictly needed by in some error conditions, when using
MEMORY storage, you see PartitionLocations assigned to HDFS, even if HDFS is
not configured.
I can move this to another PR, but I think it's relevant to this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
