attilapiros commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#discussion_r346258949
########## File path: common/network-common/src/main/java/org/apache/spark/network/client/AsyncResponseCallback.java ########## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.network.client; + +/** + * Callback for the result of a single call. + * This will be invoked once with either success or failure. + */ +public interface AsyncResponseCallback<T> { Review comment: I prefer this solution because the asynchronous access of local dirs for executors not just goes trough multiple classes but also the asynchronous result itself is used in multiple places. It starts from https://github.com/apache/spark/blob/eafeb478aab411b3517b1e2b4b66099225c4b8c9/core/src/main/scala/org/apache/spark/storage/ShuffleBlockFetcherIterator.scala#L492 and which goes to https://github.com/apache/spark/blob/eafeb478aab411b3517b1e2b4b66099225c4b8c9/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L142 Both places are using the asynchronous result: the first one to access the blocks the second one for caching the directories. With `AsyncResponseCallback[java.util.Map[String, Array[String]]]` I can avoid decoding the message again and again and it is more type safe (I would even say typed) as with `RpcResponseCallback` there is only just a simple `ByteBuffer` which after decoding needs to be casted to the exact `BlockTransferMessage` to get the directories. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
