vanzin commented on a change in pull request #25299: [SPARK-27651][Core] Avoid the network when shuffle blocks are fetched from the same host URL: https://github.com/apache/spark/pull/25299#discussion_r346459236
########## File path: common/network-common/src/main/java/org/apache/spark/network/client/AsyncResponseCallback.java ########## @@ -0,0 +1,32 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.network.client; + +/** + * Callback for the result of a single call. + * This will be invoked once with either success or failure. + */ +public interface AsyncResponseCallback<T> { Review comment: Do this then. - Get rid of this interface. It doesn't really belong in network-common anyway. At worst it would belong in network-shuffle, but I think it's not needed. - Make `getHostLocalDirs` return a `java.util.concurrent.Future`, and internally use a `java.util.concurrent.CompletableFuture` to update it. You should also use `sendRpc` instead of `sendRpcSync` + a thread pool. That makes retries a little more complicated (but not much; `fetchBlocks` handles that, and in this case should be much simpler). Or, to simplify, you could just avoid retries altogether. How often will the call fail, really? And if it does, you just lose an optimization; you can try again next time there is a shuffle. On the Scala side, just wrap the Java future: ``` val scalaFuture = Future { javaFuture.get } ``` Or expose `CompletableFuture` and use its callback methods directly. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
