otterc commented on a change in pull request #32031: URL: https://github.com/apache/spark/pull/32031#discussion_r681252225
########## File path: remote-shuffle-service/src/main/java/org/apache/spark/remoteshuffle/StreamServerConfig.java ########## @@ -0,0 +1,366 @@ +/* + * This file is copied from Uber Remote Shuffle Service +(https://github.com/uber/RemoteShuffleService) and modified. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * http://www.apache.org/licenses/LICENSE-2.0 + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.remoteshuffle; + +import org.apache.spark.remoteshuffle.clients.ClientConstants; +import org.apache.spark.remoteshuffle.execution.ShuffleExecutor; +import org.apache.spark.remoteshuffle.handlers.UploadChannelManager; +import org.apache.spark.remoteshuffle.metadata.ServiceRegistry; +import org.apache.spark.remoteshuffle.storage.ShuffleFileStorage; +import org.apache.spark.remoteshuffle.storage.ShuffleStorage; +import org.apache.commons.lang3.StringUtils; +import org.apache.hadoop.conf.Configuration; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.concurrent.TimeUnit; + +public class StreamServerConfig { + public static final long DEFAULT_SERVER_SIDE_CONNECTION_IDLE_TIMEOUT_MILLIS = 2 * 60 * 60 * 1000; + + public static final String DEFAULT_DATA_CENTER = "dataCenter1"; + + private boolean useEpoll = false; + + private int shufflePort = 19190; + + private int httpPort = -1; + + private String rootDir = ""; + + // number of threads for netty to accept incoming socket + private int nettyAcceptThreads = 2; Review comment: Curious to know why you have chosen number of threads in the boss group to be 2? Do you also run with this default in your production cluster (please ignore if this information cannot be shared)? I am asking because in the ESS the boss event group has just 1 thread and this isn't configurable. Is that something we should change for the ESS? I looked at the boss event-loop group a while back and found the threads in this group are responsible for accepting connection and assigning a worker thread to handle it (read and write to the socket connection). In general worker threads work for more time than the boss thread. So, did you face any issues with number of threads in this boss group being 1? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
