[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] Xu Zhang edited comment on SPARK-21701 at 8/11/17 2:23 PM: --- Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding SO_SNDBUF param (although we can set the value in OS, if no set, use default value), then peers may use a relatively small "sliding window" for later communication. Thus the param set in SparkConf does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side deals with these params from SparkConf. I have already created a PR for this update and some refinement work, for more details please visit https://github.com/apache/spark/pull/18922. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding SO_SNDBUF param (although we can set the value in OS, if no set, use default value), then peers may use a relatively small "sliding window" for later communication. Thus the param set in SparkConf does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. I have already created a PR for this update and some refinement work, for more details please visit https://github.com/apache/spark/pull/18922. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: Xu Zhang >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] Xu Zhang edited comment on SPARK-21701 at 8/11/17 2:22 PM: --- Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding SO_SNDBUF param (although we can set the value in OS, if no set, use default value), then peers may use a relatively small "sliding window" for later communication. Thus the param set in SparkConf does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. I have already created a PR for this update and some refinement work, for more details please visit https://github.com/apache/spark/pull/18922. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding SO_SNDBUF param (although we can set the value in OS, if no set, use default value), then peers may use a relatively small "sliding window" for later communication. Thus the param set in SparkConf does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: Xu Zhang >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] Xu Zhang edited comment on SPARK-21701 at 8/11/17 9:53 AM: --- Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding SO_SNDBUF param (although we can set the value in OS, if no set, use default value), then peers may use a relatively small "sliding window" for later communication. Thus the param set in SparkConf does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: Xu Zhang >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] Xu Zhang edited comment on SPARK-21701 at 8/11/17 9:50 AM: --- Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side does not take effect in transport phase as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side is useless and does not take effect as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: Xu Zhang >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] neoremind edited comment on SPARK-21701 at 8/11/17 9:46 AM: Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side is useless and does not take effect as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark RPC module is not for high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label. In a word, my point is it would be better to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side is useless and does not take effect as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark is not high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label and lower priority. In a word, my point is to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[ https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096 ] neoremind edited comment on SPARK-21701 at 8/11/17 9:44 AM: Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side is useless and does not take effect as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark is not high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label and lower priority. In a word, my point is to keep an entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks was (Author: xu.zhang): Hi Sean, Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in server side, the two parameters are specified in {code:java} org.apache.spark.network.server.TransportServer {code} through SparkConf. If server side specify a bigger SO_RCVBUF size, while client does not have any place to set the corresponding param (although we can set the value in OS, if no set, use default value), then peers may set a relatively small sliding window size for later communication. Thus the param set in SparkConf in server side is useless and does not take effect as expected. To achieve consistency in both client and server side, enable client to get these params from SparkConf would make sense. Moreover, due to the fact that spark is not high throughput and performant C/S service system, it should not to be a big problem, so I set the ticket to improvement label and lower priority. In a word, my point is to keep a entrance to the outside world to set these params in client side and keep consistent with how server side specifies these params. Thanks > Add TCP send/rcv buffer size support for RPC client > --- > > Key: SPARK-21701 > URL: https://issues.apache.org/jira/browse/SPARK-21701 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > > For TransportClientFactory class, there are no params derived from SparkConf > to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. > Increasing the receive buffer size can increase the I/O performance for > high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org