[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[
https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096
]
neoremind edited comment on SPARK-21701 at 8/11/17 9:46 AM:
Hi Sean,
Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in
server side, the two parameters are specified in
{code:java}
org.apache.spark.network.server.TransportServer
{code}
through SparkConf. If server side specify a bigger SO_RCVBUF size, while client
does not have any place to set the corresponding param (although we can set the
value in OS, if no set, use default value), then peers may set a relatively
small sliding window size for later communication. Thus the param set in
SparkConf in server side is useless and does not take effect as expected. To
achieve consistency in both client and server side, enable client to get these
params from SparkConf would make sense. Moreover, due to the fact that spark
RPC module is not for high throughput and performant C/S service system, it
should not to be a big problem, so I set the ticket to improvement label.
In a word, my point is it would be better to keep an entrance to the outside
world to set these params in client side and keep consistent with how server
side specifies these params.
Thanks
was (Author: xu.zhang):
Hi Sean,
Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in
server side, the two parameters are specified in
{code:java}
org.apache.spark.network.server.TransportServer
{code}
through SparkConf. If server side specify a bigger SO_RCVBUF size, while client
does not have any place to set the corresponding param (although we can set the
value in OS, if no set, use default value), then peers may set a relatively
small sliding window size for later communication. Thus the param set in
SparkConf in server side is useless and does not take effect as expected. To
achieve consistency in both client and server side, enable client to get these
params from SparkConf would make sense. Moreover, due to the fact that spark is
not high throughput and performant C/S service system, it should not to be a
big problem, so I set the ticket to improvement label and lower priority.
In a word, my point is to keep an entrance to the outside world to set these
params in client side and keep consistent with how server side specifies these
params.
Thanks
> Add TCP send/rcv buffer size support for RPC client
> ---
>
> Key: SPARK-21701
> URL: https://issues.apache.org/jira/browse/SPARK-21701
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.6.0
>Reporter: neoremind
>Priority: Trivial
>
> For TransportClientFactory class, there are no params derived from SparkConf
> to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty.
> Increasing the receive buffer size can increase the I/O performance for
> high-volume transport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[
https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096
]
neoremind edited comment on SPARK-21701 at 8/11/17 9:44 AM:
Hi Sean,
Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in
server side, the two parameters are specified in
{code:java}
org.apache.spark.network.server.TransportServer
{code}
through SparkConf. If server side specify a bigger SO_RCVBUF size, while client
does not have any place to set the corresponding param (although we can set the
value in OS, if no set, use default value), then peers may set a relatively
small sliding window size for later communication. Thus the param set in
SparkConf in server side is useless and does not take effect as expected. To
achieve consistency in both client and server side, enable client to get these
params from SparkConf would make sense. Moreover, due to the fact that spark is
not high throughput and performant C/S service system, it should not to be a
big problem, so I set the ticket to improvement label and lower priority.
In a word, my point is to keep an entrance to the outside world to set these
params in client side and keep consistent with how server side specifies these
params.
Thanks
was (Author: xu.zhang):
Hi Sean,
Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in
server side, the two parameters are specified in
{code:java}
org.apache.spark.network.server.TransportServer
{code}
through SparkConf. If server side specify a bigger SO_RCVBUF size, while client
does not have any place to set the corresponding param (although we can set the
value in OS, if no set, use default value), then peers may set a relatively
small sliding window size for later communication. Thus the param set in
SparkConf in server side is useless and does not take effect as expected. To
achieve consistency in both client and server side, enable client to get these
params from SparkConf would make sense. Moreover, due to the fact that spark is
not high throughput and performant C/S service system, it should not to be a
big problem, so I set the ticket to improvement label and lower priority.
In a word, my point is to keep a entrance to the outside world to set these
params in client side and keep consistent with how server side specifies these
params.
Thanks
> Add TCP send/rcv buffer size support for RPC client
> ---
>
> Key: SPARK-21701
> URL: https://issues.apache.org/jira/browse/SPARK-21701
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.6.0
>Reporter: neoremind
>Priority: Trivial
>
> For TransportClientFactory class, there are no params derived from SparkConf
> to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty.
> Increasing the receive buffer size can increase the I/O performance for
> high-volume transport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
[
https://issues.apache.org/jira/browse/SPARK-21701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123096#comment-16123096
]
neoremind commented on SPARK-21701:
---
Hi Sean,
Thanks for your quick response. SO_RCVBUF and SO_SNDBUF are for TCP, and in
server side, the two parameters are specified in
{code:java}
org.apache.spark.network.server.TransportServer
{code}
through SparkConf. If server side specify a bigger SO_RCVBUF size, while client
does not have any place to set the corresponding param (although we can set the
value in OS, if no set, use default value), then peers may set a relatively
small sliding window size for later communication. Thus the param set in
SparkConf in server side is useless and does not take effect as expected. To
achieve consistency in both client and server side, enable client to get these
params from SparkConf would make sense. Moreover, due to the fact that spark is
not high throughput and performant C/S service system, it should not to be a
big problem, so I set the ticket to improvement label and lower priority.
In a word, my point is to keep a entrance to the outside world to set these
params in client side and keep consistent with how server side specifies these
params.
Thanks
> Add TCP send/rcv buffer size support for RPC client
> ---
>
> Key: SPARK-21701
> URL: https://issues.apache.org/jira/browse/SPARK-21701
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 1.6.0
>Reporter: neoremind
>Priority: Trivial
>
> For TransportClientFactory class, there are no params derived from SparkConf
> to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty.
> Increasing the receive buffer size can increase the I/O performance for
> high-volume transport.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
[ https://issues.apache.org/jira/browse/SPARK-21703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123071#comment-16123071 ] neoremind edited comment on SPARK-21703 at 8/11/17 9:18 AM: Thanks Sean to guide me to the right place. Here's the http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-CORE-Why-RPC-message-are-transferred-with-header-and-body-separately-in-TCP-frame-td29054.html. was (Author: xu.zhang): Thanks Sean to guide me to the right place. Here's the [link](http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-CORE-Why-RPC-message-are-transferred-with-header-and-body-separately-in-TCP-frame-td29054.html). > Why RPC message are transferred with header and body separately in TCP frame > > > Key: SPARK-21703 > URL: https://issues.apache.org/jira/browse/SPARK-21703 > Project: Spark > Issue Type: Question > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > Labels: RPC > > After seeing the details of how spark leverage netty, I found one question, > typically RPC message wire format would have a header+payload structure, and > netty uses a TransportFrameDecoder to deal with how to determine a complete > message from remote peer. But after using Wireshark sniffing tool, I found > that the message are sent separately with header and then followed by body, > although this works fine, but for underlying TCP there would be ACK segments > sent back to acknowledge, there might be a little bit redundancy since we can > sent them together and the header are usually very small. > The main reason can be found in MessageWithHeader class, since transferTo > method write tow times for header and body. > Could someone help me understand the background story on why to implement in > such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
[ https://issues.apache.org/jira/browse/SPARK-21703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123071#comment-16123071 ] neoremind commented on SPARK-21703: --- Thanks Sean to guide me to the right place. Here's the [link](http://apache-spark-user-list.1001560.n3.nabble.com/SPARK-CORE-Why-RPC-message-are-transferred-with-header-and-body-separately-in-TCP-frame-td29054.html). > Why RPC message are transferred with header and body separately in TCP frame > > > Key: SPARK-21703 > URL: https://issues.apache.org/jira/browse/SPARK-21703 > Project: Spark > Issue Type: Question > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > Labels: RPC > > After seeing the details of how spark leverage netty, I found one question, > typically RPC message wire format would have a header+payload structure, and > netty uses a TransportFrameDecoder to deal with how to determine a complete > message from remote peer. But after using Wireshark sniffing tool, I found > that the message are sent separately with header and then followed by body, > although this works fine, but for underlying TCP there would be ACK segments > sent back to acknowledge, there might be a little bit redundancy since we can > sent them together and the header are usually very small. > The main reason can be found in MessageWithHeader class, since transferTo > method write tow times for header and body. > Could someone help me understand the background story on why to implement in > such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
[ https://issues.apache.org/jira/browse/SPARK-21703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] neoremind updated SPARK-21703: -- Description: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then followed by body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could someone help me understand the background story on how to implement in such way? Thanks! was: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then followed by body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could some one help me understand the background story on how to implement in such way? Thanks! > Why RPC message are transferred with header and body separately in TCP frame > > > Key: SPARK-21703 > URL: https://issues.apache.org/jira/browse/SPARK-21703 > Project: Spark > Issue Type: Question > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > Labels: RPC > > After seeing the details of how spark leverage netty, I found one question, > typically RPC message wire format would have a header+payload structure, and > netty uses a TransportFrameDecoder to deal with how to determine a complete > message from remote peer. But after using Wireshark sniffing tool, I found > that the message are sent separately with header and then followed by body, > although this works fine, but for underlying TCP there would be ACK segments > sent back to acknowledge, there might be a little bit redundancy since we can > sent them together and the header are usually very small. > The main reason can be found in MessageWithHeader class, since transferTo > method write tow times for header and body. > Could someone help me understand the background story on how to implement in > such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
[ https://issues.apache.org/jira/browse/SPARK-21703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] neoremind updated SPARK-21703: -- Description: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then followed by body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could someone help me understand the background story on why to implement in such way? Thanks! was: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then followed by body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could someone help me understand the background story on how to implement in such way? Thanks! > Why RPC message are transferred with header and body separately in TCP frame > > > Key: SPARK-21703 > URL: https://issues.apache.org/jira/browse/SPARK-21703 > Project: Spark > Issue Type: Question > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > Labels: RPC > > After seeing the details of how spark leverage netty, I found one question, > typically RPC message wire format would have a header+payload structure, and > netty uses a TransportFrameDecoder to deal with how to determine a complete > message from remote peer. But after using Wireshark sniffing tool, I found > that the message are sent separately with header and then followed by body, > although this works fine, but for underlying TCP there would be ACK segments > sent back to acknowledge, there might be a little bit redundancy since we can > sent them together and the header are usually very small. > The main reason can be found in MessageWithHeader class, since transferTo > method write tow times for header and body. > Could someone help me understand the background story on why to implement in > such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
[ https://issues.apache.org/jira/browse/SPARK-21703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] neoremind updated SPARK-21703: -- Description: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then followed by body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could some one help me understand the background story on how to implement in such way? Thanks! was: After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then a body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could some one help me understand the background story on how to implement in such way? Thanks! > Why RPC message are transferred with header and body separately in TCP frame > > > Key: SPARK-21703 > URL: https://issues.apache.org/jira/browse/SPARK-21703 > Project: Spark > Issue Type: Question > Components: Spark Core >Affects Versions: 1.6.0 >Reporter: neoremind >Priority: Trivial > Labels: RPC > > After seeing the details of how spark leverage netty, I found one question, > typically RPC message wire format would have a header+payload structure, and > netty uses a TransportFrameDecoder to deal with how to determine a complete > message from remote peer. But after using Wireshark sniffing tool, I found > that the message are sent separately with header and then followed by body, > although this works fine, but for underlying TCP there would be ACK segments > sent back to acknowledge, there might be a little bit redundancy since we can > sent them together and the header are usually very small. > The main reason can be found in MessageWithHeader class, since transferTo > method write tow times for header and body. > Could some one help me understand the background story on how to implement in > such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21703) Why RPC message are transferred with header and body separately in TCP frame
neoremind created SPARK-21703: - Summary: Why RPC message are transferred with header and body separately in TCP frame Key: SPARK-21703 URL: https://issues.apache.org/jira/browse/SPARK-21703 Project: Spark Issue Type: Question Components: Spark Core Affects Versions: 1.6.0 Reporter: neoremind Priority: Trivial After seeing the details of how spark leverage netty, I found one question, typically RPC message wire format would have a header+payload structure, and netty uses a TransportFrameDecoder to deal with how to determine a complete message from remote peer. But after using Wireshark sniffing tool, I found that the message are sent separately with header and then a body, although this works fine, but for underlying TCP there would be ACK segments sent back to acknowledge, there might be a little bit redundancy since we can sent them together and the header are usually very small. The main reason can be found in MessageWithHeader class, since transferTo method write tow times for header and body. Could some one help me understand the background story on how to implement in such way? Thanks! -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21701) Add TCP send/rcv buffer size support for RPC client
neoremind created SPARK-21701: - Summary: Add TCP send/rcv buffer size support for RPC client Key: SPARK-21701 URL: https://issues.apache.org/jira/browse/SPARK-21701 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 1.6.0 Reporter: neoremind Priority: Trivial For TransportClientFactory class, there are no params derived from SparkConf to set ChannelOption.SO_RCVBUF and ChannelOption.SO_SNDBUF for netty. Increasing the receive buffer size can increase the I/O performance for high-volume transport. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
