[jira] [Commented] (FLINK-15032) Remove the eager serialization from `RemoteRpcInvocation`
[ https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17008617#comment-17008617 ] Guowei Ma commented on FLINK-15032: --- Thanks [~trohrmann] and [~SleePy] for your concerns. As [~SleePy] said there is no good way to check the size of load without serialization. For reducing the memory pressure we could only compute size of load but not save the serialization result. But it is only useful for the specific scenarios for example (large parallelism and network is limited). So I would close this issue now. > Remove the eager serialization from `RemoteRpcInvocation` > -- > > Key: FLINK-15032 > URL: https://issues.apache.org/jira/browse/FLINK-15032 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Guowei Ma >Priority: Major > > Currently, the constructor of `RemoteRpcInvocation` serializes the > `parameterTypes` and `arg` of an RPC call. This could lead to a problem: > Consider a job that has 1k parallelism and has a 1m union list state. When > deploying the 1k tasks, the eager serialization would use 1G memory > instantly(Some time the serialization amplifies the memory usage). However, > the serialized object is only used when the Akka sends the message. So we > could reduce the memory pressure if we only serialize the object when the > message would be sent by the Akka. > Akka would serialize the message at last and all the XXXGateway related class > could be visible by the RPC level. Because of that, I think the eager > serialization in the constructor of `RemoteRpcInvocation` could be avoided. I > also do a simple test and find this could reduce the time cost of the RPC > call. The 1k number of RPC calls with 1m `String` message: The current > version costs around 2700ms; the Nonserialization version cost about 37ms. > > In summary, this Jira proposes to remove the eager serialization at the > constructor of `RemoteRpcInvocation`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15032) Remove the eager serialization from `RemoteRpcInvocation`
[ https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16991225#comment-16991225 ] Biao Liu commented on FLINK-15032: -- [~trohrmann], thanks for explanation! I think you have raised a critical question. Besides the message size checking, the serialization checking might be more sticky. We might avoid the message size checking somehow, see the discussion under FLINK-4399. But I don't think we could do the same thing on serialization checking. It seems that this synchronous pre-checking can not be avoided, because some rpc invocations are "fire and forget". It's hard to tell invoker that there is something wrong with the rpc message without a synchronous pre-checking. Here are some rough ideas of mine. I think maybe we could do this optimization on the invocation without user-defined fields. I believe we could guarantee that the message is small and serializable in some scenarios. > Remove the eager serialization from `RemoteRpcInvocation` > -- > > Key: FLINK-15032 > URL: https://issues.apache.org/jira/browse/FLINK-15032 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Reporter: Guowei Ma >Priority: Major > > Currently, the constructor of `RemoteRpcInvocation` serializes the > `parameterTypes` and `arg` of an RPC call. This could lead to a problem: > Consider a job that has 1k parallelism and has a 1m union list state. When > deploying the 1k tasks, the eager serialization would use 1G memory > instantly(Some time the serialization amplifies the memory usage). However, > the serialized object is only used when the Akka sends the message. So we > could reduce the memory pressure if we only serialize the object when the > message would be sent by the Akka. > Akka would serialize the message at last and all the XXXGateway related class > could be visible by the RPC level. Because of that, I think the eager > serialization in the constructor of `RemoteRpcInvocation` could be avoided. I > also do a simple test and find this could reduce the time cost of the RPC > call. The 1k number of RPC calls with 1m `String` message: The current > version costs around 2700ms; the Nonserialization version cost about 37ms. > > In summary, this Jira proposes to remove the eager serialization at the > constructor of `RemoteRpcInvocation`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15032) Remove the eager serialization from `RemoteRpcInvocation`
[
https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988778#comment-16988778
]
Till Rohrmann commented on FLINK-15032:
---
Thanks for opening this issue [~maguowei]. The main reason why eager
serialization was introduced was to check if we are exceeding the maximum
framesize of Akka. If that's the case, then we fail with an explicit message
instead of dropping the message and logging a message as Akka does. There was
also the idea to offload large messages to the {{BlobService}} in order to
transmit large messages.
If we are able to maintain these properties while postponing the serialization,
then I'm fine with the proposal. But I would need a bit more details on how we
will achieve this.
> Remove the eager serialization from `RemoteRpcInvocation`
> --
>
> Key: FLINK-15032
> URL: https://issues.apache.org/jira/browse/FLINK-15032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Guowei Ma
>Priority: Major
>
> Currently, the constructor of `RemoteRpcInvocation` serializes the
> `parameterTypes` and `arg` of an RPC call. This could lead to a problem:
> Consider a job that has 1k parallelism and has a 1m union list state. When
> deploying the 1k tasks, the eager serialization would use 1G memory
> instantly(Some time the serialization amplifies the memory usage). However,
> the serialized object is only used when the Akka sends the message. So we
> could reduce the memory pressure if we only serialize the object when the
> message would be sent by the Akka.
> Akka would serialize the message at last and all the XXXGateway related class
> could be visible by the RPC level. Because of that, I think the eager
> serialization in the constructor of `RemoteRpcInvocation` could be avoided. I
> also do a simple test and find this could reduce the time cost of the RPC
> call. The 1k number of RPC calls with 1m `String` message: The current
> version costs around 2700ms; the Nonserialization version cost about 37ms.
>
> In summary, this Jira proposes to remove the eager serialization at the
> constructor of `RemoteRpcInvocation`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15032) Remove the eager serialization from `RemoteRpcInvocation`
[
https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986935#comment-16986935
]
Guowei Ma commented on FLINK-15032:
---
Hi [~SleePy]
>>>Do you mean we could postpone the serialization till
>>>{{RemoteRpcInvocation#writeObject}}?
yes.
>>>Could you explain it a bit more? Do you mean there is no visible issue of
>>>postponing the serialization? Or we don't need serialization anymore?
What I mean is that there is no visible issue of postponing the serialization.
>>>Do we still need serialization in {{wirteObject}} after removing
>>>serialization in constructor?
The serialization is still needed.
> Remove the eager serialization from `RemoteRpcInvocation`
> --
>
> Key: FLINK-15032
> URL: https://issues.apache.org/jira/browse/FLINK-15032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Guowei Ma
>Priority: Major
>
> Currently, the constructor of `RemoteRpcInvocation` serializes the
> `parameterTypes` and `arg` of an RPC call. This could lead to a problem:
> Consider a job that has 1k parallelism and has a 1m union list state. When
> deploying the 1k tasks, the eager serialization would use 1G memory
> instantly(Some time the serialization amplifies the memory usage). However,
> the serialized object is only used when the Akka sends the message. So we
> could reduce the memory pressure if we only serialize the object when the
> message would be sent by the Akka.
> Akka would serialize the message at last and all the XXXGateway related class
> could be visible by the RPC level. Because of that, I think the eager
> serialization in the constructor of `RemoteRpcInvocation` could be avoided. I
> also do a simple test and find this could reduce the time cost of the RPC
> call. The 1k number of RPC calls with 1m `String` message: The current
> version costs around 2700ms; the Nonserialization version cost about 37ms.
>
> In summary, this Jira proposes to remove the eager serialization at the
> constructor of `RemoteRpcInvocation`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
