[
https://issues.apache.org/jira/browse/FLINK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921355#comment-16921355
]
Till Rohrmann commented on FLINK-4399:
--------------------------------------
Thanks for the detailed answer [~SleePy]. Given that we don't have an acute
problem with too large messages I'm aware of atm and only limited review
capacities I'd suggest that we add some utilities to offload large payloads on
the {{BlobServer}} the next time we run into this problem.
For improving the RPC service to support large messages, I think it is not
super critical. If you want, you can work on it but I cannot promise to help
you with the review/merging in this release cycle. I would try to squeeze it in
whenever there is time. So if you are ok with not getting fast feedback, then
it could work.
> Add support for oversized messages
> ----------------------------------
>
> Key: FLINK-4399
> URL: https://issues.apache.org/jira/browse/FLINK-4399
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
> Environment: FLIP-6 feature branch
> Reporter: Stephan Ewen
> Assignee: Biao Liu
> Priority: Major
> Labels: flip-6
>
> Currently, messages larger than the maximum Akka Framesize cause an error
> when being transported. We should add a way to pass messages that are larger
> than the Framesize, as may happen for:
> - {{collect()}} calls that collect large data sets (via accumulators)
> - Job submissions and operator deployments where the functions closures are
> large (for example because it contains large pre-loaded data)
> - Function restore in cases where restored state is larger than
> checkpointed state (union state)
> I suggest to use the {{BlobManager}} to transfer large payload.
> - On the sender side, oversized messages are stored under a transient blob
> (which is deleted after first retrieval, or after a certain number of minutes)
> - The sender sends a "pointer to blob message" instead.
> - The receiver grabs the message from the blob upon receiving the pointer
> message
> The RPC Service should be optionally initializable with a "large message
> handler" which is internally the {{BlobManager}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
