[ https://issues.apache.org/jira/browse/FLINK-4399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16786078#comment-16786078 ]
Alexander Lehmann commented on FLINK-4399: ------------------------------------------ Hi, as of late I'm also running into this issue and hope you don't mind me asking whether there's been some progress in this matter? Thanks a lot! > Add support for oversized messages > ---------------------------------- > > Key: FLINK-4399 > URL: https://issues.apache.org/jira/browse/FLINK-4399 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination > Environment: FLIP-6 feature branch > Reporter: Stephan Ewen > Priority: Major > Labels: flip-6 > Fix For: 1.8.0 > > > Currently, messages larger than the maximum Akka Framesize cause an error > when being transported. We should add a way to pass messages that are larger > than the Framesize, as may happen for: > - {{collect()}} calls that collect large data sets (via accumulators) > - Job submissions and operator deployments where the functions closures are > large (for example because it contains large pre-loaded data) > - Function restore in cases where restored state is larger than > checkpointed state (union state) > I suggest to use the {{BlobManager}} to transfer large payload. > - On the sender side, oversized messages are stored under a transient blob > (which is deleted after first retrieval, or after a certain number of minutes) > - The sender sends a "pointer to blob message" instead. > - The receiver grabs the message from the blob upon receiving the pointer > message > The RPC Service should be optionally initializable with a "large message > handler" which is internally the {{BlobManager}}. -- This message was sent by Atlassian JIRA (v7.6.3#76005)