Stephan Ewen created FLINK-4399:
-----------------------------------
Summary: Add support for oversized messages
Key: FLINK-4399
URL: https://issues.apache.org/jira/browse/FLINK-4399
Project: Flink
Issue Type: Sub-task
Components: Distributed Coordination
Environment: FLIP-6 feature branch
Reporter: Stephan Ewen
Fix For: 1.2.0
Currently, messages larger than the maximum Akka Framesize cause an error when
being transported. We should add a way to pass messages that are larger than
the Framesize, as may happen for:
- {{collect()}} calls that collect large data sets (via accumulators)
- Job submissions and operator deployments where the functions closures are
large (for example because it contains large pre-loaded data)
- Function restore in cases where restored state is larger than checkpointed
state (union state)
I suggest to use the {{BlobManager}} to transfer large payload.
- On the sender side, oversized messages are stored under a transient blob
(which is deleted after first retrieval, or after a certain number of minutes)
- The sender sends a "pointer to blob message" instead.
- The receiver grabs the message from the blob upon receiving the pointer
message
The RPC Service should be optionally initializable with a "large message
handler" which is internally the {{BlobManager}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
