GitHub user sachingoel0101 opened a pull request:
https://github.com/apache/flink/pull/975
[WIP] User defined communication between tasks at Runtime
This PR implements an interface for allowing users to broadcast messages to
parallel running instances of a task.
This allows for users to communicate with the parallel instances and share
data. This of course lets one instance know the outputs being evaluated by the
other instances which can further with computation.
This can also be used to implement asynchronous iterations, as shown in the
example `RuntimeStatistics` [that name was for something else :')]. Further,
this can also be used to implement better statistics, by sharing data such as
time per element, number of elements processed at each slot, etc.
There are of course a few drawbacks:
1. Uses Akka for sending messages, which leads to an inherent limit on the
size of messages being transmitted.
2. Limited number of messages: Tasks will need to consume the messages, not
just keep broadcasting them. The message store is managed by the task manager
on heap memory, so we need to keep the size in check. I've defined a
conservative limit of 50 for now.
3. Some type casting: Users will need to cast the message received back
into their intended message type. This can perhaps be solved using reflection.
I haven't started working on this yet.
@tillrohrmann can you review this again? This is mostly a complete
re-design over the previous one, and I have addressed the issues about too much
memory consumption [an efficient multi-consumer queue] and allowing different
tasks to set up their own message stores.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sachingoel0101/flink runtime_comm
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/975.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #975
----
commit 4e19f5c58454f318d1f7b50d13a73aba4054baca
Author: Sachin Goel <[email protected]>
Date: 2015-08-03T16:32:01Z
Implemented runtime communication between tasks. Wrote examples to show
that it works and sends messages even while performing the job itself, in a map
function.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
