[
https://issues.apache.org/jira/browse/FLINK-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14556435#comment-14556435
]
ASF GitHub Bot commented on FLINK-2084:
---------------------------------------
GitHub user StephanEwen opened a pull request:
https://github.com/apache/flink/pull/718
[FLINK-2084] [core] Add an option to start Flink in streaming mode
The streaming mode subsumes the batch mode, as it allows to run both kinds
of programs. Batch programs will most likely run slightly less efficient than
in pure batch mode, due to lazy memory allocation (see below)
Adds new startup options to the JobManager and TaskManager.
Adds additional scripts `bin/start-cluster-streaming.sh` and
`start-local-streaming.sh`.
Currently, streaming mode only sets the memory manager to lazy memory
allocation to ensure heap is not blocked by batch memory manager
Includes [FLINK-2085] To support on-demand memory allocation in the memory
manager.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/StephanEwen/incubator-flink streaming_mode
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/718.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #718
----
commit 7b58e2d49521e955ddacfa5586659df01fe57c57
Author: Stephan Ewen <[email protected]>
Date: 2015-05-22T16:35:40Z
[FLINK-2085] [runtime] Add an option to let the MemoryManager allocate and
release memory as needed.
This is an alternative mode to the current mode that pre-allocates all
memory.
The default remains to pre-allocate all memory.
commit 43978cd979114858ad2f200f04bd1723693c2c3e
Author: Stephan Ewen <[email protected]>
Date: 2015-05-22T15:12:45Z
[FLINK-2084] [core] Add an option to start Flink in streaming mode
- Streaming mode sets the memory manager to lazy memory allocation to
ensure
heap is not blocked by batch memory manager
commit 1df1f12e367b4b89b56e9e47dbe5a7a11e5e1583
Author: Stephan Ewen <[email protected]>
Date: 2015-05-22T15:24:19Z
[tests] Adjust tests for dedicated streaming mode and clean up test bases.
----
> Create a dedicated streaming mode
> ---------------------------------
>
> Key: FLINK-2084
> URL: https://issues.apache.org/jira/browse/FLINK-2084
> Project: Flink
> Issue Type: Improvement
> Components: Distributed Runtime
> Affects Versions: 0.9
> Reporter: Stephan Ewen
> Assignee: Stephan Ewen
> Fix For: 0.9
>
>
> As per discussion on the mailing list
> http://mail-archives.apache.org/mod_mbox/flink-dev/201505.mbox/browser
> - We add a dedicated streaming mode for now. The streaming mode supersedes
> the batch mode, so it can run both type of programs.
> - The streaming mode sets the memory manager to "lazy allocation".
> -> So long as it runs pure streaming jobs, the full heap will be
> available to window buffers and UDFs.
> -> Batch programs can still run, so mixed workloads are not prevented.
> Batch programs are a bit less robust there, because the memory manager does
> not pre-allocate memory. UDFs can eat into Flink's memory portion.
> - The streaming mode starts the necessary configured components/services for
> state backups
> Over the next versions, we want to bring these things together:
> - use the managed memory for window buffers
> - on-demand starting of the state backend
> Then, we deprecate the streaming mode, let both modes start the cluster in
> the same way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
