[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Ufuk Celebi (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101945#comment-15101945
 ] 

Ufuk Celebi commented on FLINK-3073:


Minor fix added in 21882af

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Henry Saputra (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15057463#comment-15057463
 ] 

Henry Saputra commented on FLINK-3073:
--

The only thing I am bit concern is how fast we add and remove feature, in this 
case the streaming mode, which could add con

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048456#comment-15048456
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163184249
  
Looks good!

One thing that just came to my mind is whether we should call the config 
flag "taskmanager.memory.preallocate" (default false).

I think preallocate is somehow easier to get than lazy allocate...


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048426#comment-15048426
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163177539
  
Any objections against me merging this now?


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048458#comment-15048458
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163184597
  
Can do, yes. 


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048632#comment-15048632
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163221930
  
`preallocate` can be seen as one word, no?


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048475#comment-15048475
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163187358
  
PR looks good. +1 for `taskmanager.memory.preallocate`.


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048508#comment-15048508
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163192234
  
+1 from my side, after the switch to "preallocate"


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048527#comment-15048527
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163194389
  
is `pre-allocate` also ok? btw, we have a wild mixture of option styles, 
there is camelCase, under_score, and with-hyphens.


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048739#comment-15048739
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha closed the pull request at:

https://github.com/apache/flink/pull/1433


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048738#comment-15048738
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-163264394
  
Manually merged


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046719#comment-15046719
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162846459
  
I also discovered some other stuff, for example, the off-heap memory usage 
is not printed by the `MemoryLogger` (max is fixing this) and some stats in the 
web fronted about TaskManagers are wrong. For example "Free Memory" is simply 
the size of the JVM Heap.


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046718#comment-15046718
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162846304
  
Any more comments?

The log messages now look like this:
```
11:28:58,212 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - Limiting managed memory to 0.7 of the currently free heap space (295 
MB), memory will be allocated lazily.
```
or 
```
11:30:09,859 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - Using 0.7 of the currently free heap space for managed heap memory (295 
MB).
```


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044690#comment-15044690
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162469441
  
Did the changes to boolean option and merging of log messages.


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044649#comment-15044649
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162464044
  
@StephanEwen So you want to merge the two messages @rmetzger posted? I'm on 
it.

I'll also change the option to a boolean `lazy_alloc`, no need for the 
String option.


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044003#comment-15044003
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user rmetzger commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334401
  
Tested the change on a cluster with a job reading some 100GBs of data ...
looks good ..

```
19:07:19,537 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - Using 0.7 of the currently free heap space for Flink managed heap memory 
(9001 MB).
19:07:19,538 INFO  org.apache.flink.runtime.taskmanager.TaskManager 
 - MemoryManager will be initialized with lazy memory allocation.
```


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044007#comment-15044007
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user StephanEwen commented on the pull request:

https://github.com/apache/flink/pull/1433#issuecomment-162334879
  
As followup to this, I would suggest two changes:

1. Adjust the log message, to reflect that not 0.7 are either used 
immediately (eager allocation) or that the managed memory is capped at 0.7 * 
total memory. That makes it clear from reading the log that the memory is not 
pre-allocated.

2. The web frontend should display for managed memory: (used / max). I have 
talked to users who were confused, thinking despite starting the system in 
streaming mode, it pre-allocated all the heap way, 


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037865#comment-15037865
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46559041
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1617,7 +1598,16 @@ object TaskManager {
   }
 }
 
-val preAllocateMemory: Boolean = streamingMode == 
StreamingMode.BATCH_ONLY
+val allocationMode = configuration.getString(
+  ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY,
+  ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION);
+val preAllocateMemory: Boolean = allocationMode.equals("eager")
--- End diff --

will do



> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037853#comment-15037853
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46558419
  
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then
 #
 TM_MAX_OFFHEAP_SIZE="8388607T"
 
-if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then
+if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; 
then
--- End diff --

For the off-heap memory case, the JVM heap memory should be reduced 
regardless of eager or lazy memory allocation. I think this can be changed to

```bash
if useOffHeapMemory; then
```


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037855#comment-15037855
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46558585
  
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then
 #
 TM_MAX_OFFHEAP_SIZE="8388607T"
 
-if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then
+if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; 
then
--- End diff --

are you sure? I just preserved the previous behavior. (eager == batch_mode, 
lazy == streaming_mode)


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037856#comment-15037856
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46558633
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1617,7 +1598,16 @@ object TaskManager {
   }
 }
 
-val preAllocateMemory: Boolean = streamingMode == 
StreamingMode.BATCH_ONLY
+val allocationMode = configuration.getString(
+  ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY,
+  ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION);
+val preAllocateMemory: Boolean = allocationMode.equals("eager")
--- End diff --

Could we verify whether we only have "eager" or "lazy" and throw an 
Exception otherwise?


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037873#comment-15037873
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user StephanEwen commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46560453
  
--- Diff: 
flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala
 ---
@@ -1617,7 +1598,16 @@ object TaskManager {
   }
 }
 
-val preAllocateMemory: Boolean = streamingMode == 
StreamingMode.BATCH_ONLY
+val allocationMode = configuration.getString(
+  ConfigConstants.TASK_MANAGER_MEMORY_ALLOCATION_KEY,
+  ConfigConstants.DEFAULT_TASK_MANAGER_MEMORY_ALLOCATION);
+val preAllocateMemory: Boolean = allocationMode.equals("eager")
--- End diff --

There is already a lazy allocation flag (boolean) Can you reuse that simply?


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037830#comment-15037830
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

GitHub user aljoscha opened a pull request:

https://github.com/apache/flink/pull/1433

[FLINK-3073] Replace Streaming Mode by Memory Allocation Mode

Before, streaming mode (either batch or streaming) would specify how
memory is allocated on task managers.

This introduces a new configuration value taskmanager.memory.allocation
that can take values "lazy" or "eager". This controls how memory is
allocated.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/aljoscha/flink stream-mode-default

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/flink/pull/1433.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1433


commit bea4c112b7dbf0b7465b7419251173a742514967
Author: Aljoscha Krettek 
Date:   2015-12-01T15:13:55Z

[FLINK-3073] Replace Streaming Mode by Memory Allocation Mode

Before, streaming mode (either batch or streaming) would specify how
memory is allocated on task managers.

This introduces a new configuration value taskmanager.memory.allocation
that can take values "lazy" or "eager". This controls how memory is
allocated.




> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037846#comment-15037846
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user aljoscha commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46557639
  
--- Diff: docs/setup/cluster_setup.md ---
@@ -236,34 +236,18 @@ bin/start-cluster.sh
 
 To stop Flink, there is also a `stop-cluster.sh` script.
 
-
-### Starting Flink in the streaming mode
-
-~~~bash
-bin/start-cluster-streaming.sh
-~~~
-
-The streaming mode changes the startup behavior of Flink: The system is 
not 
-bringing up the managed memory services with preallocated memory at the 
beginning.
-Flink streaming is not using the managed memory employed by the batch 
operators.
-By not starting these services with preallocated memory, streaming jobs 
can benefit
-from more heap space being available.
-
-Note that you can still start batch jobs in the streaming mode. The memory 
manager
-will then allocate memory segments from the Java heap as needed.
-
 ### Optional: Adding JobManager/TaskManager instances to a cluster
 
 You can add both TaskManager or JobManager instances to your running 
cluster with the `bin/taskmanager.sh` and `bin/jobmanager.sh` scripts.
 
  Adding a TaskManager
 
-bin/taskmanager.sh (start [batch|streaming])|stop|stop-all)
+bin/taskmanager.sh start|stop|stop-all
 
 
  Adding a JobManager
 
-bin/jobmanager.sh (start cluster [batch|streaming])|stop|stop-all)
+bin/jobmanager.sh start|stop|stop-all
--- End diff --

fixed it


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037911#comment-15037911
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46563274
  
--- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh ---
@@ -63,7 +57,7 @@ if [[ $STARTSTOP == "start" ]]; then
 #
 TM_MAX_OFFHEAP_SIZE="8388607T"
 
-if [[ "${STREAMINGMODE}" == "batch" ]] && useOffHeapMemory; then
+if [[ "${FLINK_TM_MEM_ALLOC}" == "eager" ]] && useOffHeapMemory; 
then
--- End diff --

You might want to allocate lazily in batch jobs with off-heap memory. In 
the current state of the pull request, we could run into out of memory errors 
then. On the other hand, I see that we don't want to subtract heap memory in 
streaming with lazy memory allocation (because we don't allocate managed memory 
for streaming jobs for now).


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037843#comment-15037843
 ] 

ASF GitHub Bot commented on FLINK-3073:
---

Github user mxm commented on a diff in the pull request:

https://github.com/apache/flink/pull/1433#discussion_r46557510
  
--- Diff: docs/setup/cluster_setup.md ---
@@ -236,34 +236,18 @@ bin/start-cluster.sh
 
 To stop Flink, there is also a `stop-cluster.sh` script.
 
-
-### Starting Flink in the streaming mode
-
-~~~bash
-bin/start-cluster-streaming.sh
-~~~
-
-The streaming mode changes the startup behavior of Flink: The system is 
not 
-bringing up the managed memory services with preallocated memory at the 
beginning.
-Flink streaming is not using the managed memory employed by the batch 
operators.
-By not starting these services with preallocated memory, streaming jobs 
can benefit
-from more heap space being available.
-
-Note that you can still start batch jobs in the streaming mode. The memory 
manager
-will then allocate memory segments from the Java heap as needed.
-
 ### Optional: Adding JobManager/TaskManager instances to a cluster
 
 You can add both TaskManager or JobManager instances to your running 
cluster with the `bin/taskmanager.sh` and `bin/jobmanager.sh` scripts.
 
  Adding a TaskManager
 
-bin/taskmanager.sh (start [batch|streaming])|stop|stop-all)
+bin/taskmanager.sh start|stop|stop-all
 
 
  Adding a JobManager
 
-bin/jobmanager.sh (start cluster [batch|streaming])|stop|stop-all)
+bin/jobmanager.sh start|stop|stop-all
--- End diff --

Could you preserve the "cluster" or "local" parameter?


> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Maximilian Michels (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035879#comment-15035879
 ] 

Maximilian Michels commented on FLINK-3073:
---

+1

There is no difference between streaming and batch apart from the prellocation 
of the memory. If there is ever one, we can introduce a dedicated 
batch/streaming option again. Please provide an option in the config to 
preallocate the managed memory. I agree it should be turned off by default.

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Robert Metzger (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15034151#comment-15034151
 ] 

Robert Metzger commented on FLINK-3073:
---

I would remove all the code for specific startup modes and just use the lazy 
allocating memory manager by default.

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15033528#comment-15033528
 ] 

Aljoscha Krettek commented on FLINK-3073:
-

Should we keep the distinction between streaming-mode and batch-mode nn case we 
have some future systems that would only need to be started if using the 
cluster primarily for batch or for streaming.

Or should I change the name of the config value and internal fields and such to 
something like "memory pre-allocation mode". Because right now, it only changes 
the way how managed memory is allocated, either lazily (streaming) or upon 
startup (batch).

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
>Assignee: Aljoscha Krettek
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (FLINK-3073) Activate streaming mode by default

[Aljoscha Krettek (JIRA)]

[ 
https://issues.apache.org/jira/browse/FLINK-3073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029808#comment-15029808
 ] 

Aljoscha Krettek commented on FLINK-3073:
-

+1

> Activate streaming mode by default
> --
>
> Key: FLINK-3073
> URL: https://issues.apache.org/jira/browse/FLINK-3073
> Project: Flink
>  Issue Type: Improvement
>  Components: TaskManager
>Reporter: Robert Metzger
> Fix For: 1.0.0
>
>
> Currently, TaskManagers are still started in the batch mode.
> I have the impression that more users are actually using Flink for stream 
> processing, and, the streaming mode also allows batch workloads.
> It would be nice to change that for the 1.0 release



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)