[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-16 Thread fhueske
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1129#issuecomment-140749638 I will merge this --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/1129 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488711 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39490684 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493894 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi -

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493902 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi -

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488877 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488925 --- Diff: flink-dist/src/main/flink-bin/bin/config.sh --- @@ -173,12 +183,37 @@ IS_NUMBER="^[0-9]+$" # Define FLINK_JM_HEAP if it is not already set

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488168 --- Diff: flink-dist/src/main/flink-bin/bin/config.sh --- @@ -173,12 +183,37 @@ IS_NUMBER="^[0-9]+$" # Define FLINK_JM_HEAP if it is not already

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488510 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39489771 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39489755 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39494490 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39488999 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493918 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493898 --- Diff: flink-dist/src/main/flink-bin/bin/taskmanager.sh --- @@ -51,13 +51,41 @@ if [[ $STARTSTOP == "start" ]]; then fi fi -

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493879 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread mxm
Github user mxm commented on a diff in the pull request: https://github.com/apache/flink/pull/1129#discussion_r39493933 --- Diff: flink-runtime/src/main/scala/org/apache/flink/runtime/taskmanager/TaskManager.scala --- @@ -1586,32 +1586,29 @@ object TaskManager {

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1129#issuecomment-140336732 Thanks for the PR @mxm. I left a few comments inline. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-15 Thread fhueske
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/1129#issuecomment-140349703 Thanks for the update. Good to merge, IMO --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140035559 How about we rename `taskmanager.memory.size` to `taskmanager.memory`? Then there wouldn't be any possible clash. I could also live with `taskmanager.jvm.memory.size`.

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140148408 Closing this in favor of #1129. For further discussion please see https://issues.apache.org/jira/browse/FLINK-2667. --- If your project is set up for it, you can reply to

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm closed the pull request at: https://github.com/apache/flink/pull/1125 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1129 [FLINK-2641] integrate off-heap memory configuration Following the discussion in #1125, this pull request introduces a less "invasive" change to make the off-heap memory configurable. - add

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/1125 [FLINK-2641] integrate off-heap memory configuration - remove taskmanager.heap.mb and jobmanager.heap.mb - introduce taskmanager.memory.size and jobmanger.memory.size which allow to control

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread StephanEwen
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140019328 Also, the mechanism for network memory configuration changed (memory size now, was number of buffers). Why changing that? Was the old mechanism bad? --- If your

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140022781 In addition to the discussion in the JIRA, I had to change the way the network memory is configured because otherwise I cannot calculate the direct memory size correctly.

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread StephanEwen
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140028178 I think this is tricky, a config flag that used to refer to something different (memory manager size) now refers to a new concept (jvm memory size). This is bound

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140029554 That's why the pull request has the check for obsolete keys which fails to start Flink when old config entries are detected. In addition, it has sanity checks to check

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread mxm
Github user mxm commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140030087 Yes, that will follow. I wanted to kick-off the discussion first because I had this gut-feeling that it might cause more discussion than in the JIRA once the pull request

[GitHub] flink pull request: [FLINK-2641] integrate off-heap memory configu...

2015-09-14 Thread rmetzger
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/1125#issuecomment-140029908 The pull request should also update the documentation. In particular these configuration values affect the documentation in many places. --- If your project is set up