[GitHub] flink issue #6272: [FLINK-9755][network] forward exceptions in RemoteInputCh...

2018-07-19 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6272 Thanks for the review, merging... ---

[GitHub] flink issue #6115: [FLINK-9435][java] Remove per-key selection Tuple instant...

2018-07-19 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6115 Thanks for the review, merging... ---

[GitHub] flink issue #6272: [FLINK-9755][network] forward exceptions in RemoteInputCh...

2018-07-19 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6272 rebased to solve the merge conflict (auto-solved by git though) ---

[GitHub] flink pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl ...

2018-07-19 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6355#discussion_r203652882 --- Diff: docs/ops/security-ssl.md --- @@ -33,6 +33,9 @@ SSL can be enabled for all network communication between Flink components. SSL k

[GitHub] flink pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl ...

2018-07-19 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6355#discussion_r203617345 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/SecurityOptions.java --- @@ -160,4 +160,41 @@ key("security.ssl.v

[GitHub] flink pull request #6363: [FLINK-9860][REST] fix buffer leak in FileUploadHa...

2018-07-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6363#discussion_r203432013 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/netty/NettyLeakDetectionResource.java --- @@ -0,0 +1,84 @@ +/* + * Licensed

[GitHub] flink pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl ...

2018-07-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6355#discussion_r203405530 --- Diff: flink-core/src/main/java/org/apache/flink/configuration/SecurityOptions.java --- @@ -160,4 +160,41 @@ key("security.ssl.v

[GitHub] flink pull request #6363: [FLINK-9860][REST] fix buffer leak in FileUploadHa...

2018-07-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6363#discussion_r203398509 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/rest/FileUploadHandlerTest.java --- @@ -50,6 +55,24 @@ private static final

[GitHub] flink pull request #6363: [FLINK-9860][REST] fix buffer leak in FileUploadHa...

2018-07-18 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6363 [FLINK-9860][REST] fix buffer leak in FileUploadHandler ## What is the purpose of the change The current implementation of `FileUploadHandler` has two cases where the given message

[GitHub] flink issue #6355: [FLINK-9878][network][ssl] add more low-level ssl options

2018-07-17 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6355 about the travis error: I tried regenerating the configuration page from the sources but it does not change at all and the "documentation outdated" remains :( ---

[GitHub] flink pull request #6355: [FLINK-9878][network][ssl] add more low-level ssl ...

2018-07-17 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6355 [FLINK-9878][network][ssl] add more low-level ssl options ## What is the purpose of the change This is mostly to tackle bugs like https://github.com/netty/netty/issues/832 (JDK issue

[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r202720265 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java --- @@ -65,7 +65,11 @@ key

[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r202720821 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering

[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r202719580 --- Diff: flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java --- @@ -65,7 +65,11 @@ key

[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r202722162 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering

[GitHub] flink pull request #6340: [FLINK-9842][rest] Pass actual configuration to Bl...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6340#discussion_r202653111 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/rest/handler/job/JobSubmitHandlerTest.java --- @@ -57,24 +60,39 @@ /** * Tests

[GitHub] flink issue #6327: [FLINK-9839][e2e] add a streaming allround test with SSL ...

2018-07-16 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6327 Thanks for the review @zentol. Yes, this PR will fail until #6340 is merged. Regarding the warning: I did not want to bother with different formats here, but you're right: a follow-up PR

[GitHub] flink pull request #6327: [FLINK-9839][e2e] add a streaming allround test wi...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6327#discussion_r202650892 --- Diff: flink-end-to-end-tests/test-scripts/common.sh --- @@ -148,6 +151,41 @@ function create_ha_config() { EOL } +function set_conf_ssl

[GitHub] flink pull request #6327: [FLINK-9839][e2e] add a streaming allround test wi...

2018-07-16 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6327#discussion_r202650635 --- Diff: flink-end-to-end-tests/test-scripts/common.sh --- @@ -183,9 +221,15 @@ function start_cluster { "$FLINK_DIR"/bin/start-

[GitHub] flink issue #6327: [FLINK-9839][e2e] add a streaming allround test with SSL ...

2018-07-16 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6327 Rebased and enabled SSL only for already existing e2e tests - since the old configuration values for enabling SSL communication for all components `security.ssl.enabled` was retained though, #6326 did

[GitHub] flink issue #6326: Mutual authentication for internal communication

2018-07-13 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6326 Can you also update `ops/security-ssl.md` with the updated configuration? ---

[GitHub] flink issue #6327: [FLINK-9839][e2e] add a streaming allround test with SSL ...

2018-07-13 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6327 Thanks, I'll have a second look at this PR once I can actually test it :) (maybe enabling SSL for the `test_streaming_kafka010.sh` test or so) ---

[GitHub] flink pull request #6328: [FLINK-9816][network] add option to configure SSL ...

2018-07-13 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6328 [FLINK-9816][network] add option to configure SSL engine provider for TM communication ## What is the purpose of the change Netty has the ability to run with different `SSLEngine

[GitHub] flink pull request #6327: [FLINK-9839][e2e] add a streaming allround test wi...

2018-07-13 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6327 [FLINK-9839][e2e] add a streaming allround test with SSL enabled ## What is the purpose of the change Currently, there's no simple streaming job test like the `test_batch_allround.sh

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-07-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r202066470 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer

[GitHub] flink issue #6291: [FLINK-9785][network] add remote address information to L...

2018-07-12 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6291 merging... ---

[GitHub] flink issue #6271: [FLINK-9766][network][tests] fix cleanup in RemoteInputCh...

2018-07-12 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6271 merging... ---

[GitHub] flink pull request #6272: [FLINK-9755][network] forward exceptions in Remote...

2018-07-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6272#discussion_r201971896 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -360,32 +360,44 @@ public boolean

[GitHub] flink pull request #6272: [FLINK-9755][network] forward exceptions in Remote...

2018-07-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6272#discussion_r201970654 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -360,32 +360,44 @@ public boolean

[GitHub] flink pull request #6272: [FLINK-9755][network] forward exceptions in Remote...

2018-07-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6272#discussion_r201969455 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java --- @@ -277,37 +277,17 @@ public void recycle

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-07-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r201965094 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -101,23 +111,210 @@ allows to [import Maven projects](http://books.sonatype.com/m2eclipse-book/refer

[GitHub] flink issue #5381: [FLINK-8523][network] Stop assigning floating buffers for...

2018-07-11 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5381 @zhijiangW is this PR still useful? if so, was there any progress? ---

[GitHub] flink issue #6291: [FLINK-9785][network] add remote address information to L...

2018-07-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6291 @yanghua sure, why not ... although there's no performance problem in string concatenation for exceptions - we'll abort anyway ---

[GitHub] flink issue #6115: [FLINK-9435][java] Remove per-key selection Tuple instant...

2018-07-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6115 alright - tried that change which also has the advantage of not using up memory for the `templateKey` - these are the results from the same Benchmark (same hardware, new software though, so I also re

[GitHub] flink issue #6272: [FLINK-9755][network] forward exceptions in RemoteInputCh...

2018-07-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6272 actually, I forgot to adapt the unit test which relied on the previous order of the calls and assumed that the buffer got recycled. With the changes from the fixup, the buffer is not recycled

[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...

2018-07-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5900 Hi @twalthr, I just created a fixup commit for this PR and the one for `flink-web`. Can you have a second look? ---

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-07-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r201310579 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-07-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r201286036 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #6254: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6254#discussion_r201273133 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -479,6 +508,9 @@ void

[GitHub] flink pull request #6254: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6254#discussion_r201272052 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -594,22 +626,22 @@ public String

[GitHub] flink issue #6272: [FLINK-9755][network] forward exceptions in RemoteInputCh...

2018-07-10 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6272 thanks for the review - I changed the code as requested ---

[GitHub] flink pull request #6272: [FLINK-9755][network] forward exceptions in Remote...

2018-07-10 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6272#discussion_r201260221 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/consumer/RemoteInputChannel.java --- @@ -360,32 +360,45 @@ public boolean

[GitHub] flink pull request #6291: [FLINK-9785][network] add remote address informati...

2018-07-10 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6291 [FLINK-9785][network] add remote address information to LocalTransportException instances ## What is the purpose of the change In contrast to the messages inside `RemoteTransportException`s

[GitHub] flink pull request #6272: [FLINK-9755][network] forward exceptions in Remote...

2018-07-06 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6272 [FLINK-9755][network] forward exceptions in RemoteInputChannel#notifyBufferAvailable() to the responsible thread ## What is the purpose of the change Exceptions in `RemoteInputChannel

[GitHub] flink pull request #6271: [FLINK-9766][network][tests] fix cleanup in Remote...

2018-07-05 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6271 [FLINK-9766][network][tests] fix cleanup in RemoteInputChannelTest ## What is the purpose of the change If an assertion in the tests of `RemoteInputChannelTest` fails and as a result

[GitHub] flink pull request #6257: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-05 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6257#discussion_r200283214 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java --- @@ -251,27 +257,56 @@ private MemorySegment

[GitHub] flink pull request #6257: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-05 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6257#discussion_r200274433 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java --- @@ -251,27 +257,56 @@ private MemorySegment

[GitHub] flink pull request #6257: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-05 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6257#discussion_r200274038 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/LocalBufferPool.java --- @@ -251,27 +257,56 @@ private MemorySegment

[GitHub] flink pull request #6254: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-04 Thread NicoK
Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/6254 ---

[GitHub] flink issue #6254: [FLINK-9676][network] clarify contracts of BufferListener...

2018-07-04 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6254 I updated the commit with a fixed unit test and added a big TODO on what else would need to be adapted in order to go with this solution (no updated commit message yet). Since this is even

[GitHub] flink pull request #6257: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6257 [FLINK-9676][network] clarify contracts of BufferListener#notifyBufferAvailable() and fix a deadlock ## What is the purpose of the change When recycling exclusive buffers

[GitHub] flink pull request #6254: [FLINK-9676][network] clarify contracts of BufferL...

2018-07-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6254 [FLINK-9676][network] clarify contracts of BufferListener#notifyBufferAvailable() and fix a deadlock ## What is the purpose of the change When recycling exclusive buffers

[GitHub] flink issue #6238: [FLINK-9636][network] fix inconsistency with failed buffe...

2018-07-03 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6238 actually, it was quite easy to reproduce and the fix was also just as you proposed - please see the new commits (the old one was only renamed since I created a separate issue for that now) ---

[GitHub] flink pull request #6238: [FLINK-9636][network] fix inconsistency with faile...

2018-07-03 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6238#discussion_r199702444 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java --- @@ -147,7 +151,12 @@ public void recycle

[GitHub] flink pull request #6238: [FLINK-9636][network] fix inconsistency with faile...

2018-07-02 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6238 [FLINK-9636][network] fix inconsistency with failed buffer redistribution ## What is the purpose of the change If an exception is thrown in `NetworkBufferPool#requestMemorySegments()`'s

[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...

2018-06-12 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5900 Thanks for the review, Timo. Regarding the copy of the gradle build file: since we did not really have a pristine version (only one that is generated by the shell script) I wanted to put one

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194902416 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194902217 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194901682 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194901455 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194900846 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-06-12 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5900#discussion_r194900211 --- Diff: docs/quickstart/java_api_quickstart.md --- @@ -108,16 +118,201 @@ In Eclipse, choose box: `-Xmx800m`. In IntelliJ IDEA recommended way

[GitHub] flink pull request #6115: [FLINK-9435][java] Remove per-key selection Tuple ...

2018-06-04 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6115 [FLINK-9435][java] Remove per-key selection Tuple instantiation via reflection in ComparableKeySelector and ArrayKeySelector ## What is the purpose of the change Inside `KeySelectorUtil

[GitHub] flink pull request #6064: [FLINK-9409][mvn] add missing assembly dependencie...

2018-05-23 Thread NicoK
Github user NicoK closed the pull request at: https://github.com/apache/flink/pull/6064 ---

[GitHub] flink issue #6064: [FLINK-9409][mvn] add missing assembly dependencies to fl...

2018-05-23 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/6064 apparently, `flink-json` and `flink-avro` should not even be in `opt/` - closing this PR ---

[GitHub] flink pull request #6064: [FLINK-9409][mvn] add missing assembly dependencie...

2018-05-23 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/6064 [FLINK-9409][mvn] add missing assembly dependencies to flink-dist/pom… ## What is the purpose of the change Building Flink via `mvn clean install -pl flink-dist -am` currently fails

[GitHub] flink issue #5923: [FLINK-9253][network] make the maximum floating buffers c...

2018-05-07 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5923 Apparently, there's a problem with unknown input channels that surfaced with the newly-merged extra tests: they don't get exclusive buffers (naturally) but the floating buffers are calculated

[GitHub] flink issue #5923: [FLINK-9253][network] make the maximum floating buffers c...

2018-05-07 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5923 Honestly, I did not find the proposed change too convincing either due to the additional branch into bounded vs. unbounded partitions which caused the variable names to be wrong in the unbounded case

[GitHub] flink pull request #5963: [FLINK-9305][s3] also register flink-s3-fs-hadoop'...

2018-05-07 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5963 [FLINK-9305][s3] also register flink-s3-fs-hadoop's factory for the s3a:// scheme ## What is the purpose of the change For enhanced user experience, we should also register our Hadoop S3A

[GitHub] flink issue #5900: [FLINK-9222][docs] add documentation for setting up Gradl...

2018-04-30 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5900 I'm pretty sure I got all this covered by using a separate "configuration": `myShadowJar` - not the standard dependency configuration*, but does the trick: Everything added to `myShadowJ

[GitHub] flink pull request #5943: [FLINK-9275][streaming] add taskName to the output...

2018-04-30 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5943 [FLINK-9275][streaming] add taskName to the output flusher thread's name ## What is the purpose of the change All output flusher threads are named "OutputFlusher" while at the o

[GitHub] flink pull request #5942: [FLINK-9274][kafka] add thread name for partition ...

2018-04-30 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5942 [FLINK-9274][kafka] add thread name for partition discovery ## What is the purpose of the change For debugging, threads should have names to filter on and get a quick overview. The Kafka

[GitHub] flink pull request #5923: [FLINK-9253][network] make the maximum floating bu...

2018-04-26 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5923 [FLINK-9253][network] make the maximum floating buffers count channel-type independent ## What is the purpose of the change The credit-based flow control path assigns exclusive buffers only

[GitHub] flink issue #5915: [FLINK-9243][tests] fix flaky SuccessAfterNetworkBuffersF...

2018-04-26 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5915 true, `points.rebalance()` is redundant, thanks for having a look, I created a follow-up fix in #5916 ---

[GitHub] flink pull request #5916: [hotfix][tests] remove redundant rebalance in Succ...

2018-04-26 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5916 [hotfix][tests] remove redundant rebalance in SuccessAfterNetworkBuffersFailureITCase This removes a redundant `rebalance()` operation introduced by #5915. You can merge this pull request into a Git

[GitHub] flink pull request #5915: [FLINK-9243][tests] fix flaky SuccessAfterNetworkB...

2018-04-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5915 [FLINK-9243][tests] fix flaky SuccessAfterNetworkBuffersFailureITCase this adds some `rebalance()` operations into `SuccessAfterNetworkBuffersFailureITCase#runKMeans` which should be set up to fail

[GitHub] flink pull request #5914: [FLINK-9256][network] fix NPE in SingleInputGate#u...

2018-04-25 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5914 [FLINK-9256][network] fix NPE in SingleInputGate#updateInputChannel() for non-credit based flow control ## What is the purpose of the change `SingleInputGate#updateInputChannel()` fails

[GitHub] flink pull request #5900: [FLINK-9222][docs] add documentation for setting u...

2018-04-23 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5900 [FLINK-9222][docs] add documentation for setting up Gradle projects ## What is the purpose of the change Together with https://github.com/apache/flink-web/pull/105, this will add a project

[GitHub] flink pull request #5870: [FLINK-8967][tests] Port NetworkStackThroughputITC...

2018-04-19 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5870#discussion_r182699788 --- Diff: flink-tests/src/test/java/org/apache/flink/test/runtime/NetworkStackThroughputITCase.java --- @@ -327,7 +335,8 @@ private JobGraph createJobGraph

[GitHub] flink pull request #5870: [FLINK-8967][tests] Port NetworkStackThroughputITC...

2018-04-19 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5870#discussion_r182668925 --- Diff: flink-tests/src/test/java/org/apache/flink/test/runtime/NetworkStackThroughputITCase.java --- @@ -234,43 +232,51 @@ public void testThroughput

[GitHub] flink pull request #5874: [FLINK-9076][network] reduce minimum number of flo...

2018-04-18 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5874 [FLINK-9076][network] reduce minimum number of floating buffers to 0 ## What is the purpose of the change Currently, floating buffers (per gate) are always required in case credit-based flow

[GitHub] flink pull request #5872: [FLINK-9206][checkpoints] add job IDs to Checkpoin...

2018-04-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5872#discussion_r182478661 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java --- @@ -857,7 +859,7 @@ public void run

[GitHub] flink issue #5872: [FLINK-9206][checkpoints] add job IDs to CheckpointCoordi...

2018-04-18 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5872 alright...there you go ---

[GitHub] flink pull request #5873: [FLINK-9208][tests] fix naming of StreamNetworkThr...

2018-04-18 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5873 [FLINK-9208][tests] fix naming of StreamNetworkThroughputBenchmarkTest This PR fixes the naming of `StreamNetworkThroughputBenchmarkTest` so that it gets included in the maven tests. You can merge

[GitHub] flink pull request #5872: [FLINK-9206][checkpoints] add job IDs to Checkpoin...

2018-04-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5872#discussion_r182455513 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java --- @@ -857,7 +859,7 @@ public void run

[GitHub] flink pull request #5872: [FLINK-9206][checkpoints] add job IDs to Checkpoin...

2018-04-18 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5872#discussion_r182455202 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java --- @@ -1214,7 +1216,7 @@ public void run

[GitHub] flink pull request #5872: [FLINK-9206][checkpoints] add job IDs to Checkpoin...

2018-04-18 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5872 [FLINK-9206][checkpoints] add job IDs to CheckpointCoordinator log messages ## What is the purpose of the change The `CheckpointCoordinator` exists per job but its log messages (all bundled

[GitHub] flink pull request #5842: [FLINK-9144][network] fix SpillableSubpartition ca...

2018-04-12 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5842 [FLINK-9144][network] fix SpillableSubpartition causing jobs to hang when spilling ## What is the purpose of the change This should fix various scenarios where the backlog accounting

[GitHub] flink pull request #5841: [FLINK-9163[e2e-tests] harden signal traps and con...

2018-04-12 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5841 [FLINK-9163[e2e-tests] harden signal traps and config restoration ## What is the purpose of the change Signal traps on certain systems, e.g. Linux, may be called concurrently when the trap

[GitHub] flink issue #5672: [FLINK-8872][flip6] fix yarn detached mode command parsin...

2018-04-05 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5672 ok, after fixing one typo, this should pass Travis now ---

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

2018-04-05 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5802#discussion_r179453256 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/api/writer/RecordWriterTest.java --- @@ -309,8 +309,6 @@ public void

[GitHub] flink pull request #5802: [FLINK-9087] [runtime] close the BufferConsumer in...

2018-04-05 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5802#discussion_r179451610 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/api/writer/RecordWriter.java --- @@ -164,7 +164,7 @@ public BufferConsumer

[GitHub] flink issue #5672: [FLINK-8872][flip6] fix yarn detached mode command parsin...

2018-04-03 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5672 I reworked the code as desired, i.e. adding `-yd` and `--yarndetached` as deprecated parameters to `CliFrontendParser` and adapting `ProgramOptions` and `FlinkYarnSessionCli` accordingly. ---

[GitHub] flink pull request #5672: [FLINK-8872][flip6] fix yarn detached mode command...

2018-04-03 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5672#discussion_r178804566 --- Diff: flink-yarn/pom.xml --- @@ -326,6 +334,62 @@ under the License

[GitHub] flink issue #5790: [FLINK-9107][docs] document timer coalescing for ProcessF...

2018-04-03 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5790 thanks - fixed and extended with an example for "schedule with the next watermark" for event-time timers ---

[GitHub] flink issue #5790: [FLINK-9107][docs] document timer coalescing for ProcessF...

2018-03-29 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5790 Actually, more advanced schemes using `current watermark + 1` (which fires with the next watermark) for the event time timer should also go into the documentation. I'll extend the PR ... ---

[GitHub] flink pull request #5790: [FLINK-9107][docs] document timer coalescing for P...

2018-03-29 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5790 [FLINK-9107][docs] document timer coalescing for ProcessFunction ## What is the purpose of the change In a ProcessFunction, registering timers for each event via `ctx.timerService

[GitHub] flink pull request #5695: [FLINK-8704][tests] Port PartialConsumerPipelinedR...

2018-03-29 Thread NicoK
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/5695#discussion_r178069787 --- Diff: flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/PartialConsumePipelinedResultTest.java --- @@ -32,41 +31,51

[GitHub] flink pull request #5788: [FLINK-9110][docs] fix local bundler installation

2018-03-29 Thread NicoK
GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/5788 [FLINK-9110][docs] fix local bundler installation ## What is the purpose of the change The fallback to installing `bundler` locally within `docs/build_docs.sh` did not work for several

[GitHub] flink issue #5624: [FLINK-8402][s3][tests] fix hadoop/presto S3 IT cases for...

2018-03-29 Thread NicoK
Github user NicoK commented on the issue: https://github.com/apache/flink/pull/5624 Indeed, Presto-S3 does better in `com.facebook.presto.hive.PrestoS3FileSystem#create()`: ``` if ((!overwrite) && exists(path)) { throw new IOException("File already ex

  1   2   3   4   5   6   7   8   9   10   >