[GitHub] flink pull request: [FLINK-3688] WindowOperator.trigger() does not...

2016-04-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1861#issuecomment-207440713 Could you please close if github does not do so automatically. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request: [FLINK-3688] WindowOperator.trigger() does not...

2016-04-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1861#issuecomment-207287141 Thanks for your work! I'll merge it today. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink pull request: [FLINK-3730] Fix RocksDB Local Directory Initi...

2016-04-12 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1873 [FLINK-3730] Fix RocksDB Local Directory Initialization You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink rocksdb-fix-dir

[GitHub] flink pull request: [FLINK-3711][docs] Documentation of Scala fold...

2016-04-07 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1860#issuecomment-206766766 Thanks for the fix! I'm merging... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-07 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1850 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-07 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1850#issuecomment-206799977 Manually merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-3645] [tests] HDFSCopyUtilitiesTest fai...

2016-03-22 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1825#issuecomment-199771908 :+1: good fix! Could you please merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore

2016-03-23 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1830 [FLINK-3651] Fix faulty RollingSink Restore On restore the sink for subtask index i has to cleanup leftover files for subtask i. The pattern used for checking this was not properly

[GitHub] flink pull request: [FLINK-3659] Allow ConnectedStreams to Be Keye...

2016-03-23 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1831 [FLINK-3659] Allow ConnectedStreams to Be Keyed on Only One Side You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink connected

[GitHub] flink pull request: [FLINK-3651] Fix faulty RollingSink Restore

2016-03-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1830#issuecomment-200863307 Yes, what @rmetzger said is correct. Without the fix and the increase of parallelism it will be a flaky test that still succeeds sometimes. --- If your project

[GitHub] flink pull request: [FLINK-3659] Allow ConnectedStreams to Be Keye...

2016-03-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1831#issuecomment-200933436 To elaborate on this. State right now works well if you stick to the (admittedly somewhat hidden) rules. That is, you should only access state if there is a key

[GitHub] flink pull request: [FLINK-3583] User configuration visible in web...

2016-03-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1768#issuecomment-193730962 Ok, makes sense. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-3583] User configuration visible in web...

2016-03-08 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1768#discussion_r55342808 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionGraph.java --- @@ -306,6 +306,12 @@ public ExecutionGraph

[GitHub] flink pull request: [FLINK-3583] User configuration visible in web...

2016-03-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1768#issuecomment-193720952 Thanks for your PR! I'll look into whether loading the ExecutionConfig earlier breaks anything and then merge it. --- If your project is set up for it, you can reply

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1774#issuecomment-193830427 Thanks @uce as well! :smiley: I incorporated all the fixes. The only thing I'm not happy with is the section with the window. I feel that I can't do

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1774 [FLINK-3591] Replace Quickstart K-Means Example by Streaming Example I attached a rough pdf rendering of the changed page. [Apache Flink 1.1-SNAPSHOT Documentation: Quick Start: Monitoring

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1774#discussion_r55373645 --- Diff: docs/quickstart/run_example_quickstart.md --- @@ -27,116 +27,360 @@ under the License. * This will be replaced by the TOC {:toc

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1774#issuecomment-193822714 Thanks for the thorough review @vasia, that was quick :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1774#discussion_r55376360 --- Diff: docs/quickstart/run_example_quickstart.md --- @@ -27,116 +27,360 @@ under the License. * This will be replaced by the TOC {:toc

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1774#discussion_r55371997 --- Diff: docs/quickstart/run_example_quickstart.md --- @@ -27,116 +27,360 @@ under the License. * This will be replaced by the TOC {:toc

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1774 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: [FLINK-3591] Replace Quickstart K-Means Exampl...

2016-03-08 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1774#issuecomment-193844948 Alright, I merged it with the fixes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request: [FLINK-3554] [streaming] Emit a MAX Watermark ...

2016-03-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1750#issuecomment-192241682 Yes, that's a problem. Could you maybe open an issue for that here: https://issues.apache.org/jira/browse/FLINK In the meantime, I have a workaround for you

[GitHub] flink pull request: FLINK-3428: Adds a fixed time trailing waterma...

2016-03-04 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1764#discussion_r55019932 --- Diff: flink-streaming-scala/src/main/scala/org/apache/flink/streaming/api/scala/DataStream.scala --- @@ -750,6 +750,34 @@ class DataStream[T](stream

[GitHub] flink pull request: [FLINK-3578] Support RichFunctions for Scala W...

2016-03-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1765#issuecomment-192269684 One thing is that we should maybe have tests/ITCases that verify that rich functions actually work. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request: [FLINK-3557] [stream, scala] Introduce seconda...

2016-03-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1748#discussion_r54560116 --- Diff: flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowFoldITCase.scala --- @@ -70,7 +70,7 @@ class WindowFoldITCase

[GitHub] flink pull request: [FLINK-2788] [apis] Add TypeHint class to allo...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1744#issuecomment-190660978 I think we might abuse the `of()` name a bit to much. Maybe `from()` or `for()` would be more appropriate here. So it would be `TypeInformation.for(new TypeHint

[GitHub] flink pull request: [FLINK-3557] [stream, scala] Introduce seconda...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1748#issuecomment-190702497 Very good fix. :+1: If we have another RC I would like to include this in 1.0 --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] flink pull request: Enhance Partitioned State and use it in Window...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1562#issuecomment-190649095 Hi @wenlonglwl, the reason for this were mostly practical concerns. We want to use our partitioned state abstraction for the window state because that makes it easy

[GitHub] flink pull request: [FLINK-3550] [FLINK-3552] Rework WindowJoin an...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1745#issuecomment-190667026 With two small inline comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request: [FLINK-3550] [FLINK-3552] Rework WindowJoin an...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1745#issuecomment-190666910 Looks good :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: Enhance Partitioned State and use it in Window...

2016-03-03 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1562#issuecomment-191681857 It is certainly possible to find some way of doing it but it is not straightforward. So for now I wanted to keep it simple until we figure out a good way to do

[GitHub] flink pull request: [FLINK-3554] [streaming] Emit a MAX Watermark ...

2016-03-03 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1750#issuecomment-191693706 Yes, you are right and we are thinking about it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] flink pull request: [FLINK-3554] [streaming] Emit a MAX Watermark ...

2016-03-03 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1750#issuecomment-191695612 Yes, right now we favor correctness over completeness. For example in a Count window, if we emitted an unfinished count window then the count of the elements

[GitHub] flink pull request: [FLINK-3575] Update Working With State Section...

2016-03-03 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1760 [FLINK-3575] Update Working With State Section in Doc You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink doc-update-state

[GitHub] flink pull request: [FLINK-3554] [streaming] Emit a MAX Watermark ...

2016-03-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1750#issuecomment-190735172 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink pull request: [FLINK-3554] [streaming] Emit a MAX Watermark ...

2016-03-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1750#issuecomment-191260012 Hi, what are you referring to? The watermark does not interact with count windows. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1802#issuecomment-205302445 I did the changes, I introduced `canMerge()` and added a default `onMerge()` that throws a `RuntimeExeption` in `Trigger`. This way we don't break the API for already

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1850#issuecomment-205391579 Can do, yes. But we shouldn't block, as you said. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] flink pull request:

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/commit/76968c6360c17d5deb4e42727c16bc1b9a891b26#commitcomment-16956214 Done --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1850#issuecomment-205398515 I'm seeing this failure on Travis: https://travis-ci.org/aljoscha/flink/jobs/120638971 Could be a bug introduced by the change. I'm investigating

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1850#issuecomment-205406051 done :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-3174] Add MergingWindowAssigner and Ses...

2016-04-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1802#issuecomment-205394604 No, we introduced the `InternalWindowFunction` before 1.0.0 to decouple the user `WindowFunction` from the window operator implementation. There are now special

[GitHub] flink pull request: [FLINK-3654] Disable Write-Ahead-Log in RocksD...

2016-04-04 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1850 [FLINK-3654] Disable Write-Ahead-Log in RocksDB State You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink rocksdb-no-wal

[GitHub] flink pull request: [FLINK-3614] Remove Non-Keyed Window Operator

2016-03-31 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1805#issuecomment-203875675 Any thoughts on this? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-3756] [state] Add state hierarchy to Ch...

2016-04-14 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1883#discussion_r59687315 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/PendingCheckpoint.java --- @@ -139,13 +144,43 @@ public boolean acknowledgeTask

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60919358 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/CheckpointableInputFormat.java --- @@ -0,0 +1,51 @@ +/* + * Licensed

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60920236 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/CheckpointableInputFormat.java --- @@ -0,0 +1,51 @@ +/* + * Licensed

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60916945 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java --- @@ -398,16 +401,20 @@ public FileBaseStatistics

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1895#issuecomment-214358848 I had some inline comments but overall the changes look good! I think can simplify the `BinaryInputFormat` by getting rid of the `filePos

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60913386 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/CheckpointableInputFormat.java --- @@ -0,0 +1,51 @@ +/* + * Licensed

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60914650 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java --- @@ -40,36 +41,46 @@ import java.util.List

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-25 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r60915635 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java --- @@ -337,4 +402,18 @@ public int read(byte[] b, int off, int

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61402512 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunctionITCase.java --- @@ -0,0 +1,271

[GitHub] flink pull request: [FLINK-3840] Remove Testing Files in RocksDB B...

2016-04-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1942#issuecomment-215405047 Ok, thanks for checking! This can go into the 1.0.3 release then @uce. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61402951 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,225

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61402967 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,224

[GitHub] flink pull request: Refactor the whole file monitoring source to t...

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61403602 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,184

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r61444801 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/BinaryInputFormat.java --- @@ -284,18 +322,24 @@ public T nextRecord(T record) throws

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1895#discussion_r61445422 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java --- @@ -427,18 +434,37 @@ public void open(FileInputSplit split

[GitHub] flink pull request: FLINK-3717 - Be able to re-start reading from ...

2016-04-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1895#issuecomment-215460466 This looks good! 😃 I just had one inline question where I was hoping that you already checked it out and can provide some insights. --- If your project is set up

[GitHub] flink pull request: FLINK-3808: Refactor the whole file monitoring...

2016-04-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1929#issuecomment-215446453 I'm very happy with this now. 😃 Looks good to merge. Now for the checkpointable `InputFormat` and actually putting it into the API... --- If your project is set up

[GitHub] flink pull request: Timer coalescing across keys and cleanup of un...

2016-04-29 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1944#discussion_r61567670 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java --- @@ -542,14 +561,24 @@ public

[GitHub] flink pull request: [FLINK-3756] [state] Add state hierarchy to Ch...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1883#issuecomment-214683500 Also, I think we should rename the task business classes to accurately reflect what they are for. But this can be done once the rest of the state reworking

[GitHub] flink pull request: [FLINK-3756] [state] Add state hierarchy to Ch...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1883#issuecomment-214683587 👍 good to merge, then --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-3756] [state] Add state hierarchy to Ch...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1883#issuecomment-214680409 Also, we'll still call `StateForTask` like that, even though it's more like an operator state? And `KvStateForTasks` has the s at the end because it is theoretically

[GitHub] flink pull request: [FLINK-3741] [build] Replace maven-scala-plugi...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1931#issuecomment-214714799 @zentol only `flink-ml` used the outdated `maven-scale-plugin`, no idea why... --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] flink pull request: [FLINK-3815] Replace Guava Preconditions usage...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1932#issuecomment-214715290 👍 for merging. (I think you can merge all of your recently opened Preconditions-related PRs right away) --- If your project is set up for it, you can reply

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-26 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61063235 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunctionITCase.java --- @@ -0,0 +1,279

[GitHub] flink pull request: [FLINK-3756] [state] Add state hierarchy to Ch...

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1883#issuecomment-214678348 Unrelated, but shouldn't we consolidate the code in `SavepointCoordinator.restoreSavepoint()` and `CheckpointCoordiantor.restoreLatestCheckpointedState`? --- If your

[GitHub] flink pull request:

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/commit/879bb1bb029ed37e33adc7a655328940135cfcb3#commitcomment-17254978 Could do, but it does not really impact performance very much. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61228675 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,234

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61230608 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61231507 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61229319 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,234

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61230370 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,234

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61232860 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunctionTest.java --- @@ -0,0 +1,370

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61228546 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunction.java --- @@ -0,0 +1,234

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61230524 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61230544 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61231326 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61233316 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunctionITCase.java --- @@ -0,0 +1,279

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61233284 --- Diff: flink-streaming-java/src/test/java/org/apache/flink/streaming/api/functions/source/FileSplitMonitoringFunctionITCase.java --- @@ -0,0 +1,279

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61231656 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-27 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1929#discussion_r61232298 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FileSplitReadOperator.java --- @@ -0,0 +1,176

[GitHub] flink pull request: [FLINK-3740] Make Session Window State Checkpo...

2016-04-25 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/1878 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: Refactoring the File Monitoring Source.

2016-04-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1929#issuecomment-214664097 One additional remark. I'd like to get rid of these fields in `FileSplitMonitoringFunctionTest`: ``` private Set hdPaths = new HashSet<>(); priva

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-23 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2020#issuecomment-220988362 Thanks, the changes look good. R: @StephanEwen for taking a look at the API, you would only look at `StreamExecutionEnvironment`, for this. --- If your

[GitHub] flink pull request: [FLINK-3933] [streaming API] Add AbstractDeser...

2016-05-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2010#issuecomment-220293226 LGTM except for a minor comment in the doc --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] flink pull request: [FLINK-3933] [streaming API] Add AbstractDeser...

2016-05-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2010#issuecomment-220293427 Good addition that simplifies user's lives, I just saw someone on stack overflow who was struggling with how to get the `TypeInformation` for `getOutputType

[GitHub] flink pull request: [FLINK-3933] [streaming API] Add AbstractDeser...

2016-05-19 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/2010#discussion_r63859068 --- Diff: docs/apis/streaming/connectors/kafka.md --- @@ -142,18 +142,24 @@ for querying the list of topics and partitions. For this to work

[GitHub] flink pull request: [FLINK-3909] update Maven Failsafe version

2016-05-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2003#issuecomment-220272107 👍 and make sure that the tests were not silently failing with 2.17 --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request: [FLINK-3909] update Maven Failsafe version

2016-05-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2003#issuecomment-220269755 But maybe it only works since the exceptions are silently ignored and not forwarded? --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request: [FLINK-3761] Introduction of key groups

2016-05-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1988#issuecomment-220293946 Yeah, we were also wondering wether it would make sense to allow the state itself to be repartitioned, i.e. union and then split into the new parallelism. In this way

[GitHub] flink pull request: [FLINK-3933] [streaming API] Add AbstractDeser...

2016-05-19 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/2010#discussion_r63869803 --- Diff: docs/apis/streaming/connectors/kafka.md --- @@ -142,18 +142,24 @@ for querying the list of topics and partitions. For this to work

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-23 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/2020#discussion_r64221544 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/datastream/DataStreamSource.java --- @@ -44,6 +44,11 @@ public DataStreamSource

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-23 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/2020#discussion_r64220867 --- Diff: flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/FilePathFilter.java --- @@ -0,0 +1,74

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-23 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2020#issuecomment-220981894 All in all, very good work! One thing I'd like to change is the order of parameters in the `readFile` methods. For these telescoping methods is usual to append

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/2020#issuecomment-221227518 CC: @StephanEwen By the way, it might not look like it but the only additional methods this introduces on `StreamExecutionEnvironment` are are these three

[GitHub] flink pull request: [FLINK-2314] Make Streaming File Sources Persi...

2016-05-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/2020#discussion_r64392589 --- Diff: flink-tests/src/test/java/org/apache/flink/test/checkpointing/StreamFaultToleranceTestBase.java --- @@ -104,8 +106,25 @@ public void

<    4   5   6   7   8   9   10   11   12   13   >