[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-06-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/742#issuecomment-108740723 I will run some benchmarks, then we can decide about that. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] flink pull request: [FLINK-2191] Fix inconsistent use of closure c...

2015-06-10 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/813 [FLINK-2191] Fix inconsistent use of closure cleaner in Scala Streaming The closure cleaner still cannot be disabled for the Timestamp extractor in Time and for the delta function in Delta

[GitHub] flink pull request: [FLINK-1962] Add Gelly Scala API

2015-06-18 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/808#issuecomment-113150313 Yes, the formatting would indeed have to be changed to conform to the pre-existing Flink Scala style. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request: [FLINK-2230] handling null values for TupleSer...

2015-06-25 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/867#issuecomment-115248743 @StephanEwen hinted that the best way to go would be to decuple the RowTypeInfo completely from the TupleTypeInfo/TupleSerializerBase. This way, we get null-value

[GitHub] flink pull request: New operator state interfaces

2015-06-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/747#issuecomment-114797858 @senorcarbone You mentioned that the KeyedDataStream disallows calls to shuffle()/repartition()/groupBy() and so on. But KeyedDataStream doesn't seem to have

[GitHub] flink pull request: New operator state interfaces

2015-06-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/747#discussion_r33129995 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/KeyedDataStream.java --- @@ -0,0 +1,78

[GitHub] flink pull request: New operator state interfaces

2015-06-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/747#discussion_r33129956 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java --- @@ -301,16 +301,68

[GitHub] flink pull request: New operator state interfaces

2015-06-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/747#issuecomment-114803586 Ahh I see, my bad. :sweat_smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request: [FLINK-2261] [streaming] Remove reduce/fold/ag...

2015-06-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/860#discussion_r33139196 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/operators/StreamFoldTest.java --- @@ -1,57 +0,0

[GitHub] flink pull request: New operator state interfaces

2015-06-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/747#discussion_r33164735 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/state/PartitionedStreamOperatorState.java

[GitHub] flink pull request: [FLINK-1962] Add Gelly Scala API

2015-06-23 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/808#issuecomment-114412170 ExecutionEnvironment.scala, UnionOperator.java and Graph.java have a lot of whitespace/formatting changes. This makes it hard to see the actual changes to functionality

[GitHub] flink pull request: [FLINK-2279] [streaming] Add Connected Iterati...

2015-06-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/870#issuecomment-115750835 This looks reasonable. How does the user not specify which elements are feedback and which go to the downstream operator? --- If your project is set up for it, you can

[GitHub] flink pull request: [FLINK-2261] [streaming] Remove reduce/fold/ag...

2015-06-27 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/860#issuecomment-11616 +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] flink pull request: [FLINK-2279] [streaming] Add Connected Iterati...

2015-06-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/870#issuecomment-115762324 Ah ok, I was just confused since in the new test for it the same stream is used as feedback and result. :sweat_smile: --- If your project is set up for it, you can

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-15 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-112116663 Yes, I see your point. Also, right now an expression like a.count, a.sum (where a only contains 0 or 1) returns different results if field a is null for some rows

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-15 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-11231 But right now the jobs fail with NPEs, that's also not very nice. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-16 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-112566051 Thanks for your work. :smile: Could you please close the PR manually, I forgot to add a tag for closing it in the commit. --- If your project is set up

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/831#issuecomment-111717366 I had some more remarks, sorry for being so picky. :sweat_smile: Other than that, I think the change looks really good now! --- If your project is set up

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/831#discussion_r32371128 --- Diff: flink-staging/flink-table/src/test/scala/org/apache/flink/api/table/typeinfo/RowSerializerTest.scala --- @@ -0,0 +1,70 @@ +/* + * Licensed

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/831#discussion_r32371124 --- Diff: flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/typeinfo/RowSerializer.scala --- @@ -102,20 +119,39 @@ class RowSerializer

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/831#discussion_r32369782 --- Diff: flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/typeinfo/RowSerializer.scala --- @@ -89,11 +99,16 @@ class RowSerializer

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/831#discussion_r32369781 --- Diff: flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/typeinfo/RowSerializer.scala --- @@ -74,11 +79,16 @@ class RowSerializer

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/831#issuecomment-111701489 The changes look good except for comments I had about the loops. For the tests, did you try doing it as in TraversableSerializerTest.scala. Here, we override

[GitHub] flink pull request: [FLINK-2203] handling null values for RowSeria...

2015-06-13 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/831#issuecomment-111741260 Thanks, nice work. :+1: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-16 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-112316099 Very good, I would have forgotten about the test case with all-null values. :smiley: I only had a minor comment about Boolean.compare() --- If your project

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-16 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/834#discussion_r32495592 --- Diff: flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/codegen/ExpressionCodeGenerator.scala --- @@ -489,6 +489,25 @@ abstract class

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-16 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/834#discussion_r32495598 --- Diff: flink-staging/flink-table/src/main/scala/org/apache/flink/api/table/codegen/ExpressionCodeGenerator.scala --- @@ -489,6 +489,25 @@ abstract class

[GitHub] flink pull request: [FLINK-2221] [docs] Docs for not using local f...

2015-06-15 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/839#issuecomment-112161893 Looks good +1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-15 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-112127936 OK, I had some offline discussion with @mxm and @fhueske. We thought that we should have this behaviour: https://en.wikipedia.org/wiki/Null_(SQL)#Aggregate_functions

[GitHub] flink pull request: [hotfix] Some small fixups in README.md

2015-06-16 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/841 [hotfix] Some small fixups in README.md You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink readme-fixups Alternatively you can

[GitHub] flink pull request: [FLINK-2210] Table API support for aggregation...

2015-06-15 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/834#issuecomment-112295698 I think we need to have a special AST node along with code generation for it. This means: - Addition of something like NumericIsNotNull in comparisons.scala

[GitHub] flink pull request: Implemented TwitterSourceFilter and adapted Tw...

2015-06-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/695#issuecomment-107422496 Is this good to merge now? I'm rewriting all the sources (again) and this should probably go in. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-06-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/742#issuecomment-107388477 As per the discussion on the mailing list I'm rewriting the Source interface to only have the run()/cancel() variant. --- If your project is set up for it, you can

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-06-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/742#issuecomment-107589378 I reworked the sources now. Could someone please have another pass over this. I think this is very critical code. --- If your project is set up for it, you can reply

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-06-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/742#discussion_r31463699 --- Diff: flink-staging/flink-streaming/flink-streaming-connectors/flink-connector-twitter/src/main/java/org/apache/flink/streaming/connectors/twitter

[GitHub] flink pull request: [FLINK-2096] Remove implicit conversion from W...

2015-05-28 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/726 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: [FLINK-2096] Remove implicit conversion from W...

2015-05-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/726#issuecomment-106237798 manually merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-05-28 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/742 [FLINK-2098] Ensure checkpoints and element emission are in order Before, it could happen that a streaming source would keep emitting elements while a checkpoint is being performed. This can

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-05-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/742#issuecomment-106471093 This also enables the exactly once checkpointing test added earlier by @StephanEwen. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request: [FLINK-2069] Fix Scala CSV Output Format

2015-06-02 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/759 [FLINK-2069] Fix Scala CSV Output Format Before, the Scala Streaming API used the Java CsvOutputFormat. This is not compatible with Scala Tuples. The FileSinkFunction would silently swallow

[GitHub] flink pull request: [FLINK-2103] Expose partitionBy to user

2015-05-28 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/743 [FLINK-2103] Expose partitionBy to user You can merge this pull request into a Git repository by running: $ git pull https://github.com/aljoscha/flink user-partition-by Alternatively you can

[GitHub] flink pull request: [FLINK-2098] Ensure checkpoints and element em...

2015-06-01 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/742#issuecomment-107811407 I will fix the remaining cancel() races. The twitter stuff I just commented out because I assumed that the new TwitterSource could be merged right away and I

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-01 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33688371 --- Diff: flink-core/src/main/java/org/apache/flink/api/common/ExecutionConfig.java --- @@ -141,6 +144,25 @@ public boolean isClosureCleanerEnabled

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-01 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/879 [FLINK-1967] Introduce (Event)time in Streaming (This PR consists of other commits because of bugs discovered while writing tests for the new feature and existing features

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33756016 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/functions/source/ManualTimestampSourceFunction.java

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33864220 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33864229 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33868015 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33864328 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33868988 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-117987956 But if you do any sort of time windowing, you have to get the time for each record at some point. So in my opinion it does not matter where we do it so we might as well

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-117993213 Forget about the performance numbers I posted. When running the same benchmark multiple times without shutting down the cluster the performance degenerates from about

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-117976364 The timestamp is serialised as one `long`. The problem with sending watermarks as records is that they need to be broadcast. I think it's not possible right now

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-117979128 Ok, I thought JodaTime was the de-facto time library for Java. I can change it of course. --- If your project is set up for it, you can reply to this email and have

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-118037363 Ok, the test results I posted earlier seem correct. I also ran the tests on my machine with two TaskManagers, There the result is the same for master/event-time/event

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-118037541 Also, when using TaskManagers with 512mb the performance degrades heavily when running successive tests, seems to be related to garbage collection. (On both master

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-03 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/879#discussion_r33877140 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/streamrecord/StreamRecord.java --- @@ -17,87

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-02 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-118058537 I ran some more numbers on the cluster. With watermark interval 1 Second there is no noticeable difference in performance. I will now investigate sending the watermarks

[GitHub] flink pull request: [FLINK-1967] Introduce (Event)time in Streamin...

2015-07-06 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/879#issuecomment-11844 I thought about how we could change it so that the Watermarks are transmitted in-band with the stream records. This would require the StreamRecordSerializer to handle

[GitHub] flink pull request: [FLINK-2074] Fix erroneous emission of Sliding...

2015-05-25 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/716#issuecomment-105233972 @rmetzger I added tests for both SlidingTimePreReducer and SlidingTimeGroupedPreReducer to verify that they correctly flush their states after a burst of elements

[GitHub] flink pull request: [FLINK-2074] Fix erroneous emission of Sliding...

2015-05-26 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/716 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: [FLINK-1907] Scala shell

2015-05-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/672#issuecomment-105504716 Do we want to have this in now? I can cherry pick it on top of the latest master and merge it. --- If your project is set up for it, you can reply to this email

[GitHub] flink pull request: [Flink-1985] Streaming does not correctly forw...

2015-05-21 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/682#issuecomment-104245060 +1, can you do it @rmetzger ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] flink pull request: Some fixes for Scala type analysis

2015-05-21 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/669#issuecomment-104207389 This excludes static fields in Scala Pojo analysis (because static fields should not be serialised/deserializerd), removes legacy code from Scala Type Descriptors

[GitHub] flink pull request: [FLINK-2074] Fix erroneous emission of Sliding...

2015-05-22 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/716 [FLINK-2074] Fix erroneous emission of Sliding Time PreReducer Before this, a sliding time window would keep emitting the last result because the number of elements per pre-aggregation result

[GitHub] flink pull request: [FLINK-2074] Fix erroneous emission of Sliding...

2015-05-22 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/716#issuecomment-104706869 I don't know, all this time we had the wrong behaviour and no tests were catching it. --- If your project is set up for it, you can reply to this email and have your

[GitHub] flink pull request: [FLINK-2096] Remove implicit conversion from W...

2015-05-26 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/726 [FLINK-2096] Remove implicit conversion from Window Stream to Stream This causes unexpected behaviour, for example in: ds.window(...).map(..) The user would expect some kind

[GitHub] flink pull request: [FLINK-2069] Add flush() in CsvOutputFormat.wr...

2015-05-26 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/724 [FLINK-2069] Add flush() in CsvOutputFormat.writeRecord() This was causing problems in Streaming because the output would not appear or not be immediately visible. You can merge this pull

[GitHub] flink pull request: [FLINK-1907] Scala shell

2015-05-26 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/672#issuecomment-105530009 Ok, I'm running the last Travis tests and then I'm merging :smile: --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request: [FLINK-2462] [streaming] Major cleanup of stre...

2015-08-17 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1017#discussion_r37168797 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/SourceStreamTask.java --- @@ -39,61

[GitHub] flink pull request: [FLINK-2462] [streaming] Major cleanup of stre...

2015-08-17 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1017#issuecomment-131735854 This looks like a very nice continuation of the cleanup work. I'd suggest to merge it rather sooner than later. --- If your project is set up for it, you can reply

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-17 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/988#issuecomment-131892889 I changed it to execute dangling operators now. There is, however, a strange feature. This code works on master: https://gist.github.com/aljoscha/bbe74309a31a16ca8413

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/988#issuecomment-132585797 I also updated the documentation to reflect the changes in shipping strategies/partitioning. --- If your project is set up for it, you can reply to this email and have

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/988#issuecomment-132582717 What I did is basically adding every operator to the list of sinks. When `execute` is called the translation starts from every operator, which is ok, since an operator

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-19 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/988#issuecomment-132689802 Manually merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-19 Thread aljoscha
Github user aljoscha closed the pull request at: https://github.com/apache/flink/pull/988 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-21 Thread aljoscha
GitHub user aljoscha opened a pull request: https://github.com/apache/flink/pull/1040 [FLINK-2558] Add Streaming Connector for Elasticsearch This adds an ITCase, an example and documentation. You can merge this pull request into a Git repository by running: $ git pull https

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-21 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1040#issuecomment-133465114 I'm currently in the process of setting up a cluster to test it with an actual Elasticsearch cluster. Once I have the results of that I'll post them here

[GitHub] flink pull request: [FLINK-2555] Properly pass security credential...

2015-08-21 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1038#issuecomment-133505584 The Bases exist because there is a java-specific and a scals-specific version of each HadoopInputFormat. --- If your project is set up for it, you can reply

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-21 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1040#issuecomment-133515409 I'm happy to report that I managed to set up a distributed Elasticsearch cluster and the sink did work with it :smile: --- If your project is set up for it, you can

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1040#issuecomment-134197297 I updated the doc, if no-one objects I would like to merge this tonight. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...

2015-08-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1048#discussion_r37749500 --- Diff: flink-clients/src/main/java/org/apache/flink/client/LocalExecutor.java --- @@ -176,8 +175,7 @@ public JobExecutionResult executePlan(Plan plan

[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...

2015-08-24 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/1048#issuecomment-134206474 Looks good, modulo some minor comments. But maybe @tillrohrmann has something to say, let's wait for him. --- If your project is set up for it, you can reply

[GitHub] flink pull request: [FLINK-2543] Fix user object deserialization f...

2015-08-24 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1048#discussion_r37752056 --- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/util/SerializedThrowable.java --- @@ -0,0 +1,106 @@ +/* + * Licensed to the Apache

[GitHub] flink pull request: [FLINK-2398][api-breaking] Introduce StreamGra...

2015-08-18 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/988#issuecomment-132260208 I think this is good now. I adapted the Streaming Tests to always use `StreamingMultipleProgramTestBase` when appropriate. The earlier problems where caused by some

[GitHub] flink pull request: [FLINK-2462] [streaming] Major cleanup of stre...

2015-08-17 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1017#discussion_r37166759 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/TwoInputStreamTask.java --- @@ -27,114

[GitHub] flink pull request: [FLINK-2462] [streaming] Major cleanup of stre...

2015-08-17 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1017#discussion_r37167205 --- Diff: flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/operators/Output.java --- @@ -33,9 +33,8

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-22 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1040#discussion_r37697987 --- Diff: docs/apis/streaming_guide.md --- @@ -1517,11 +1517,24 @@ Stream connectors !-- TODO: reintroduce flume

[GitHub] flink pull request: [FLINK-2558] Add Streaming Connector for Elast...

2015-08-22 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/1040#discussion_r37698000 --- Diff: docs/apis/streaming_guide.md --- @@ -1661,6 +1674,165 @@ More about Kafka can be found [here](https://kafka.apache.org/documentation.html

[GitHub] flink pull request: [FLINK-2231] Create a Serializer for Scala Enu...

2015-07-28 Thread aljoscha
Github user aljoscha commented on a diff in the pull request: https://github.com/apache/flink/pull/935#discussion_r35631275 --- Diff: flink-tests/src/test/scala/org/apache/flink/api/scala/runtime/ScalaSpecialTypesSerializerTest.scala --- @@ -93,8 +98,11 @@ class

[GitHub] flink pull request: [FLINK-2231] Create a Serializer for Scala Enu...

2015-07-28 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/935#issuecomment-125536383 Looks good, any objections to me merging this? I can fix the one comment I had about using serializer.getLength when I merge it. --- If your project is set up

[GitHub] flink pull request: [FLINK-2324] [streaming] Partitioned state che...

2015-07-29 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/937#issuecomment-126070524 I was too busy but I would like to take a quick look. I will look at it tomorrow morning. --- If your project is set up for it, you can reply to this email and have

[GitHub] flink pull request: [FLINK-2324] [streaming] Partitioned state che...

2015-07-30 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/937#issuecomment-126296064 I think the changes are ok for now but we need to think hard about how we can unify incremental state, partitioned state, out-of-core state and scale-out

[GitHub] flink pull request: [FLINK-2324] [streaming] Partitioned state che...

2015-07-30 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/937#issuecomment-126302762 Yes, of course, that's nice. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request: [FLINK-2477][Add]Add tests for SocketClientSin...

2015-08-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/977#issuecomment-127533577 Hi, you don't need to create a new Pull Request. You can also just push your changes to your old branch (HuangWHWHW:FLINK-2477 in this case) and then the Pull

[GitHub] flink pull request: [FLINK-2477][Add]Add tests for SocketClientSin...

2015-08-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/977#issuecomment-127536235 Don't worry, everyone was a novice once. :smile: You can do it either way. Either close this PR and we'll look at the other one or close the other one

[GitHub] flink pull request: [FLINK-2477][Add]Add tests for SocketClientSin...

2015-08-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/977#issuecomment-127541271 You're welcome. And yes the streaming API is quite a bit newer than the other code so it is not that well covered with tests. If you want to, you can open a Jira Issue

[GitHub] flink pull request: [FLINK-2477][Add]Add tests for SocketClientSin...

2015-08-04 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/977#issuecomment-127522856 Hi, you can rerun the tests on Travis by changing your commit slightly and re-pushing it to your branch, for example using: ```bash git commit --amend

[GitHub] flink pull request: [FLINK-2231] Create a Serializer for Scala Enu...

2015-07-29 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/935#issuecomment-125936680 Merged, thanks for your work. :smile: --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] flink pull request: [FLINK-1851] Add support for casting in Table ...

2015-08-05 Thread aljoscha
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/592#issuecomment-128130305 Correct, nothing is added. I need to spent some time on this when I'm back from vacation. --- If your project is set up for it, you can reply to this email and have

<    1   2   3   4   5   6   7   8   9   10   >