Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1605#discussion_r52469417
--- Diff:
flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBFoldingState.java
---
@@ -0,0 +1,175
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1605#issuecomment-182421429
I addressed the comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1617#issuecomment-182421979
Yes @mjsax you have the right ideas here. :smile: Those are the things that
need to be figured out.
---
If your project is set up for it, you can reply to this email
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1605#issuecomment-182583799
You can also do just a "fold". The window function is for when you need the
meta information about the window because the fold function does not get that.
-
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1606#issuecomment-182776315
Some of the internal sinks sources, such as File* Socket are Internal while
others are PublicEvolving. Is there a reason behind this?
---
If your project is set up
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1606#issuecomment-182778504
I see. Might seem inconsistent, though. :smile: So maybe we should make all
of them PublicEvolving.
---
If your project is set up for it, you can reply to this email
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1629
[FLINK-3359] Make RocksDB File Copies Asynchronous
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink state-async-kv
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1629#issuecomment-183325582
@uce This also moves the determination of the state size in
`StreamTaskStateList` from the constructor to the method `getSize()`. I had to
do this because the size of
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1513#issuecomment-183326898
Any more comments? Ideas? I think we should merge this soon because it
should go into the 1.0 release.
---
If your project is set up for it, you can reply to this
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1631
[FLINK-3352] Use HDFS Config in RocksDB Copy Utilities
This also moves the utilities (HDFSCopyFromLocal and HDFSCopyToLocal) to
the RocksDB package because we would need a HDFS dependency in
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1563#issuecomment-183414977
I think the `writeOutputFormat` method name could be misleading. To me it
implies writing the `OutputFormat` not writing something by using the
`OutputFormat
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1631#issuecomment-183418609
We could write it only once, but is it worth it? It would require piping
another parameter through all the rocksdb related classes and I would assume
there is only one
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1630#issuecomment-183428695
Thanks for the fix. I'll merge as soon as travis is all green.
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1631#issuecomment-183432667
Ahhh, rebased on latest master changes again (folding window state...).
:sweat_smile:
---
If your project is set up for it, you can reply to this email and have your
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1631
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1629
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1630#issuecomment-183506103
I merged it. Could you please close the PR if github doesn't.
---
If your project is set up for it, you can reply to this email and have your
reply appear on G
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1638#issuecomment-184192170
Thanks :smile: You tried this on a cluster, I guess?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1638#issuecomment-184195688
Ok, could you please go ahead and merge it if no one else objects. :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1638#issuecomment-184196345
thanks
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1513#issuecomment-184196790
I rebased on top of current master.
Btw, still still has the issue that the time characteristic only applies to
window operations that are created after
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1641
[FLINK-3315] Fix Slot Sharing in Streaming API
This changes slot sharing settings to single method
slotSharingGroup(String) on DataStream.
Operations inherit the slot sharing group of
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1641#discussion_r53017220
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/transformations/StreamTransformation.java
---
@@ -202,6 +203,29 @@ public
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1513#issuecomment-184715239
@rmetzger @StephanEwen Do you still have objections? I think this should go
in ASAP.
---
If your project is set up for it, you can reply to this email and have your
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1644
[FLINK-3400] Move RocksDB Copy Utils to flink-streaming-java
They are not specific to RocksDB, just utilities for copying local
folders to/from HDFS. Moving them to flink-streaming-java means
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1645
Refactor Common Parts of Stream/Batch Documentation
The main bit is the refactoring, I also added a basic RocksDB documentation
and created a new "Working with Time" section for the
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1645#discussion_r53139897
--- Diff: docs/apis/common/index.md ---
@@ -0,0 +1,1356 @@
+---
+title: "Basic Concepts"
+
+# Top-level navigation
+top-nav-g
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1644
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1659
[FLINK-3436] Remove ComplexIntegrationITCase
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink complex-remove
Alternatively you
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1645#issuecomment-185126063
@uce Yes, they could also be moved to separate pages.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1641#issuecomment-185230357
rebasing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1655
[FLINK-3430] Remove "no POJO" warning in TypeAnalyzer
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink scala-remove-po
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1645#issuecomment-185132226
Manually merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1513
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1645
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1513#issuecomment-185132200
Manually merged
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1648#issuecomment-185083603
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1646#issuecomment-185083361
+1 This should go into 1.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1665#issuecomment-185636882
Anyone against merging this as a (maybe temporary) solution?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1659
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1677
[FLINK-3439] Remove final Long.MAX_VALUE Watermark in StreamSource
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink watermark
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1677#issuecomment-186605514
I added the checks, will merge later once travis runs through.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1686
[FLINK-3450] Duplicate TypeSerializer in StateDescriptor.writeObject
The StateDescriptor can be serializer asynchronously in case of
asynchronous checkpoints. In that case two threads would try
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1677
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1686
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1687
[FLINK-3418] Don't run RocksDB copy utils in external process
This was causing to many problems with security tokens and yarn. Now,
let the RocksDB backup run in a thread but
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1460
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1460#issuecomment-187187459
This will have to be reworked now that windows use the keyed state
abstraction.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1617#issuecomment-187201289
I don't know enough about redis. Not sure how we can deal with the testing
situation.
---
If your project is set up for it, you can reply to this email and have
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1687#discussion_r53629014
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/util/HDFSCopyFromLocal.java
---
@@ -26,32 +25,46 @@
import java.io.File
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1687#issuecomment-187202268
Should be the correct code now. :smile:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1641#issuecomment-187190568
Updated on top of master again. This should have the behavior that we want
for 1.0.
---
If your project is set up for it, you can reply to this email and have your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1687#issuecomment-187328780
Ahh, stupid beginners mistake. Fixed it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1617#issuecomment-187716265
Have you thought about how the state backups would work if there was only
one previously running instance of redis available?
---
If your project is set up for it
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1641#discussion_r53811188
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraphGenerator.java
---
@@ -533,4 +563,33 @@ private StreamGraph
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1641#discussion_r53817674
--- Diff:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/SlotAllocationTest.java
---
@@ -40,17 +48,142 @@ public void test
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1702
[hotfix] Add logging for RocksDB backup durations
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink rocksdb-time-logging
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1703#issuecomment-188204348
+1, looks good to merge
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1702
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1699#issuecomment-188322336
The comment above is not valid anymore, right?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1699#issuecomment-188322569
About not allowing negative timestamps/watermarks and so on.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1706
[hotfix] Expose current watermark to Triggers
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink window-watermark-in-trigger
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1699#discussion_r53984915
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/IngestionTimeExtractor.java
---
@@ -0,0 +1,40
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1699#discussion_r53985422
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java
---
@@ -572,7 +577,7 @@ public JobGraph getJobGraph
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1699#discussion_r53985538
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamGraph.java
---
@@ -572,7 +577,7 @@ public JobGraph getJobGraph
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1699#discussion_r53986228
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/TimestampedCollector.java
---
@@ -51,17 +48,19 @@ public
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1699#issuecomment-188406380
+1 changes look good, I had some minor comments but that's just naming
preferences.
Btw, this PR also removes the distinction between `EventTimeSourceFun
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1699#issuecomment-188424651
That's fine with me. As I said, that's just personal preferences.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1699#issuecomment-188431435
There's also some commented out code but I guess we'll leave that in until
we fix the timestamp handling?
---
If your project is set up for it, you can rep
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1706#issuecomment-188448934
I want to add this because it enables triggers that can essentially discard
late elements. It does not change any previous behavior but just enables some
nifty custom
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1712#issuecomment-188858495
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1636#issuecomment-188874847
I would prefer to just add the compassable way, yes. Sorry that you now
already put effort into this.
---
If your project is set up for it, you can reply to this
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1723
[FLINK-3521] Make Iterable part of method signature for WindowFunction
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink window
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1727#issuecomment-189445750
It worked before because of this setProcessingTime flag in the
WindowOperator that would put the current processing time into that timestamp
field.
The
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1727#issuecomment-189447001
It was removed in the first commit that changed the interplay with
StreamTimeCharacteristic
---
If your project is set up for it, you can reply to this email and have
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1728
[FLINK-3527] Add Scala DataStream.transform()
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink scala-transform
Alternatively
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1723#issuecomment-189458811
Yes, this is correct. (about the fold() on non-keyed windows)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1723#issuecomment-189462738
The `PassThroughFunction` is created because at that point it is not yet
clear what type of `InternalWindowFunction` will be created. I can fix it by
duplicating all
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1723#issuecomment-189463014
1. Is very hard to fix, ditching the non-keyed window operators altogether
and using a dummy key selector would be easier.
---
If your project is set up for it, you
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1728
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1739
[FLINK-3536] Make clearer distinction between event time and processiâ¦
â¦ng time
This brings it more in line with *ProcessingTimeWindows and makes it
clear what type of window
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1740
[FLINK-3538] Streamline Scala DataStream.join/coGroup
This enforces that the user always has to specify keys for both inputs
before .window() can be called.
You can merge this pull request
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1740#issuecomment-190315085
I think it's simply "api breaking". Will add the tag and merge. ok? Will
then also put it on the release-1.0 branch.
---
If your project is set up
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1739#issuecomment-190319471
I think it would be good to have it in, yes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/1742
[FLINK-3548] [api-breaking] Remove unnecessary generic parameter fromâ¦
⦠SingleOutputStreamOperator
You can merge this pull request into a Git repository by running:
$ git pull https
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1740
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1739
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha closed the pull request at:
https://github.com/apache/flink/pull/1742
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1562#issuecomment-190649095
Hi @wenlonglwl, the reason for this were mostly practical concerns. We want
to use our partitioned state abstraction for the window state because that
makes it easy to
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1542#issuecomment-190650790
Hi @wenlonglwl,
you are right about this. We are thinking about how we can change the
system so that finite sources can emit a final Long.MAX_VALUE watermark when
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1744#issuecomment-190660978
I think we might abuse the `of()` name a bit to much. Maybe `from()` or
`for()` would be more appropriate here. So it would be `TypeInformation.for(new
TypeHint
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1744#issuecomment-190661279
But otherwise, big +1 for this change. :smile:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1745#issuecomment-190666910
Looks good :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1745#issuecomment-190667026
With two small inline comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user aljoscha commented on a diff in the pull request:
https://github.com/apache/flink/pull/1748#discussion_r54560116
--- Diff:
flink-streaming-scala/src/test/scala/org/apache/flink/streaming/api/scala/WindowFoldITCase.scala
---
@@ -70,7 +70,7 @@ class WindowFoldITCase
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1748#issuecomment-190702497
Very good fix. :+1: If we have another RC I would like to include this in
1.0
---
If your project is set up for it, you can reply to this email and have your
reply
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1750#issuecomment-190735172
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1750#issuecomment-191260012
Hi,
what are you referring to? The watermark does not interact with count
windows.
---
If your project is set up for it, you can reply to this email and have your
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1562#issuecomment-191681857
It is certainly possible to find some way of doing it but it is not
straightforward. So for now I wanted to keep it simple until we figure out a
good way to do it
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/1750#issuecomment-191693706
Yes, you are right and we are thinking about it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
1201 - 1300 of 4534 matches
Mail list logo
