Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
well..major runtime changes are coming with FLIP-6, 15 and 16 so I would
suggest you watch those. Loop FT will be included in one of these along with
other loop redesign features.
---
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
sweet! thanks @StefanRRichter
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/1668#discussion_r108166093
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamIterationHead.java
---
@@ -17,100 +17,164
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/1668#discussion_r108152267
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamIterationTail.java
---
@@ -64,6 +70,22 @@ public void init
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Thanks for the review @gyfora and @StephanEwen , these are very good points.
@StephanEwen makes sense to not really index/keep metadata of individual
records in log slices, it is extra
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey. Any update/opinion/something anyone?
Just a gentle reminder, sorry if this sounds a bit desperate :)
---
If your project is set up for it, you can reply to this email and have your
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
I have just rebased to the current master.
Please let me wrap this up. It has been over a year :)
Unregistering state in the OperatorStateStore is very tiny fix.
@StephanEwen
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey @addisonj.
Sure! You could perhaps review the changes and maybe see how to discard
empty operator states if you are motivated. This is the only pending issue for
this PR. thanks
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
The last update implements a variant of what @StephanEwen proposes. We have
put some more thought on this offline too thanks to @gyfora ! The idea is that
instead of putting records to each
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/3088
Hey! In this upcoming [PR ](https://github.com/apache/flink/pull/1668)
there is a custom operator responsible for the logging at the iteration head.
Hope this does not break your tests in case
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
@StephanEwen could you check my question above?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Working on it atm . I decided to make the following optimisations but want
to very quickly make sure that async checkpointing works the way I believe it
does:
- Most importantly, I am
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
agreed @StephanEwen! I will do that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
These are some good points @StephanEwen, thanks for checking it.
How about the following, regarding each issue:
- `Concurrent Checkpoints`: Looks like an improvement but I can sure
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Exactly, these two issues do not depend on each other. No doubt loop FT is
the first thing that can enable iterations in a production deployment so I
would merge that first.
Thank you
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey again, @StephanEwen @uce .
When you have 10 min can you take a look to see if this is acceptable?
I would not like to leave this here for months again, it has been out way
too long
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Ok, so I am progressing this a bit independently from the termination stuff
and then we rebase to the first PR that is merged. I just changed everything
and rebased to the current master
Github user senorcarbone commented on the issue:
https://github.com/apache/flink/pull/1668
Hey! Good to be back :) . Let's fix this properly, as @StephanEwen
recommended it now that there is some time.
We are writing together with @FouadMA a FLIP to address major loop fixes
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-214883298
ok good to know @uce! Let me get back to it in a couple of weeks and make
it right, now it is a bit impossible to find time.
---
If your project is set up
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-202818056
Thanks @StephanEwen and @uce for looking into it! I really appreciate it.
How about the following:
1. I update this PR with the patch that uses ListState
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-190279441
You can find an alternative version using `ListState` in the following
branch:
https://github.com/senorcarbone/flink/commits/ftloopsalt
So I noticed
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1668#issuecomment-187825999
Thanks @tillrohrmann for the feedback!
I merged `ForwardingOneInputTask` into `StreamIterationSink`.
One more think I missed pointing out is that when
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/1668
[FLINK-3257] Add Exactly-Once Processing Guarantees for Iterative
DataStream Jobs
# **[WIP]**
This is a first version of the adapted snapshot algorithm to support
iterations
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1537#issuecomment-174500162
Looks cool.
Just so I understand exactly, what is wrong again if the Coordinator simply
aborts expired checkpoint attempts? Wouldn't the protocol be the same
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1526#issuecomment-174486113
I guess this fix ok to merge. Any objections?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1526#issuecomment-173575376
I removed the groups from the ExecutionGraph. Hope it's good now.
---
If your project is set up for it, you can reply to this email and have your
reply appear
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1526#issuecomment-173260776
Sure, I will do it on the fly. Thanks for checking it!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/1526
[FLINK-3256] Fix colocation group re-instantiation
This PR deals with the problem of inconsistent colocation groups upon
reconfiguration. The problem was that we were removing shared
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/1239#discussion_r41493576
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java
---
@@ -322,73
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1239#issuecomment-146472181
on a related note, is anyone planning to complete snapshotting for cyclc
graphs for the 0.10 release? Unfortunately I do not have time to work on it
this week
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1239#issuecomment-146511645
@ktzoumas that's understandable! We could consider it for the maintenance
release maybe if there is enough time
---
If your project is set up for it, you can
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1208#issuecomment-145032778
Ah Indeed. I just reviewed this with the prospect of custom
pre-aggregations in mind and it seems like pre-aggregation strategies operate
on bucket-granularity
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1208#issuecomment-145025577
Nice doc so far ^^
One tiny fix on the *Advanced window constructs* subsection:
[0,1000], [100,1100], **[200,1200]**, ..., [1000, 2000
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1208#issuecomment-145028978
From a quick read on the documentation (and prior knowledge from google
dataflow) it is easy to get a full picture of the new semantics. Even though I
like it from
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1154#issuecomment-141934683
That's much better ^^
I approve :+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1150#issuecomment-141907506
At first I found it a bit odd to include serialized checkpointers on every
single statehandle but on a second look maybe that is the only way to
generalize
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/1147#issuecomment-141817879
wow! I also want to look into this in more detail, it looks promising.
@StephanEwen if you find a bit of time, can you write some more details to
guide
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/947#issuecomment-126054868
:+1: from me. looks that it does the job
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/937#issuecomment-126067262
That would be good ^^
then it's :+1: from me, at least for now.
It's generally good performance-wise to have less serialised states. This
means that we
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/900#discussion_r34352021
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/IterativeDataStream.java
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/900#issuecomment-120405183
There are quite a few changes in this PR! In general I am very keen to
eliminating DataStream mutations and adding that logic into the jobgraph
building phase
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/900#discussion_r34359293
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/IterativeDataStream.java
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/870#issuecomment-115724842
no more wrapper types :+1:
I approve
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/747#discussion_r33166608
--- Diff:
flink-staging/flink-streaming/flink-streaming-core/src/main/java/org/apache/flink/streaming/api/datastream/DataStream.java
---
@@ -301,16
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/747#issuecomment-114180337
We have added a `KeyedDataStream` type to the API as you can see in the
last commit. It is basically a supertype of `GroupDataStream` and represents a
`DataStream
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/816#issuecomment-110691858
:+1:
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/810#issuecomment-110406047
:+1: Thanks Gyula, this is pretty handy for ML tasks. If possible I would
recommend including this functionality to the undergoing release.
---
If your project
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/747#discussion_r31325009
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/jobgraph/tasks/CheckpointCommittingOperator.java
---
@@ -18,7 +18,10
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/676#discussion_r30636159
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/ByteStreamStateHandle.java
---
@@ -0,0 +1,77 @@
+/*
+ * Licensed
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/676#issuecomment-103640808
Apart from my comment on the transient field I think this is ok to merge.
Perhaps we can look into standardising the task manager backend configuration
afterwards
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/551#discussion_r27763870
--- Diff:
flink-runtime/src/main/scala/org/apache/flink/runtime/jobmanager/StreamCheckpointCoordinator.scala
---
@@ -82,15 +81,18 @@ class
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/551
[FLINK-1808] Send barrier requests only when the execution graph is running
This is a simple optimisation for the current StreamCheckpointCoordinator
that makes it skip barriers whent he
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/524#issuecomment-85223415
bad style habits ^^
thanks for reminding me. I also made a minor fix to the test too.
---
If your project is set up for it, you can reply to this email and have
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/524
[Flink-1686] Streaming Iterations Slot Sharing Fix
Iteration head instances were using the same identifier for slot sharing
which resulted into incorrect execution for streaming iterations. I
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/501#issuecomment-83774008
@StephanEwen @uce I definitely agree, thanks for the feedback!
---
If your project is set up for it, you can reply to this email and have your
reply appear
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/501
Flink 1740
We now use the config parameter for numberofExecutionRetries on the
generated job graph.
Also piggypacked a minor fix for a compilation error that currently occurs
in flink
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/501#issuecomment-83197775
Indeed, cool. Good to know
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user senorcarbone commented on a diff in the pull request:
https://github.com/apache/flink/pull/501#discussion_r26711053
--- Diff:
flink-tests/src/test/java/org/apache/flink/test/javaApiOperators/GroupCombineITCase.java
---
@@ -58,7 +58,7 @@
*/
public class
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/501#issuecomment-83196555
Thanks @mbalassi! If you merge can you remove the quickfix (2872308)? It
looks like @hsaputra pushed it already
---
If your project is set up for it, you can
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/459#issuecomment-77781300
thanks for the review Stephan! The synchronous call to executionGraph
restart was a very sneaky thing, I committed it by mistake while trying some
things out
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/459#issuecomment-77538649
hold on, will commit an update very soon. :P
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/432
[FLINK-1586] [streaming] Add support for iterative streaming graphs on JSON
generation
I refactored a bit the current JSON generation for streaming job graphs to
further allow iterations
Github user senorcarbone commented on the pull request:
https://github.com/apache/flink/pull/395#issuecomment-74393323
+1 for merging asap. It's an improvement on both semantics and performance
prospects for windowing.
---
If your project is set up for it, you can reply
GitHub user senorcarbone opened a pull request:
https://github.com/apache/flink/pull/341
[FLINK-1344] [streaming] Added static StreamExecutionEnvironment
initialisation and Implicits for scala sources
This PR addresses the ticket [1] for further scala constructs
interoperability
64 matches
Mail list logo
