Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
@aljoscha yes, i review the code this day, it will jude each window whether
late , so the previous method i use will counts more lost data than the actual
situation , i have fix the error and re
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
@aljoscha i agree that the name set to be "numLateElementsDropped", and do
you mean that my result should minus the num of element that go to side output
which is skipped and lateElement?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
Hello,Is it can be merged in @aljoscha @zentol ?
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r140634868
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -405,6 +411,8 @@ public void
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r140635202
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/EvictingWindowOperator.java
---
@@ -231,6 +231,8
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r138789377
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -132,6 +133,13
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r138778844
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -132,6 +133,13
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r139742368
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -405,6 +411,8 @@ public void
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r139933981
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -405,6 +411,8 @@ public void
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/4665
[Flink-7611]add metrics to measure the num of data dropped due to the data
arrived late
## What is the purpose of the change
1. add metrics to measure the num of data dropped due
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r138515752
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -132,6 +133,13
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4665#discussion_r138515541
--- Diff:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java
---
@@ -132,6 +133,13
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
is there still anything wrong @zentol
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
is this ok to be merged in @bowenli86 @zentol @aljoscha ? ð
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4935
ping @tzulitai ~
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4935
Hi, @tzulitai
After i read kafkaConsumer code again, i found that the per partition kafka
lag metric is register in method `FetchManagerMetrics#recordPartitionLag` But
the when the client get
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4935#discussion_r148738495
--- Diff:
flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4935#discussion_r148738023
--- Diff:
flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/4935
[Flink-7945][Metrics]Fix per partition-lag metric lost in kafka
connector
## What is the purpose of the change
*When used KafkaConnector, we cant get per partition lag metric. But it has
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4935
cc @zentol @tzulitai please help review the code.
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4935
Hi @tzulitai , could you take look at this again :-) ?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4960
ð
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/4935#discussion_r148530767
--- Diff:
flink-connectors/flink-connector-kafka-0.9/src/main/java/org/apache/flink/streaming/connectors/kafka/internal/KafkaConsumerThread.java
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4935
update the code according to the comment. ping @tzulitai
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4665
Hi @zentol , can you help merge this pr ? Is this be forgetten
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/4878
Yes, i will close this one ~
---
Github user Aitozi closed the pull request at:
https://github.com/apache/flink/pull/4878
---
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/4878
[FLINK-7895][hotfix][docs]Fix error in example in get late message in
window doc
* getSideOutput api is only available in SingleOutputOperator class, and is
not the part of the base class
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6080
[Flink-9443]Remove unused parameter in generateNodeLocalHash
After Flink1.2 it used StreamGraphHasherV2 to generate hashes, The method
generateNodeLocalHash dont use the information like (parallel
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6124
Got it. Thanks for your response @dawidwys and waiting for you further
comment on this PR. I am glad to contribute to the cep library and really hope
to get more ideas or share from you guys
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6171
Hi, @dawidwys can you explain a little about how does the semantics of
`AfterMatch` differ from previous implementation, I read the doc and feel a
lille confused. thx ;-)
---
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6168
[FLINK-9588]Reused context with same computation state calculate
## What is the purpose of the change
Now cep checkFilterCondition with a newly created Conditioncontext for each
edge, which
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6168#discussion_r195737755
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -697,6 +697,7 @@ private int calculateIncreasingSelfState(int
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6168#discussion_r195738359
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -697,6 +697,7 @@ private int calculateIncreasingSelfState(int
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6168#discussion_r195745115
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -609,7 +611,7 @@ public void close() throws Exception
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6168#discussion_r195792448
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -609,7 +611,7 @@ public void close() throws Exception
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6162
Thanks for your review , remove the same block in `onProcessingTime`.
---
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6162
[FLINK-9579][CEP]Remove unneeded clear on elementQueueState
## What is the purpose of the change
Remove unneeded clear on elementQueueStateï¼ when soretedTimestamp is
empty, the elements
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6162
Is this OK? @dawidwys
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6168
please help review this pr @dawidwys , thx.
---
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6104
[FLINK-9476]Emit late elements in CEP as sideOutPut
Now, when use with Eventtime in CEP library, elements come later than
watermark will be dropped,we can put it in side Output with outPutTag
You
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6104
@bowenli86 thanks for review, i have fix the error according to the comment
and add the unit test in CEPITCase, please help review it again. cc @kl0u
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6080
@aljoscha please help review this ,thanks
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6109#discussion_r192628465
--- Diff: docs/start/building.md ---
@@ -50,7 +50,11 @@ mvn clean install -DskipTests
This instructs [Maven](http://maven.apache.org) (`mvn
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6059
@dawidwys I'm sorry, I don't have a systematic testing tool. I'm working on
the inner dynamic cep and `CEPOperator` can process serveral `NFA` when receive
an element. Then i encounter
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6104
thx, can help merge this @dawidwys
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6104
@dawidwys fix the flaw according to your suggestion ;-)
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6059#discussion_r192558813
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/EventWrapper.java
---
@@ -0,0 +1,188 @@
+/*
+ * Licensed
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6111
[FLINK-9504]Change the log level of checkpoint duration to debug
Now, every time checkpoint it will log with the OperatorStateBackend and
KeyedStateBackend with per partition/parallel time cost
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6104#discussion_r192428178
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/PatternStream.java
---
@@ -54,6 +54,12 @@
// comparator to sort events
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6104#discussion_r192428106
--- Diff: docs/dev/libs/cep.md ---
@@ -1524,7 +1524,52 @@ In `CEP` the order in which elements are processed
matters. To guarantee that el
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6104#discussion_r192428038
--- Diff: docs/dev/libs/cep.md ---
@@ -1524,7 +1524,52 @@ In `CEP` the order in which elements are processed
matters. To guarantee that el
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6104#discussion_r192428502
--- Diff:
flink-libraries/flink-cep/src/test/java/org/apache/flink/cep/CEPITCase.java ---
@@ -383,6 +386,100 @@ public String select(Map>
patt
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6059
After go through the whole change, i think it is extremely a good feature
ð, and i will apply this patch to our inner library to test how much the
performance improved.
A little question: why
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6124
[FLINK-8914][CEP]Fix wrong semantic when greedy pattern is the head of the
pattern
## What is the purpose of the change
As described in the jira
[FLINK-8914](https://issues.apache.org/jira
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6124
@dawidwys please help review this pr when you free,thx
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6111
thanks @StephanEwen for review. As you mentioned, I only set the log level
in state backends to `debug` , is there something wrong with my pull request or
misunderstand what your meaning ?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
Using the `entries#putAll` in `flushCache` lead to the count in
`NFAStateAccessTest` increased, I will check it locally , this travis will
fail.
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r198151182
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -346,16 +351,87 @@ private void lockEvent
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
fixed the error that the access to state increased in `NFAStateAccessTest`
by add the `isEmpty` judgment before update the state.
---
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6205
[FLINK-9642]Reduce the count to deal with state during a CEP process
## What is the purpose of the change
With the rework of sharedBuffer Flink-9418, the lock & release operation is
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
Hi @zhangminglei , thanks for your review. I only check the
SharedBufferTest locally before, the error in travis means the num of state
access (read and write) is less than before which
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r197629310
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -346,16 +351,87 @@ private void lockEvent
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r197629218
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -75,6 +76,9 @@
private MapState
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r197629205
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -346,16 +351,87 @@ private void lockEvent
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
The travis error this time seems unrelated.
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6168
Is it ok now ? @dawidwys
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
Is this look ok now? ping @sihuazhou @dawidwys
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r198142501
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -346,16 +351,87 @@ private void lockEvent
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6205#discussion_r198143091
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/sharedbuffer/SharedBuffer.java
---
@@ -346,16 +351,87 @@ private void lockEvent
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/6234
[FLINK-9431]Introduce time bounded condition to cep
## What is the purpose of the change
In cep the event is now driving the transformation of the NFA, I think the
time factor should also
GitHub user Aitozi opened a pull request:
https://github.com/apache/flink/pull/5405
[FLINK-8477][Window]Add api to support user to skip serval broken window
In production, some application like monitor type , it need the accuarcy
data,but in this scenario: if we start a job at 10
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/5405
cc @aljoscha please help review this patch.
see the picture above, when user
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/5405
Hi @aljoscha , you have mentioned two points :
1. The events arrived may out of order in event-time processing
2. We can use windowFunction or ProcessWindowFunction to filter serverl
window
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/5405
ping @aljoscha
---
Github user Aitozi closed the pull request at:
https://github.com/apache/flink/pull/6168
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
Aha, I meat to improve the performance of windowOperator, In the scenario
mentioned in the issue, this PR can avoid that bug, lucky hit ;-)
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
Resolved the conflicts, please help review when you free @dawidwys .
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
OK ð
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6111
Hi @StephanEwen , is there any change should be made to this PR ?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6168
Hi @dawidwys , since this commit has been merged in, is this pr need to be
closed by me?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
ping @aljoscha , could you help review it, I'd like to hear your opinion on
this PR too. thx
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6205
Could you take a look at this PR @dawidwys ?
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6186#discussion_r201895445
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlListState.java
---
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6186
Hi, after read the whole implementation, i found that the state is expired
when it is accessed, When there is the dirty data store to state and never be
queried, how does it can be expired
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6186#discussion_r201971238
--- Diff:
flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlListState.java
---
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache
Github user Aitozi commented on the pull request:
https://github.com/apache/flink/commit/8c89f3c6b5ebd0334176d9e7e57b38b4d39a594a#commitcomment-29640344
Hi, do you discuss about the logic change of Trigger in the PR:
https://github.com/apache/flink/pull/6224
---
Github user Aitozi closed the pull request at:
https://github.com/apache/flink/pull/6224
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
Got it, Thanks for your explanation ;-ï¼. I will close this PR.
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
Hi, @aljoscha
I read the PR of https://issues.apache.org/jira/browse/FLINK-5363, i think
it is not a same thing with this PR. You meant to get `TriggerResult` no matter
the window content
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
Get it, this change indeed will lead to the situation you mentioned.
> What are the cases where this change leads to an improvement
I just read code and think the order is not suita
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/3105
why it is closed ? Is it has been merged in?
---
Github user Aitozi commented on the issue:
https://github.com/apache/flink/pull/6224
The failed info in travis error shows the test with `checkClusterEmpty` is
wrong, Is this happened due to the reuse of yarn cluster? It seems unrelated
with this pull request.
---
Github user Aitozi commented on a diff in the pull request:
https://github.com/apache/flink/pull/6168#discussion_r197482701
--- Diff:
flink-libraries/flink-cep/src/main/java/org/apache/flink/cep/nfa/NFA.java ---
@@ -609,7 +611,7 @@ public void close() throws Exception
[
https://issues.apache.org/jira/browse/FLINK-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aitozi updated FLINK-7611:
--
Affects Version/s: 1.3.0
> add metrics to measure the data drop by waterm
[
https://issues.apache.org/jira/browse/FLINK-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aitozi updated FLINK-7611:
--
Affects Version/s: 1.2.0
> add metrics to measure the data drop by waterm
[
https://issues.apache.org/jira/browse/FLINK-7608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16168852#comment-16168852
]
aitozi commented on FLINK-7608:
---
i am doubt that why it need to access the value of p50,p95,p99. i think
aitozi created FLINK-7611:
-
Summary: add metrics to measure the data drop by watermark
Key: FLINK-7611
URL: https://issues.apache.org/jira/browse/FLINK-7611
Project: Flink
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/FLINK-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16164008#comment-16164008
]
aitozi commented on FLINK-7611:
---
i have add the pr, https://github.com/apache/flink/pull/4665 , anyone
[
https://issues.apache.org/jira/browse/FLINK-7945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
aitozi updated FLINK-7945:
--
Summary: Per partition-lag metric lost in kafka connector (was:
kafka-connector11 use kafkaConsumer0.9 caused
1 - 100 of 897 matches
Mail list logo
