[jira] [Created] (FLINK-23434) Inconsistent rowtypes occur in IncrementalAggregateRule

2021-07-19 Thread godfrey he (Jira)
godfrey he created FLINK-23434: -- Summary: Inconsistent rowtypes occur in IncrementalAggregateRule Key: FLINK-23434 URL: https://issues.apache.org/jira/browse/FLINK-23434 Project: Flink Issue

[jira] [Created] (FLINK-23433) Metrics cannot be initialized in format

2021-07-19 Thread Fangliang Liu (Jira)
Fangliang Liu created FLINK-23433: - Summary: Metrics cannot be initialized in format Key: FLINK-23433 URL: https://issues.apache.org/jira/browse/FLINK-23433 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-173: Support DAG of algorithms (Flink ML)

2021-07-19 Thread Becket Qin
Hi Dong, Zhipeng and Fan, Thanks for the detailed proposals. It is quite a lot of reading! Given that we are introducing a lot of stuff here, I find that it might be easier for people to discuss if we can list the fundamental differences first. From what I understand, the very fundamental

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-19 Thread Becket Qin
Hi Arvid and Chesnay, Thanks for the explanation. As outlined in the JavaDoc and in the draft PR [1], it's up to the user to > implement it in a way that fetch time always corresponds to the latest > polled record. In that case, do we still need the metric here? It seems we are creating a

[jira] [Created] (FLINK-23432) [DOCS] The 'docs/content.zh/docs/ops/state/checkpoints' page did not add a newline, which invalidated the Markdown header

2021-07-19 Thread wuguihu (Jira)
wuguihu created FLINK-23432: --- Summary: [DOCS] The 'docs/content.zh/docs/ops/state/checkpoints' page did not add a newline, which invalidated the Markdown header Key: FLINK-23432 URL:

[jira] [Created] (FLINK-23431) KafkaProducerExactlyOnceITCase.testMultipleSinkOperators fails with "The topic metadata failed to propagate to Kafka broker."

2021-07-19 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-23431: Summary: KafkaProducerExactlyOnceITCase.testMultipleSinkOperators fails with "The topic metadata failed to propagate to Kafka broker." Key: FLINK-23431 URL:

[jira] [Created] (FLINK-23430) Do not take snapshot for operator coordinators which all tasks finished

2021-07-19 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-23430: Summary: Do not take snapshot for operator coordinators which all tasks finished Key: FLINK-23430 URL: https://issues.apache.org/jira/browse/FLINK-23430

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-19 Thread Steven Wu
Thanks, Arvid! +1 for SinkWriterMetricGroup. Sink is a little more tricky, because it can have local committer (running on TM) or global committer (running on JM). In the future, it is possible to add SinkCommitterMetricGroup or SinkGlobalCommitterMetricGroup. Regarding "lastFetchTime" latency

[jira] [Created] (FLINK-23429) State Processor API failed with FileNotFoundException when working with state files on Cloud Storage

2021-07-19 Thread Jun Qin (Jira)
Jun Qin created FLINK-23429: --- Summary: State Processor API failed with FileNotFoundException when working with state files on Cloud Storage Key: FLINK-23429 URL: https://issues.apache.org/jira/browse/FLINK-23429

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Yun Gao
Hi Till, Dawid, Very thanks for the comments and discussion! Glad that it seems we have come to a convergence, and I also agree with that we could not include the optimization in the first version. Best Yun -- From:Dawid

[jira] [Created] (FLINK-23428) Performance regression in state benchmarks on Jul 16 2021

2021-07-19 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-23428: - Summary: Performance regression in state benchmarks on Jul 16 2021 Key: FLINK-23428 URL: https://issues.apache.org/jira/browse/FLINK-23428 Project: Flink

[jira] [Created] (FLINK-23427) Translate the page of "Blocking Shuffle" into Chinese

2021-07-19 Thread Aiden Gong (Jira)
Aiden Gong created FLINK-23427: -- Summary: Translate the page of "Blocking Shuffle" into Chinese Key: FLINK-23427 URL: https://issues.apache.org/jira/browse/FLINK-23427 Project: Flink Issue

[jira] [Created] (FLINK-23426) Support changelog processing in batch mode

2021-07-19 Thread Timo Walther (Jira)
Timo Walther created FLINK-23426: Summary: Support changelog processing in batch mode Key: FLINK-23426 URL: https://issues.apache.org/jira/browse/FLINK-23426 Project: Flink Issue Type:

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Dawid Wysakowicz
Small correction. I meant we need to adjust the EndOfInputEvent of course. Best, Dawid On 19/07/2021 11:48, Dawid Wysakowicz wrote: > Hey Till, > > Yes, you're right we will have to adjust the current state of > EndOfPartitionEvent and move the moment when we emit it to have what > we're

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Dawid Wysakowicz
Hey Till, Yes, you're right we will have to adjust the current state of EndOfPartitionEvent and move the moment when we emit it to have what we're discussing here. We are aware of that. As for the MAX_WATERMARK vs finish(). My take is that we should always emit MAX_WATERMARK before calling

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Till Rohrmann
Hi Yun and Dawid, Thanks for your comments. I do agree with your comments that finish() can do more than MAX_WATERMARK. I guess we should then explain how MAX_WATERMARK and finish() play together and what kind of order guarantees we provide. Concerning the EndOfPartitionEvent, I am not entirely

Re: Introduction email

2021-07-19 Thread Márton Balassi
Hi Srini, Welcome to the community. Awesome to hear that LinkedIn decided to embrace Flink, excited to see where this joint path leads us. On Mon, Jul 19, 2021 at 11:02 AM Timo Walther wrote: > Hi Srini, > > welcome aboard! Great to see more adoption in the SQL space. Looking > forward to

[jira] [Created] (FLINK-23425) The impact of cpu cores on test results for StreamingJobGraphGeneratorTest#testSlotSharingResourceConfigurationWithDefaultSlotSharingGroup

2021-07-19 Thread tartarus (Jira)
tartarus created FLINK-23425: Summary: The impact of cpu cores on test results for StreamingJobGraphGeneratorTest#testSlotSharingResourceConfigurationWithDefaultSlotSharingGroup Key: FLINK-23425 URL:

Re: Introduction email

2021-07-19 Thread Timo Walther
Hi Srini, welcome aboard! Great to see more adoption in the SQL space. Looking forward to collaboration. Regards, Timo On 19.07.21 10:58, Till Rohrmann wrote: Hi Srini, Welcome to the Flink community :-) Great to hear what you are planning to do with Flink at LinkedIn. I think sharing this

Re: Introduction email

2021-07-19 Thread Till Rohrmann
Hi Srini, Welcome to the Flink community :-) Great to hear what you are planning to do with Flink at LinkedIn. I think sharing this is very motivational for the community and also gives context for what you are focusing on. Looking forward to working with you and improving Flink. Cheers, Till

Re: [VOTE] Release flink-shaded 14.0, release candidate 1

2021-07-19 Thread Timo Walther
+1 (binding) I went through all commits one more time and could not spot anything that would block a release. Thanks Chesnay! Timo On 15.07.21 09:02, Chesnay Schepler wrote: Hi everyone, Please review and vote on the release candidate #1 for the version 14.0, as follows: [ ] +1, Approve

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Dawid Wysakowicz
Personally I don't find this optimization important and I'd rather leave it out not to complicate the codebase further. I doubt we save much there. I don't have a strong opinion though. Best, Dawid On 19/07/2021 10:31, Yun Gao wrote: > Hi, > > Very thanks Dawid for the thoughts! > > Currently I

Re: Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Yun Gao
Hi, Very thanks Dawid for the thoughts! Currently I also do not have different opinions regarding this part. But I have one more issue to confirm: during the previous discussion we have discussed that for the final checkpoint case, we might have an optmization that if a task do not have

[jira] [Created] (FLINK-23424) Disable key sorting for Transformations created within table planner

2021-07-19 Thread Timo Walther (Jira)
Timo Walther created FLINK-23424: Summary: Disable key sorting for Transformations created within table planner Key: FLINK-23424 URL: https://issues.apache.org/jira/browse/FLINK-23424 Project: Flink

[jira] [Created] (FLINK-23423) Translate the page of "Elasticsearch Connector" into Chinese

2021-07-19 Thread huxixiang (Jira)
huxixiang created FLINK-23423: - Summary: Translate the page of "Elasticsearch Connector" into Chinese Key: FLINK-23423 URL: https://issues.apache.org/jira/browse/FLINK-23423 Project: Flink

[DISCUSS] FLIP-177: Extend Sink API

2021-07-19 Thread Arvid Heise
Dear devs, today I'd like to start the discussion on the Sink API. I have drafted a FLIP [1] with an accompanying PR [2]. This FLIP is a bit special as it's actually a few smaller Amend-FLIPs in one. In this discussion, we should decide on the scope and cut out too invasive steps if we can't

Re: [DISCUSS] FLIP-171: Async Sink

2021-07-19 Thread Arvid Heise
Hi Guowei, That's a good question. FLIP-171 was mainly motivated by at-least-once sinks (Kinesis, DynamoDB, Timestream, Firehose). The idea is to expand that interface in the future to also support exactly-once. I'm currently drafting FLIP-177 (the FLIP that enables async pattern in the first

Re: [VOTE] FLIP-184: Refine ShuffleMaster lifecycle management for pluggable shuffle service framework

2021-07-19 Thread Zhu Zhu
+1 (binding) Thanks, Zhu XING JIN 于2021年7月19日周一 上午10:29写道: > +1 (non-binding) > > Best, > Jin > > Guowei Ma 于2021年7月19日周一 上午9:41写道: > > > +1(binding) > > > > Best, > > Guowei > > > > > > On Fri, Jul 16, 2021 at 5:36 PM Yingjie Cao > > wrote: > > > > > Hi all, > > > > > > I'd like to start a

Re: [DISCUSS] FLIP-179: Expose Standardized Operator Metrics

2021-07-19 Thread Arvid Heise
Hi Steven, I extended the FLIP and its draft PR to have a SourceReaderMetricGroup and a SplitEnumeratorMetricGroup. I hope that it makes it clearer. I'd like to address FLINK-21000 as part of the implementation but I'd keep it out of the FLIP discussion. Question: should we rename

Re: [DISCUSS] Watermark propagation with Sink API

2021-07-19 Thread Arvid Heise
Hi everyone, I created a FLIP and started a discussion around that topic [1]. Best, Arvid [1] https://lists.apache.org/thread.html/r8357d64b9cfdf5a233c53a20d9ac62b75c07c925ce2c43e162f1e39c%40%3Cdev.flink.apache.org%3E On Tue, Jun 8, 2021 at 11:46 PM Eron Wright wrote: > Thanks, the narrowed

Re: [DISCUSS] Definition of idle partitions

2021-07-19 Thread Arvid Heise
Hi everyone, I created a FLIP and started a discussion around that topic [1]. Best, Arvid [1] https://lists.apache.org/thread.html/r8357d64b9cfdf5a233c53a20d9ac62b75c07c925ce2c43e162f1e39c%40%3Cdev.flink.apache.org%3E On Thu, Jun 10, 2021 at 9:18 PM Eron Wright wrote: > I quickly updated

[DISCUSS] FLIP-180: Adjust StreamStatus and Idleness definition

2021-07-19 Thread Arvid Heise
Dear devs, We recently discovered that StreamStatus and Idleness is insufficiently defined [1], so I drafted a FLIP [3] to amend that situation. It would be good to hear more opinions on that matter. Ideally, we can make the changes to 1.14 as some newly added methods are affected. Best, Arvid

[jira] [Created] (FLINK-23422) window function not support in window agg function

2021-07-19 Thread silence (Jira)
silence created FLINK-23422: --- Summary: window function not support in window agg function Key: FLINK-23422 URL: https://issues.apache.org/jira/browse/FLINK-23422 Project: Flink Issue Type: Bug