Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch Job
+0 (binding) Op di 10 jan. 2023 om 13:11 schreef yuxia : > +1 (non-binding). > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Zhu Zhu" > 收件人: "dev" > 发送时间: 星期二, 2023年 1 月 10日 下午 5:50:39 > 主题: Re: [VOTE] FLIP-281: Sink Supports Speculative Execution For Batch Job > > +1 (binding) > > Thanks, > Zhu > > Biao Liu 于2023年1月5日周四 10:37写道： > > > > Hi Martijn, > > > > Sure, thanks for the reminder about the holiday period. > > Looking forward to your feedback! > > > > Thanks, > > Biao /'bɪ.aʊ/ > > > > > > > > On Thu, 5 Jan 2023 at 03:07, Martijn Visser > > wrote: > > > > > Hi Biao, > > > > > > To be honest, I haven't read the FLIP yet since this is still a holiday > > > period in Europe. I would like to read it in the next few days. Can you > > > keep the vote open a little longer? > > > > > > Best regards, > > > > > > Martijn > > > > > > On Wed, Jan 4, 2023 at 1:31 PM Biao Liu wrote: > > > > > > > Hi everyone, > > > > > > > > Thanks for all the feedback! > > > > > > > > Based on the discussion[1], we seem to have a consensus. So I'd like > to > > > > start a vote on FLIP-281: Sink Supports Speculative Execution For > Batch > > > > Job[2]. The vote will last for 72 hours, unless there is an > objection or > > > > insufficient votes. > > > > > > > > [1] https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1 > > > > [2] > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-281+Sink+Supports+Speculative+Execution+For+Batch+Job > > > > > > > > Thanks, > > > > Biao /'bɪ.aʊ/ > > > > > > > >
Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job
Hi Biao, While I rather wouldn't add new features to (to-be) deprecated features, I would be +0 for this. Best regards, Martijn Op do 12 jan. 2023 om 08:42 schreef Biao Liu : > Hi Martijn, > > Thanks for your feedback! > > Yes, we propose to support speculative execution for SinkFunction. > 1. From the perspective of compatibility, SinkFunction is the most original > Sink implementation.There are lots of implementations based on > SinkFunction, not only in Flink official codebase but also in user's > private codebase. It's a more serious issue than Sink V1. Of course we hope > users could migrate the legacy implementation to the new interface. However > migration is always hard. > 2. From the perspective of cost, we don't need to do much extra work to > support speculative execution for SinkFunction. All we need to do is check > whether the SinkFunction implementation > inherits SupportsConcurrentExecutionAttempts or not. The other parts of > work are the same with Sink V2. > > To summarize, it's cheap to support speculative execution for SinkFunction. > And it may allow more existing scenarios to run with speculative execution. > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Wed, 11 Jan 2023 at 21:22, Martijn Visser > wrote: > > > Hi Biao, > > > > Apologies for the late jumping in. My only question is about > SinkFunction, > > does this imply that we want to add support for this to the SinkFunction? > > If so, I would not be in favour of that since we would like to deprecate > (I > > actually thought that was already the case) the SinkFunction in favour of > > SinkV2. > > > > Besides that, I have no other comments. > > > > Best regards, > > > > Martijn > > > > On Wed, Jan 4, 2023 at 7:28 AM Jing Zhang wrote: > > > > > Hi Biao, > > > > > > Thanks for explanation. > > > > > > +1 for the proposal. > > > > > > Best, > > > Jing Zhang > > > > > > Lijie Wang 于2023年1月4日周三 12:11写道： > > > > > > > Hi Biao, > > > > > > > > Thanks for the explanation of how SinkV2 knows the right subtask > > > > attempt. I have no more questions, +1 for the proposal. > > > > > > > > Best, > > > > Lijie > > > > > > > > Biao Liu 于2022年12月28日周三 17:22写道： > > > > > > > > > Thanks for all your feedback! > > > > > > > > > > To @Yuxia, > > > > > > > > > > > What the sink expect to do to isolate data produced by > speculative > > > > > > executions? IIUC, if the taks failover, it also generate a new > > > > attempt. > > > > > > Does it make difference in isolating data produced? > > > > > > > > > > > > > > > Yes there is something different from the task failover scenario. > The > > > > > attempt number is more necessary for speculative execution than > > > failover. > > > > > Because there can be only one subtask instance running at the same > > time > > > > in > > > > > the failover scenario. > > > > > > > > > > Let's take FileSystemOutputFormat as an example. For the failover > > > > scenario, > > > > > the temporary directory to store produced data can be something > like > > > > > "$root_dir/task-$taskNumber/". At the initialization phase, subtask > > > > deletes > > > > > and re-creates the temporary directory. > > > > > > > > > > However in the speculative execution scenario, it does not work > > because > > > > > there might be several subtasks running at the same time. These > > > subtasks > > > > > might delete, re-create and write the same temporary directory at > the > > > > > same time. The correct temporary directory should be like > > > > > "$root_dir/task-$taskNumber/attempt-$attemptNumber". So it's > > necessary > > > to > > > > > expose the attempt number to the Sink implementation to do the data > > > > > isolation. > > > > > > > > > > > > > > > To @Lijie, > > > > > > > > > > > I have a question about this: does SinkV2 need to do the same > > thing? > > > > > > > > > > > > > > > Actually, yes. > > > > > > > > > > Should we/users do it in the committer? If yes, how does the > commiter > > > > know > > > > > > which one is the right subtask attempt? > > > > > > > > > > > > > > > Yes, we/users should do it in the committer. > > > > > > > > > > In the current design, the Committer of Sink V2 should get the > "which > > > one > > > > > is the right subtask attempt" information from the "committable > > data'' > > > > > produced by SinkWriter. Let's take the FileSink as example, the > > > > > "committable data" sent to the Committer contains the full path of > > the > > > > > files produced by SinkWriter. Users could also pass the attempt > > number > > > > > through "committable data" from SinkWriter to Committer. > > > > > > > > > > In the "Rejected Alternatives -> Introduce a way to clean leaked > data > > > of > > > > > Sink V2" section of the FLIP document, we discussed some of the > > reasons > > > > > that we didn't provide the API like OutputFormat. > > > > > > > > > > To @Jing Zhang > > > > > > > > > > I have a question about this: Speculative execution of Committer > will > > > be > > > > > > disabled. > > > > > > > >
Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job
Hi Martijn, Thanks for your feedback! Yes, we propose to support speculative execution for SinkFunction. 1. From the perspective of compatibility, SinkFunction is the most original Sink implementation.There are lots of implementations based on SinkFunction, not only in Flink official codebase but also in user's private codebase. It's a more serious issue than Sink V1. Of course we hope users could migrate the legacy implementation to the new interface. However migration is always hard. 2. From the perspective of cost, we don't need to do much extra work to support speculative execution for SinkFunction. All we need to do is check whether the SinkFunction implementation inherits SupportsConcurrentExecutionAttempts or not. The other parts of work are the same with Sink V2. To summarize, it's cheap to support speculative execution for SinkFunction. And it may allow more existing scenarios to run with speculative execution. Thanks, Biao /'bɪ.aʊ/ On Wed, 11 Jan 2023 at 21:22, Martijn Visser wrote: > Hi Biao, > > Apologies for the late jumping in. My only question is about SinkFunction, > does this imply that we want to add support for this to the SinkFunction? > If so, I would not be in favour of that since we would like to deprecate (I > actually thought that was already the case) the SinkFunction in favour of > SinkV2. > > Besides that, I have no other comments. > > Best regards, > > Martijn > > On Wed, Jan 4, 2023 at 7:28 AM Jing Zhang wrote: > > > Hi Biao, > > > > Thanks for explanation. > > > > +1 for the proposal. > > > > Best, > > Jing Zhang > > > > Lijie Wang 于2023年1月4日周三 12:11写道： > > > > > Hi Biao, > > > > > > Thanks for the explanation of how SinkV2 knows the right subtask > > > attempt. I have no more questions, +1 for the proposal. > > > > > > Best, > > > Lijie > > > > > > Biao Liu 于2022年12月28日周三 17:22写道： > > > > > > > Thanks for all your feedback! > > > > > > > > To @Yuxia, > > > > > > > > > What the sink expect to do to isolate data produced by speculative > > > > > executions? IIUC, if the taks failover, it also generate a new > > > attempt. > > > > > Does it make difference in isolating data produced? > > > > > > > > > > > > Yes there is something different from the task failover scenario. The > > > > attempt number is more necessary for speculative execution than > > failover. > > > > Because there can be only one subtask instance running at the same > time > > > in > > > > the failover scenario. > > > > > > > > Let's take FileSystemOutputFormat as an example. For the failover > > > scenario, > > > > the temporary directory to store produced data can be something like > > > > "$root_dir/task-$taskNumber/". At the initialization phase, subtask > > > deletes > > > > and re-creates the temporary directory. > > > > > > > > However in the speculative execution scenario, it does not work > because > > > > there might be several subtasks running at the same time. These > > subtasks > > > > might delete, re-create and write the same temporary directory at the > > > > same time. The correct temporary directory should be like > > > > "$root_dir/task-$taskNumber/attempt-$attemptNumber". So it's > necessary > > to > > > > expose the attempt number to the Sink implementation to do the data > > > > isolation. > > > > > > > > > > > > To @Lijie, > > > > > > > > > I have a question about this: does SinkV2 need to do the same > thing? > > > > > > > > > > > > Actually, yes. > > > > > > > > Should we/users do it in the committer? If yes, how does the commiter > > > know > > > > > which one is the right subtask attempt? > > > > > > > > > > > > Yes, we/users should do it in the committer. > > > > > > > > In the current design, the Committer of Sink V2 should get the "which > > one > > > > is the right subtask attempt" information from the "committable > data'' > > > > produced by SinkWriter. Let's take the FileSink as example, the > > > > "committable data" sent to the Committer contains the full path of > the > > > > files produced by SinkWriter. Users could also pass the attempt > number > > > > through "committable data" from SinkWriter to Committer. > > > > > > > > In the "Rejected Alternatives -> Introduce a way to clean leaked data > > of > > > > Sink V2" section of the FLIP document, we discussed some of the > reasons > > > > that we didn't provide the API like OutputFormat. > > > > > > > > To @Jing Zhang > > > > > > > > I have a question about this: Speculative execution of Committer will > > be > > > > > disabled. > > > > > > > > I agree with your point and I saw the similar requirements to disable > > > > speculative > > > > > execution for specified operators. > > > > > > > > However the requirement is not supported currently. I think there > > > > should be some > > > > > place to describe how to support it. > > > > > > > > > > > > In this FLIP design, the speculative execution of Committer of Sink > V2 > > > will > > > > be disabled by Flink. It's not an optional operation. Users can not
Re: [VOTE] Apache Flink Table Store 0.3.0, release candidate #1
Thanks Yu for your validation. I created a new staging directory [1] [1] https://repository.apache.org/content/repositories/orgapacheflink-1577/ Best, Jingsong On Thu, Jan 12, 2023 at 3:07 PM Yu Li wrote: > > Hi Jingsong, > > It seems the given staging directory [1] is not exposed, could you double > check and republish if necessary? Thanks. > > Best Regards, > Yu > > [1] https://repository.apache.org/content/repositories/orgapacheflink-1576/ > > > On Tue, 10 Jan 2023 at 16:53, Jingsong Li wrote: > > > Hi everyone, > > > > Please review and vote on the release candidate #1 for the version > > 0.3.0 of Apache Flink Table Store, as follows: > > > > [ ] +1, Approve the release > > [ ] -1, Do not approve the release (please provide specific comments) > > > > **Release Overview** > > > > As an overview, the release consists of the following: > > a) Table Store canonical source distribution to be deployed to the > > release repository at dist.apache.org > > b) Table Store binary convenience releases to be deployed to the > > release repository at dist.apache.org > > c) Maven artifacts to be deployed to the Maven Central Repository > > > > **Staging Areas to Review** > > > > The staging areas containing the above mentioned artifacts are as follows, > > for your review: > > * All artifacts for a) and b) can be found in the corresponding dev > > repository at dist.apache.org [2] > > * All artifacts for c) can be found at the Apache Nexus Repository [3] > > > > All artifacts are signed with the key > > 2C2B6A653B07086B65E4369F7C76245E0A318150 [4] > > > > Other links for your review: > > * JIRA release notes [5] > > * source code tag "release-0.3.0-rc1" [6] > > * PR to update the website Downloads page to include Table Store links [7] > > > > **Vote Duration** > > > > The voting time will run for at least 72 hours. > > It is adopted by majority approval, with at least 3 PMC affirmative votes. > > > > Best, > > Jingsong Lee > > > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release > > [2] > > https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.3.0-rc1/ > > [3] > > https://repository.apache.org/content/repositories/orgapacheflink-1576/ > > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > > [5] > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352111 > > [6] https://github.com/apache/flink-table-store/tree/release-0.3.0-rc1 > > [7] https://github.com/apache/flink-web/pull/601 > >
[jira] [Created] (FLINK-30641) docs_404_check fail
Lijie Wang created FLINK-30641:
--
Summary: docs_404_check fail
Key: FLINK-30641
URL: https://issues.apache.org/jira/browse/FLINK-30641
Project: Flink
Issue Type: Bug
Reporter: Lijie Wang
{code:java}
Cloning into 'flink-connector-rabbitmq'...
Note: switching to '325b6ba8d866bb02204736229aa54ade304119a3'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:
git switch -c
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
Start building sites …
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:45:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:46:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:53:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:54:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:62:21":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:63:21":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:103:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/formats/overview.md:104:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/datastream/dynamodb":
"/home/vsts/work/1/s/docs/content/docs/connectors/datastream/overview.md:43:22":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/datastream/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/datastream/overview.md:44:34":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/datastream/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/datastream/overview.md:45:35":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/dynamodb":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/overview.md:70:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/overview.md:76:20":
page not found
ERROR 2023/01/12 06:49:21 [en] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content/docs/connectors/table/overview.md:82:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:45:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:46:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:53:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:54:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:62:21":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:63:21":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/kinesis":
"/home/vsts/work/1/s/docs/content.zh/docs/connectors/table/formats/overview.md:103:20":
page not found
ERROR 2023/01/12 06:49:30 [zh] REF_NOT_FOUND: Ref
"docs/connectors/table/firehose":
Re: [VOTE] Apache Flink Table Store 0.3.0, release candidate #1
Hi Jingsong, It seems the given staging directory [1] is not exposed, could you double check and republish if necessary? Thanks. Best Regards, Yu [1] https://repository.apache.org/content/repositories/orgapacheflink-1576/ On Tue, 10 Jan 2023 at 16:53, Jingsong Li wrote: > Hi everyone, > > Please review and vote on the release candidate #1 for the version > 0.3.0 of Apache Flink Table Store, as follows: > > [ ] +1, Approve the release > [ ] -1, Do not approve the release (please provide specific comments) > > **Release Overview** > > As an overview, the release consists of the following: > a) Table Store canonical source distribution to be deployed to the > release repository at dist.apache.org > b) Table Store binary convenience releases to be deployed to the > release repository at dist.apache.org > c) Maven artifacts to be deployed to the Maven Central Repository > > **Staging Areas to Review** > > The staging areas containing the above mentioned artifacts are as follows, > for your review: > * All artifacts for a) and b) can be found in the corresponding dev > repository at dist.apache.org [2] > * All artifacts for c) can be found at the Apache Nexus Repository [3] > > All artifacts are signed with the key > 2C2B6A653B07086B65E4369F7C76245E0A318150 [4] > > Other links for your review: > * JIRA release notes [5] > * source code tag "release-0.3.0-rc1" [6] > * PR to update the website Downloads page to include Table Store links [7] > > **Vote Duration** > > The voting time will run for at least 72 hours. > It is adopted by majority approval, with at least 3 PMC affirmative votes. > > Best, > Jingsong Lee > > [1] > https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Table+Store+Release > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-table-store-0.3.0-rc1/ > [3] > https://repository.apache.org/content/repositories/orgapacheflink-1576/ > [4] https://dist.apache.org/repos/dist/release/flink/KEYS > [5] > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12352111 > [6] https://github.com/apache/flink-table-store/tree/release-0.3.0-rc1 > [7] https://github.com/apache/flink-web/pull/601 >
[jira] [Created] (FLINK-30640) Unstable test in CliClientITCase
yuzelin created FLINK-30640: --- Summary: Unstable test in CliClientITCase Key: FLINK-30640 URL: https://issues.apache.org/jira/browse/FLINK-30640 Project: Flink Issue Type: Bug Components: Connectors / Hive, Table SQL / Client Affects Versions: 1.17.0 Reporter: yuzelin [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44743=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4] The failed test can work normally in my local environment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi all, AFAICS, there’s no consensus so far on how we should proceed discussions while some of the participants are unresponsive, in either the Bylaws[1] or any of the ML threads that I’m aware of. Without such consensus, it is understandable that contributors take actions based on their different personal opinions, which IMO causes this argument right now. So let’s try not to point fingers at each other and focus on setting up a commonly agreed process on this. I agree that every opinion, especially the concerns and disagreements, should be respected and carefully discussed and addressed. Meantime, it is also important for the Flink project to move forward at a good pace, of course without sacrificing the quality. So the question is how do we actively resolve the disagreements and reach consensus efficiently. With the growth of the community, there’re more developers from various companies concurrently working on more FLIPs / features / efforts than before. It’s getting harder and harder for one contributor (e.g., myself) to participate in every thread that he’s interested in without slowing down the evolution of the project. In such cases, I personally would tend to trust other contributors, especially the committers, in doing a good job without my participant, rather than slowing down others due to my lack of capacity. So here’s my proposal: 1. When raising an opinion (question / concern / disagreement) that blocks the discussion from reaching consensus, the contributor that the opinion comes from should try to respond to the replies to their opinions within 1 week (5 work days since the replies are made, excluding weekends and public holidays) if possible. 2. If the response cannot be made within 1 week (due to other works, personal vacation plans, etc.), an explicit date of response should be given. The given response date should be no later than 2 weeks (10 work days since the replies are made, excluding weekends and public holidays), unless agreed by all active participants in the discussion. 3. A question / concern / disagreement can be considered addressed if: a) there’s no response in 1 week or before the given response date, and b) all the active participants agreed that it is resolved. 4. A discussion should be opened for at least 1 week, for opinions to be raised, before starting a vote. When there’re unresponsive participants, a 72h notice is required before claiming reaching out consensus for a discussion and starting a vote. This can overlap with the time waiting for responses from the irresponsive participants. 5. Here *response* is restricted to technical opinions. Quick replies such as "will take a look asap" do not count. Looking forward to your opinions. Best, Xintong [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws On Thu, Jan 12, 2023 at 10:07 AM Hang Ruan wrote: > Hi, all, > > Thanks for all discussions about this FLIP first. We all are trying to make > Flink better. But not getting a reply quickly really discourages > contributors. > OperatorCoordinatorMetricGroup and SplitEnumeratorMetricGroup are important > for many developers. And many metrics can not be reported without it. This > FLIP raised on 26 Dec 2022 and has last over a month. > So let's make the discussion on FLIP-274 be settled. > > Thanks for all helps and let's make this FLIP move on together. > > Best, > Hang > > Chesnay Schepler 于2023年1月12日周四 07:29写道： > > > You know, back when I wrote that line I actually felt a bit guilty about > > it and nearly dropped it, since it implicitly accused _someone_ of > > actually being capable/willing to push something through despite voiced > > concerns while no ones looking. > > > > Turns out things are a lot worse that I thought. You're actually > > doubling down on it and even insult me. > > > > > > As I haven't had a time to revisit the discussion and afaict my concerns > > weren't addressed, I hereby vote -1 (binding). > > > > Please be aware that since I'm effectively only back from vacation > > tomorrow and need to catch up on things in general, I may not be able to > > revisit the discussion this week. In part because I guess I now need to > > double-check everything. > > > > CC'ing the private ML because I find this very troubling behavior. > > > > On 11/01/2023 16:20, Martijn Visser wrote: > > > Hi Jark, > > > > > > I disagree with your statement that someone doesn't care about the > FLIP, > > > especially since that person participated in the initial FLIP > discussion > > > and mentioned explicitly "Since I'm on holidays soon, just so no one > > tries > > > to pull a fast one on me, if this were to go to a vote as-is I'd be > > against > > > it.". > > > > > > Anyway: I can't tell when Chesnay exactly will participate, but I do > > expect > > > that this week. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Wed, Jan 11, 2023 at 4:11 PM Jark Wu wrote: > > > > > >> Hi Martijn, > > >> > > >> Thanks for helping us to find
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Thank you all! I'm honored to join the committers and look forward to continue working with the community. Best, Lincoln Lee Shengkai Fang 于2023年1月12日周四 09:55写道： > Congratulations, Lincoln! > > Best, > Shengkai > > liu ron 于2023年1月12日周四 09:48写道： > > > Congratulations, Lincoln! > > > > Best > > > > Yu Li 于2023年1月12日周四 09:22写道： > > > > > Congratulations, Lincoln! > > > > > > Best Regards, > > > Yu > > > > > > > > > On Wed, 11 Jan 2023 at 21:17, Martijn Visser > > > > wrote: > > > > > > > Congratulations Lincoln, happy to have you on board! > > > > > > > > Best regards, Martijn > > > > > > > > > > > > On Wed, Jan 11, 2023 at 1:49 PM Dong Lin > wrote: > > > > > > > > > Congratulations, Lincoln! > > > > > > > > > > Cheers, > > > > > Dong > > > > > > > > > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a > > new > > > > > Flink > > > > > > committer. > > > > > > > > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He > > > > mainly > > > > > > works on Flink > > > > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 > (Retry > > > > Async > > > > > > I/O), FLIP-234 ( > > > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, > > He > > > > also > > > > > > contributed > > > > > > much to Streaming Semantics, including the non-determinism > problem > > > and > > > > > the > > > > > > message > > > > > > ordering problem. > > > > > > > > > > > > Please join me in congratulating Lincoln for becoming a Flink > > > > committer! > > > > > > > > > > > > Cheers, > > > > > > Jark Wu > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi, all, Thanks for all discussions about this FLIP first. We all are trying to make Flink better. But not getting a reply quickly really discourages contributors. OperatorCoordinatorMetricGroup and SplitEnumeratorMetricGroup are important for many developers. And many metrics can not be reported without it. This FLIP raised on 26 Dec 2022 and has last over a month. So let's make the discussion on FLIP-274 be settled. Thanks for all helps and let's make this FLIP move on together. Best, Hang Chesnay Schepler 于2023年1月12日周四 07:29写道： > You know, back when I wrote that line I actually felt a bit guilty about > it and nearly dropped it, since it implicitly accused _someone_ of > actually being capable/willing to push something through despite voiced > concerns while no ones looking. > > Turns out things are a lot worse that I thought. You're actually > doubling down on it and even insult me. > > > As I haven't had a time to revisit the discussion and afaict my concerns > weren't addressed, I hereby vote -1 (binding). > > Please be aware that since I'm effectively only back from vacation > tomorrow and need to catch up on things in general, I may not be able to > revisit the discussion this week. In part because I guess I now need to > double-check everything. > > CC'ing the private ML because I find this very troubling behavior. > > On 11/01/2023 16:20, Martijn Visser wrote: > > Hi Jark, > > > > I disagree with your statement that someone doesn't care about the FLIP, > > especially since that person participated in the initial FLIP discussion > > and mentioned explicitly "Since I'm on holidays soon, just so no one > tries > > to pull a fast one on me, if this were to go to a vote as-is I'd be > against > > it.". > > > > Anyway: I can't tell when Chesnay exactly will participate, but I do > expect > > that this week. > > > > Best regards, > > > > Martijn > > > > On Wed, Jan 11, 2023 at 4:11 PM Jark Wu wrote: > > > >> Hi Martijn, > >> > >> Thanks for helping us to find Chesnay. > >> Could you clarify how "soon" will Chesnay participate? A day or a week? > >> > >> I agree with your improvement proposal and Hang exactly follows your > >> points. > >> The last-call-for-discussion for Chensay has been sent for more than 72 > >> hours, > >> even after the Christmas holiday. Therefore, from my point of view, this > >> FLIP > >> should have been passed. On the other hand, the Flink Bylaws[1] don't > >> require > >> voting to begin until a specific person responds. The 3 days (even work > >> days) voting > >> length already considers the committers' reaction time. If someone > doesn't > >> vote -1 > >> during the time period, which means he/she doesn't have concerns about > the > >> proposal. > >> If you care about the proposal, PLEASE VOTE and explain the reasons. > This > >> is how > >> the 200+ FLIPs work until today. I'm just very disappointed and > surprised > >> we have to > >> wait for someone to vote for the FLIP who doesn't care about the FLIP > for > >> more than a month. > >> > >> > >> Best, > >> Jark > >> > >> [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > >> > >> On Wed, 11 Jan 2023 at 21:14, Martijn Visser > >> wrote: > >> > >>> Hi Jark and Dong, > >>> > >>> I fully understand your concerns in this case. I also think that this > >>> situation is an exception. This discussion started just before the > >> holiday > >>> season in Europe started. The request to Chesnay if he had more > comments > >>> was sent just a couple of days before Christmas. > >>> > >>> From my perspective, given how the discussion started in the > discussion > >>> thread and the context that was provided, I would have sent an email > that > >>> if there are no more comments in the next 72 hours, you would open a > vote > >>> thread. Especially if someone raised a concern first. That has > happened a > >>> lot on other discussion threads as well, even when there were no more > >> open > >>> discussion topics. > >>> > >>> While I fully understand the disappointment from your point of view, > the > >>> other way around it feels disappointing that this was brought to a > vote. > >> So > >>> let's use the disappointments from both ends to learn and to improve > >>> overall. Something like: > >>> > >>> * If someone raises concerns during a discussion, of course first try > to > >>> resolve all concerns. > >>> * If that person no longer participates in the discussion, send a > >>> last-call-for-discussions in the discussion thread for 72 hours and > else > >>> you will open a vote thread > >>> * Follow the regular voting process > >>> > >>> For this specific FLIP, I've briefly talked to Chesnay offline and I'm > >> sure > >>> he will participate soon to unblock it. > >>> > >>> Best regards, > >>> > >>> Martijn > >>> > >>> On Wed, Jan 11, 2023 at 1:47 PM Dong Lin wrote: > >>> > Hi Martijn, > > A collaborative and active community is very important for any > >>> open-source > project. Flink
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! Best, Shengkai liu ron 于2023年1月12日周四 09:48写道： > Congratulations, Lincoln! > > Best > > Yu Li 于2023年1月12日周四 09:22写道： > > > Congratulations, Lincoln! > > > > Best Regards, > > Yu > > > > > > On Wed, 11 Jan 2023 at 21:17, Martijn Visser > > wrote: > > > > > Congratulations Lincoln, happy to have you on board! > > > > > > Best regards, Martijn > > > > > > > > > On Wed, Jan 11, 2023 at 1:49 PM Dong Lin wrote: > > > > > > > Congratulations, Lincoln! > > > > > > > > Cheers, > > > > Dong > > > > > > > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a > new > > > > Flink > > > > > committer. > > > > > > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He > > > mainly > > > > > works on Flink > > > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry > > > Async > > > > > I/O), FLIP-234 ( > > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, > He > > > also > > > > > contributed > > > > > much to Streaming Semantics, including the non-determinism problem > > and > > > > the > > > > > message > > > > > ordering problem. > > > > > > > > > > Please join me in congratulating Lincoln for becoming a Flink > > > committer! > > > > > > > > > > Cheers, > > > > > Jark Wu > > > > > > > > > > > > > > >
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! Best Yu Li 于2023年1月12日周四 09:22写道： > Congratulations, Lincoln! > > Best Regards, > Yu > > > On Wed, 11 Jan 2023 at 21:17, Martijn Visser > wrote: > > > Congratulations Lincoln, happy to have you on board! > > > > Best regards, Martijn > > > > > > On Wed, Jan 11, 2023 at 1:49 PM Dong Lin wrote: > > > > > Congratulations, Lincoln! > > > > > > Cheers, > > > Dong > > > > > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > > > > > Hi everyone, > > > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new > > > Flink > > > > committer. > > > > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He > > mainly > > > > works on Flink > > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry > > Async > > > > I/O), FLIP-234 ( > > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He > > also > > > > contributed > > > > much to Streaming Semantics, including the non-determinism problem > and > > > the > > > > message > > > > ordering problem. > > > > > > > > Please join me in congratulating Lincoln for becoming a Flink > > committer! > > > > > > > > Cheers, > > > > Jark Wu > > > > > > > > > >
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! Best Regards, Yu On Wed, 11 Jan 2023 at 21:17, Martijn Visser wrote: > Congratulations Lincoln, happy to have you on board! > > Best regards, Martijn > > > On Wed, Jan 11, 2023 at 1:49 PM Dong Lin wrote: > > > Congratulations, Lincoln! > > > > Cheers, > > Dong > > > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > > > Hi everyone, > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new > > Flink > > > committer. > > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He > mainly > > > works on Flink > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry > Async > > > I/O), FLIP-234 ( > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He > also > > > contributed > > > much to Streaming Semantics, including the non-determinism problem and > > the > > > message > > > ordering problem. > > > > > > Please join me in congratulating Lincoln for becoming a Flink > committer! > > > > > > Cheers, > > > Jark Wu > > > > > >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
You know, back when I wrote that line I actually felt a bit guilty about it and nearly dropped it, since it implicitly accused _someone_ of actually being capable/willing to push something through despite voiced concerns while no ones looking. Turns out things are a lot worse that I thought. You're actually doubling down on it and even insult me. As I haven't had a time to revisit the discussion and afaict my concerns weren't addressed, I hereby vote -1 (binding). Please be aware that since I'm effectively only back from vacation tomorrow and need to catch up on things in general, I may not be able to revisit the discussion this week. In part because I guess I now need to double-check everything. CC'ing the private ML because I find this very troubling behavior. On 11/01/2023 16:20, Martijn Visser wrote: Hi Jark, I disagree with your statement that someone doesn't care about the FLIP, especially since that person participated in the initial FLIP discussion and mentioned explicitly "Since I'm on holidays soon, just so no one tries to pull a fast one on me, if this were to go to a vote as-is I'd be against it.". Anyway: I can't tell when Chesnay exactly will participate, but I do expect that this week. Best regards, Martijn On Wed, Jan 11, 2023 at 4:11 PM Jark Wu wrote: Hi Martijn, Thanks for helping us to find Chesnay. Could you clarify how "soon" will Chesnay participate? A day or a week? I agree with your improvement proposal and Hang exactly follows your points. The last-call-for-discussion for Chensay has been sent for more than 72 hours, even after the Christmas holiday. Therefore, from my point of view, this FLIP should have been passed. On the other hand, the Flink Bylaws[1] don't require voting to begin until a specific person responds. The 3 days (even work days) voting length already considers the committers' reaction time. If someone doesn't vote -1 during the time period, which means he/she doesn't have concerns about the proposal. If you care about the proposal, PLEASE VOTE and explain the reasons. This is how the 200+ FLIPs work until today. I'm just very disappointed and surprised we have to wait for someone to vote for the FLIP who doesn't care about the FLIP for more than a month. Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws On Wed, 11 Jan 2023 at 21:14, Martijn Visser wrote: Hi Jark and Dong, I fully understand your concerns in this case. I also think that this situation is an exception. This discussion started just before the holiday season in Europe started. The request to Chesnay if he had more comments was sent just a couple of days before Christmas. From my perspective, given how the discussion started in the discussion thread and the context that was provided, I would have sent an email that if there are no more comments in the next 72 hours, you would open a vote thread. Especially if someone raised a concern first. That has happened a lot on other discussion threads as well, even when there were no more open discussion topics. While I fully understand the disappointment from your point of view, the other way around it feels disappointing that this was brought to a vote. So let's use the disappointments from both ends to learn and to improve overall. Something like: * If someone raises concerns during a discussion, of course first try to resolve all concerns. * If that person no longer participates in the discussion, send a last-call-for-discussions in the discussion thread for 72 hours and else you will open a vote thread * Follow the regular voting process For this specific FLIP, I've briefly talked to Chesnay offline and I'm sure he will participate soon to unblock it. Best regards, Martijn On Wed, Jan 11, 2023 at 1:47 PM Dong Lin wrote: Hi Martijn, A collaborative and active community is very important for any open-source project. Flink can succeed today because we have many experienced and passionate developers who collaborate together to develop Flink. It is important that developers can give constructive feedback and help each other be productive. I find it kind of surprising and disappointing that we have to wait for one particular developer for more than 30 days to get a reply before being allowed forward and making progress. It is hard to image what would happen if every committer can take 30+ days to reply to a FLIP and still expects the FLIP to wait for the reply. Flink community will likely be dead if this is the culture that Flink community uses to treat contributors. Could you explain how long we have to wait before making progress for this FLIP? And in the future, what would be the resonable timeframe to wait for a reply before we can open the voting thread? Thanks, Dong On Wed, Jan 11, 2023 at 4:17 PM Martijn Visser < martijnvis...@apache.org wrote: -1 (binding) currently: I don't think this should have gone to a vote yet given that Chesnay
Re: "Introduction to Apache Flink" presentation
Hello Joe -- sorry for the late reply, a colleague just showed me this. I don't know if you're aware, but there's an incubating project called Apache Training (Incubating) [1] that collects introductory and training materials for various Apache projects. All ASF committers have write permissions on the incubator-training repository. For your first point, if you're looking for some common, shared reveal.js (and asciidoctor)-based tools, this might be a good home for a Flink presentation. There's still some room for improvement in the shared code, of course, but that's the point of sharing the code :D I did my ApacheCon Avro presentation with it (but STILL haven't gotten around to contributing it). There's not a lot of momentum in the Training project at the moment, but IMO it's fundamentally a pretty good idea for the community. If you go that route, I'd be willing to help out with review and conversion to reveal.js and asciidoctor; a lot of the conversion work is just fiddling around with layout until it looks OK. All that being said, the presentation looks really good and I'd probably be motivated to see it! (is there a video yet?) All my best, Ryan [1]: https://training.apache.org/ "Training website" [2]: https://training.apache.org/presentations/incubator/navigating-asf-incubation/index.html "Example deployed presentation" [3]: https://github.com/apache/incubator-training "GitHub repository" On Thu, Nov 17, 2022 at 11:57 AM Johannes Moser wrote: > Hi all, > > I was talking at a meetup yesterday about Apache Flink. > There is this Google Presentation that was created earlier and never has > been updated. As I had to come up with a presentation anyhow I used this > and updated it. See the result in [1]. > > First, I'd like to get feedback on the updated presentation. So please if > you got some time, have a look at it and comment. Everyone should be able > to do that. > > Having a default presentation that people can use to talk about Apache > Flink is a good thing, so I'd like to keep that. I want to test the > sentiment of the community with a couple of connected ideas. > > 1) Moving the presentation to a more sustainable technology that can also > be incorporated better into the Apache Flink workflow. > I'd suggest using something like reveal.js and adding the presentation to > flink-web, so everyone has an easier time contributing. I guess the > presentation would also benefit from a review process and an update could > be included in release processes. > > 2) Adding a list of people that might be open to giving that presentation, > so that Meetup organizers can reach out to them. Other communities have > such "community champion" setups. The Apache Flink community would benefit > from that as well. > > Happy to hear your feedback. > > Thanks, > Joe > > [1] > > https://eu01.z.antigena.com/l/FScYr7uhmgjism6OHA9nu7LhyL6Qnf6vwbVF9hyy6sxoGIX461XJZFXXYApTrdOSPMSaWQXc2cSpICl4dkelyd3n0FcDTlT4~ICYKcio~3oA3xaZByzwUqiDXr-po8ciAUIA12eHuVtoXH7ir9RJCzVwuj45r8Eql9UT9ENdseAP6mzz6vccT-L0V5YgdR98gCssQg5FhqpWPVD9Kx~twT7W5ZuyvHR9jvz91BNEXG-9ITh5 >
[jira] [Created] (FLINK-30639) Publish SBOM for externalized connectors
Márton Balassi created FLINK-30639: -- Summary: Publish SBOM for externalized connectors Key: FLINK-30639 URL: https://issues.apache.org/jira/browse/FLINK-30639 Project: Flink Issue Type: Sub-task Reporter: Márton Balassi Assignee: Zsombor Chikan Probably adding to the connector parent will be sufficient. https://github.com/apache/flink-connector-aws/blob/main/pom.xml#L27 https://github.com/zentol/flink-connector-parent/blob/master/pom.xml -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi Jark, I disagree with your statement that someone doesn't care about the FLIP, especially since that person participated in the initial FLIP discussion and mentioned explicitly "Since I'm on holidays soon, just so no one tries to pull a fast one on me, if this were to go to a vote as-is I'd be against it.". Anyway: I can't tell when Chesnay exactly will participate, but I do expect that this week. Best regards, Martijn On Wed, Jan 11, 2023 at 4:11 PM Jark Wu wrote: > Hi Martijn, > > Thanks for helping us to find Chesnay. > Could you clarify how "soon" will Chesnay participate? A day or a week? > > I agree with your improvement proposal and Hang exactly follows your > points. > The last-call-for-discussion for Chensay has been sent for more than 72 > hours, > even after the Christmas holiday. Therefore, from my point of view, this > FLIP > should have been passed. On the other hand, the Flink Bylaws[1] don't > require > voting to begin until a specific person responds. The 3 days (even work > days) voting > length already considers the committers' reaction time. If someone doesn't > vote -1 > during the time period, which means he/she doesn't have concerns about the > proposal. > If you care about the proposal, PLEASE VOTE and explain the reasons. This > is how > the 200+ FLIPs work until today. I'm just very disappointed and surprised > we have to > wait for someone to vote for the FLIP who doesn't care about the FLIP for > more than a month. > > > Best, > Jark > > [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws > > On Wed, 11 Jan 2023 at 21:14, Martijn Visser > wrote: > > > Hi Jark and Dong, > > > > I fully understand your concerns in this case. I also think that this > > situation is an exception. This discussion started just before the > holiday > > season in Europe started. The request to Chesnay if he had more comments > > was sent just a couple of days before Christmas. > > > > From my perspective, given how the discussion started in the discussion > > thread and the context that was provided, I would have sent an email that > > if there are no more comments in the next 72 hours, you would open a vote > > thread. Especially if someone raised a concern first. That has happened a > > lot on other discussion threads as well, even when there were no more > open > > discussion topics. > > > > While I fully understand the disappointment from your point of view, the > > other way around it feels disappointing that this was brought to a vote. > So > > let's use the disappointments from both ends to learn and to improve > > overall. Something like: > > > > * If someone raises concerns during a discussion, of course first try to > > resolve all concerns. > > * If that person no longer participates in the discussion, send a > > last-call-for-discussions in the discussion thread for 72 hours and else > > you will open a vote thread > > * Follow the regular voting process > > > > For this specific FLIP, I've briefly talked to Chesnay offline and I'm > sure > > he will participate soon to unblock it. > > > > Best regards, > > > > Martijn > > > > On Wed, Jan 11, 2023 at 1:47 PM Dong Lin wrote: > > > > > Hi Martijn, > > > > > > A collaborative and active community is very important for any > > open-source > > > project. Flink can succeed today because we have many experienced and > > > passionate developers who collaborate together to develop Flink. It is > > > important that developers can give constructive feedback and help each > > > other be productive. > > > > > > I find it kind of surprising and disappointing that we have to wait for > > one > > > particular developer for more than 30 days to get a reply before being > > > allowed forward and making progress. It is hard to image what would > > happen > > > if every committer can take 30+ days to reply to a FLIP and still > expects > > > the FLIP to wait for the reply. Flink community will likely be dead if > > this > > > is the culture that Flink community uses to treat contributors. > > > > > > Could you explain how long we have to wait before making progress for > > this > > > FLIP? And in the future, what would be the resonable timeframe to wait > > for > > > a reply before we can open the voting thread? > > > > > > Thanks, > > > Dong > > > > > > > > > On Wed, Jan 11, 2023 at 4:17 PM Martijn Visser < > martijnvis...@apache.org > > > > > > wrote: > > > > > > > -1 (binding) currently: I don't think this should have gone to a vote > > yet > > > > given that Chesnay deliberately mentioned that he would vote against > it > > > > as-is. The discussion should have been settled first. > > > > > > > > On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Thanks, > > > > > Zhu > > > > > > > > > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Best, > > > > > > Jark > > > > > > > > > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > > > >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi Martijn, Thanks for helping us to find Chesnay. Could you clarify how "soon" will Chesnay participate? A day or a week? I agree with your improvement proposal and Hang exactly follows your points. The last-call-for-discussion for Chensay has been sent for more than 72 hours, even after the Christmas holiday. Therefore, from my point of view, this FLIP should have been passed. On the other hand, the Flink Bylaws[1] don't require voting to begin until a specific person responds. The 3 days (even work days) voting length already considers the committers' reaction time. If someone doesn't vote -1 during the time period, which means he/she doesn't have concerns about the proposal. If you care about the proposal, PLEASE VOTE and explain the reasons. This is how the 200+ FLIPs work until today. I'm just very disappointed and surprised we have to wait for someone to vote for the FLIP who doesn't care about the FLIP for more than a month. Best, Jark [1]: https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws On Wed, 11 Jan 2023 at 21:14, Martijn Visser wrote: > Hi Jark and Dong, > > I fully understand your concerns in this case. I also think that this > situation is an exception. This discussion started just before the holiday > season in Europe started. The request to Chesnay if he had more comments > was sent just a couple of days before Christmas. > > From my perspective, given how the discussion started in the discussion > thread and the context that was provided, I would have sent an email that > if there are no more comments in the next 72 hours, you would open a vote > thread. Especially if someone raised a concern first. That has happened a > lot on other discussion threads as well, even when there were no more open > discussion topics. > > While I fully understand the disappointment from your point of view, the > other way around it feels disappointing that this was brought to a vote. So > let's use the disappointments from both ends to learn and to improve > overall. Something like: > > * If someone raises concerns during a discussion, of course first try to > resolve all concerns. > * If that person no longer participates in the discussion, send a > last-call-for-discussions in the discussion thread for 72 hours and else > you will open a vote thread > * Follow the regular voting process > > For this specific FLIP, I've briefly talked to Chesnay offline and I'm sure > he will participate soon to unblock it. > > Best regards, > > Martijn > > On Wed, Jan 11, 2023 at 1:47 PM Dong Lin wrote: > > > Hi Martijn, > > > > A collaborative and active community is very important for any > open-source > > project. Flink can succeed today because we have many experienced and > > passionate developers who collaborate together to develop Flink. It is > > important that developers can give constructive feedback and help each > > other be productive. > > > > I find it kind of surprising and disappointing that we have to wait for > one > > particular developer for more than 30 days to get a reply before being > > allowed forward and making progress. It is hard to image what would > happen > > if every committer can take 30+ days to reply to a FLIP and still expects > > the FLIP to wait for the reply. Flink community will likely be dead if > this > > is the culture that Flink community uses to treat contributors. > > > > Could you explain how long we have to wait before making progress for > this > > FLIP? And in the future, what would be the resonable timeframe to wait > for > > a reply before we can open the voting thread? > > > > Thanks, > > Dong > > > > > > On Wed, Jan 11, 2023 at 4:17 PM Martijn Visser > > > wrote: > > > > > -1 (binding) currently: I don't think this should have gone to a vote > yet > > > given that Chesnay deliberately mentioned that he would vote against it > > > as-is. The discussion should have been settled first. > > > > > > On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > > > > > > > +1 (binding) > > > > > > > > Thanks, > > > > Zhu > > > > > > > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > > > > > > > +1 (binding) > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > > > > > > > Thanks for the FLIP! > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > Best, > > > > > > Qingsheng > > > > > > > > > > > > On Wed, Jan 4, 2023 at 10:08 AM Hang Ruan < > ruanhang1...@gmail.com> > > > > wrote: > > > > > > > > > > > >> Hi all, > > > > > >> > > > > > >> Thanks for all the feedback so far. > > > > > >> Based on the discussion[1], we have come to a consensus, so I > > would > > > > like to > > > > > >> start a vote on FLIP-274: Introduce metric group for > > > > > >> OperatorCoordinator[2]. > > > > > >> > > > > > >> The vote will last for at least 72 hours (Jan 7th at 11:00 GMT) > > > unless > > > > > >> there is an objection or insufficient votes. > > > > > >> > > > > > >> [1] > > >
[jira] [Created] (FLINK-30638) Operator doesn't forward allowNonRestoredState flag
Alexis Sarda-Espinosa created FLINK-30638: - Summary: Operator doesn't forward allowNonRestoredState flag Key: FLINK-30638 URL: https://issues.apache.org/jira/browse/FLINK-30638 Project: Flink Issue Type: Bug Components: Kubernetes Operator Affects Versions: kubernetes-operator-1.3.1 Reporter: Alexis Sarda-Espinosa Attachments: flink-kubernetes-operator_json.txt My Flink image is currently based on 1.15, and I have HA enabled. After an upgrade where I removed some transformations from the pipeline, I see the following exception in the deployec Flink cluster: Job 38585f46d840b3c2e9f843e230398d35 reached terminal state FAILED. org.apache.flink.runtime.client.JobInitializationException: Could not start the JobMaster. ... Caused by: java.util.concurrent.CompletionException: java.lang.IllegalStateException: There is no operator for the state d3bab17db5b56f81eb48b3e7d5cce15d I'm attaching some of the operator's logs, but I don't see anything strange there. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] FLIP-281 Sink Supports Speculative Execution For Batch Job
Hi Biao, Apologies for the late jumping in. My only question is about SinkFunction, does this imply that we want to add support for this to the SinkFunction? If so, I would not be in favour of that since we would like to deprecate (I actually thought that was already the case) the SinkFunction in favour of SinkV2. Besides that, I have no other comments. Best regards, Martijn On Wed, Jan 4, 2023 at 7:28 AM Jing Zhang wrote: > Hi Biao, > > Thanks for explanation. > > +1 for the proposal. > > Best, > Jing Zhang > > Lijie Wang 于2023年1月4日周三 12:11写道： > > > Hi Biao, > > > > Thanks for the explanation of how SinkV2 knows the right subtask > > attempt. I have no more questions, +1 for the proposal. > > > > Best, > > Lijie > > > > Biao Liu 于2022年12月28日周三 17:22写道： > > > > > Thanks for all your feedback! > > > > > > To @Yuxia, > > > > > > > What the sink expect to do to isolate data produced by speculative > > > > executions? IIUC, if the taks failover, it also generate a new > > attempt. > > > > Does it make difference in isolating data produced? > > > > > > > > > Yes there is something different from the task failover scenario. The > > > attempt number is more necessary for speculative execution than > failover. > > > Because there can be only one subtask instance running at the same time > > in > > > the failover scenario. > > > > > > Let's take FileSystemOutputFormat as an example. For the failover > > scenario, > > > the temporary directory to store produced data can be something like > > > "$root_dir/task-$taskNumber/". At the initialization phase, subtask > > deletes > > > and re-creates the temporary directory. > > > > > > However in the speculative execution scenario, it does not work because > > > there might be several subtasks running at the same time. These > subtasks > > > might delete, re-create and write the same temporary directory at the > > > same time. The correct temporary directory should be like > > > "$root_dir/task-$taskNumber/attempt-$attemptNumber". So it's necessary > to > > > expose the attempt number to the Sink implementation to do the data > > > isolation. > > > > > > > > > To @Lijie, > > > > > > > I have a question about this: does SinkV2 need to do the same thing? > > > > > > > > > Actually, yes. > > > > > > Should we/users do it in the committer? If yes, how does the commiter > > know > > > > which one is the right subtask attempt? > > > > > > > > > Yes, we/users should do it in the committer. > > > > > > In the current design, the Committer of Sink V2 should get the "which > one > > > is the right subtask attempt" information from the "committable data'' > > > produced by SinkWriter. Let's take the FileSink as example, the > > > "committable data" sent to the Committer contains the full path of the > > > files produced by SinkWriter. Users could also pass the attempt number > > > through "committable data" from SinkWriter to Committer. > > > > > > In the "Rejected Alternatives -> Introduce a way to clean leaked data > of > > > Sink V2" section of the FLIP document, we discussed some of the reasons > > > that we didn't provide the API like OutputFormat. > > > > > > To @Jing Zhang > > > > > > I have a question about this: Speculative execution of Committer will > be > > > > disabled. > > > > > > I agree with your point and I saw the similar requirements to disable > > > speculative > > > > execution for specified operators. > > > > > > However the requirement is not supported currently. I think there > > > should be some > > > > place to describe how to support it. > > > > > > > > > In this FLIP design, the speculative execution of Committer of Sink V2 > > will > > > be disabled by Flink. It's not an optional operation. Users can not > > change > > > it. > > > And as you said, "disable speculative execution for specified > operators" > > is > > > not supported in the FLIP. Because it's a bit out of scope: "Sink > > Supports > > > Speculative Execution For Batch Job". I think it's better to start > > another > > > FLIP to discuss it. "Fine-grained control of enabling speculative > > execution > > > for operators" can be the title of that FLIP. And we can discuss there > > how > > > to enable or disable speculative execution for specified operators > > > including Committer and pre/post-committer of Sink V2. > > > > > > What do you think? > > > > > > Thanks, > > > Biao /'bɪ.aʊ/ > > > > > > > > > > > > On Wed, 28 Dec 2022 at 11:30, Jing Zhang wrote: > > > > > > > Hi Biao, > > > > > > > > Thanks for driving this FLIP. It's meaningful to support speculative > > > > execution > > > > of sinks is important. > > > > > > > > I have a question about this: Speculative execution of Committer will > > be > > > > disabled. > > > > > > > > I agree with your point and I saw the similar requirements to disable > > > > speculative execution for specified operators. > > > > > > > > However the requirement is not supported currently. I think there > > should > > > be > > > > some place to
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations Lincoln, happy to have you on board! Best regards, Martijn On Wed, Jan 11, 2023 at 1:49 PM Dong Lin wrote: > Congratulations, Lincoln! > > Cheers, > Dong > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new > Flink > > committer. > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He mainly > > works on Flink > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry Async > > I/O), FLIP-234 ( > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He also > > contributed > > much to Streaming Semantics, including the non-determinism problem and > the > > message > > ordering problem. > > > > Please join me in congratulating Lincoln for becoming a Flink committer! > > > > Cheers, > > Jark Wu > > >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi Jark and Dong, I fully understand your concerns in this case. I also think that this situation is an exception. This discussion started just before the holiday season in Europe started. The request to Chesnay if he had more comments was sent just a couple of days before Christmas. >From my perspective, given how the discussion started in the discussion thread and the context that was provided, I would have sent an email that if there are no more comments in the next 72 hours, you would open a vote thread. Especially if someone raised a concern first. That has happened a lot on other discussion threads as well, even when there were no more open discussion topics. While I fully understand the disappointment from your point of view, the other way around it feels disappointing that this was brought to a vote. So let's use the disappointments from both ends to learn and to improve overall. Something like: * If someone raises concerns during a discussion, of course first try to resolve all concerns. * If that person no longer participates in the discussion, send a last-call-for-discussions in the discussion thread for 72 hours and else you will open a vote thread * Follow the regular voting process For this specific FLIP, I've briefly talked to Chesnay offline and I'm sure he will participate soon to unblock it. Best regards, Martijn On Wed, Jan 11, 2023 at 1:47 PM Dong Lin wrote: > Hi Martijn, > > A collaborative and active community is very important for any open-source > project. Flink can succeed today because we have many experienced and > passionate developers who collaborate together to develop Flink. It is > important that developers can give constructive feedback and help each > other be productive. > > I find it kind of surprising and disappointing that we have to wait for one > particular developer for more than 30 days to get a reply before being > allowed forward and making progress. It is hard to image what would happen > if every committer can take 30+ days to reply to a FLIP and still expects > the FLIP to wait for the reply. Flink community will likely be dead if this > is the culture that Flink community uses to treat contributors. > > Could you explain how long we have to wait before making progress for this > FLIP? And in the future, what would be the resonable timeframe to wait for > a reply before we can open the voting thread? > > Thanks, > Dong > > > On Wed, Jan 11, 2023 at 4:17 PM Martijn Visser > wrote: > > > -1 (binding) currently: I don't think this should have gone to a vote yet > > given that Chesnay deliberately mentioned that he would vote against it > > as-is. The discussion should have been settled first. > > > > On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > > > > > +1 (binding) > > > > > > Thanks, > > > Zhu > > > > > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Jark > > > > > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > > > > > Thanks for the FLIP! > > > > > > > > > > +1 (binding) > > > > > > > > > > Best, > > > > > Qingsheng > > > > > > > > > > On Wed, Jan 4, 2023 at 10:08 AM Hang Ruan > > > wrote: > > > > > > > > > >> Hi all, > > > > >> > > > > >> Thanks for all the feedback so far. > > > > >> Based on the discussion[1], we have come to a consensus, so I > would > > > like to > > > > >> start a vote on FLIP-274: Introduce metric group for > > > > >> OperatorCoordinator[2]. > > > > >> > > > > >> The vote will last for at least 72 hours (Jan 7th at 11:00 GMT) > > unless > > > > >> there is an objection or insufficient votes. > > > > >> > > > > >> [1] > > https://lists.apache.org/thread/63m9w60rndqnrqvgb6qosvt2bcbww53k > > > > >> [2] > > > > >> > > > > >> > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-274%3A+Introduce+metric+group+for+OperatorCoordinator > > > > >> > > > > >> Best, > > > > >> Hang > > > > >> > > > > > > > > > >
[jira] [Created] (FLINK-30637) In linux-aarch64 environment, using “is” judgment to match the window type of overwindow have returned incorrect matching results
Xin Chen created FLINK-30637: Summary: In linux-aarch64 environment, using “is” judgment to match the window type of overwindow have returned incorrect matching results Key: FLINK-30637 URL: https://issues.apache.org/jira/browse/FLINK-30637 Project: Flink Issue Type: Bug Components: API / Python Affects Versions: 1.13.6 Environment: Linux version 5.10.0-60.18.0.50.oe2203.aarch64 (abuild@obs-worker-002) (gcc_old (GCC) 10.3.1, GNU ld (GNU Binutils) 2.37) #1 SMP Wed Mar 30 02:43:08 UTC 2022 pyflink-version：1.13.6 Reporter: Xin Chen Fix For: 1.13.6 In linux-arch64 environment, “window_type is OverWindow.ROW_UNBOUNDED_FOLLOWING” in in the PandasBatchOverWindowAggregateFunctionOperation class of the pyflink source code has returned the wrong result. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! Cheers, Dong On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new Flink > committer. > > Lincoln Lee has been a long-term Flink contributor since 2017. He mainly > works on Flink > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry Async > I/O), FLIP-234 ( > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He also > contributed > much to Streaming Semantics, including the non-determinism problem and the > message > ordering problem. > > Please join me in congratulating Lincoln for becoming a Flink committer! > > Cheers, > Jark Wu >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi Martijn, A collaborative and active community is very important for any open-source project. Flink can succeed today because we have many experienced and passionate developers who collaborate together to develop Flink. It is important that developers can give constructive feedback and help each other be productive. I find it kind of surprising and disappointing that we have to wait for one particular developer for more than 30 days to get a reply before being allowed forward and making progress. It is hard to image what would happen if every committer can take 30+ days to reply to a FLIP and still expects the FLIP to wait for the reply. Flink community will likely be dead if this is the culture that Flink community uses to treat contributors. Could you explain how long we have to wait before making progress for this FLIP? And in the future, what would be the resonable timeframe to wait for a reply before we can open the voting thread? Thanks, Dong On Wed, Jan 11, 2023 at 4:17 PM Martijn Visser wrote: > -1 (binding) currently: I don't think this should have gone to a vote yet > given that Chesnay deliberately mentioned that he would vote against it > as-is. The discussion should have been settled first. > > On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > > > +1 (binding) > > > > > > Best, > > > Jark > > > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > > > Thanks for the FLIP! > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Qingsheng > > > > > > > > On Wed, Jan 4, 2023 at 10:08 AM Hang Ruan > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> Thanks for all the feedback so far. > > > >> Based on the discussion[1], we have come to a consensus, so I would > > like to > > > >> start a vote on FLIP-274: Introduce metric group for > > > >> OperatorCoordinator[2]. > > > >> > > > >> The vote will last for at least 72 hours (Jan 7th at 11:00 GMT) > unless > > > >> there is an objection or insufficient votes. > > > >> > > > >> [1] > https://lists.apache.org/thread/63m9w60rndqnrqvgb6qosvt2bcbww53k > > > >> [2] > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-274%3A+Introduce+metric+group+for+OperatorCoordinator > > > >> > > > >> Best, > > > >> Hang > > > >> > > > > > >
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congrats, Lincoln! Well deserved! > On Jan 11, 2023, at 11:23 AM, Xintong Song wrote: > > Congratulations, Lincoln! Well deserved. > > Best, > > Xintong > > > > On Wed, Jan 11, 2023 at 5:48 PM Matthias Pohl > wrote: > >> Congratulations, Lincoln! >> >> On Wed, Jan 11, 2023 at 10:17 AM Shuo Cheng wrote: >> >>> Congratulations, Lincoln! >>> >>> On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: >>> Hi everyone, On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new >>> Flink committer. Lincoln Lee has been a long-term Flink contributor since 2017. He >> mainly works on Flink SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry >> Async I/O), FLIP-234 ( Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He >> also contributed much to Streaming Semantics, including the non-determinism problem and >>> the message ordering problem. Please join me in congratulating Lincoln for becoming a Flink >> committer! Cheers, Jark Wu >>> >>
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! Well deserved. Best, Xintong On Wed, Jan 11, 2023 at 5:48 PM Matthias Pohl wrote: > Congratulations, Lincoln! > > On Wed, Jan 11, 2023 at 10:17 AM Shuo Cheng wrote: > > > Congratulations, Lincoln! > > > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > > > Hi everyone, > > > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new > > Flink > > > committer. > > > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He > mainly > > > works on Flink > > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry > Async > > > I/O), FLIP-234 ( > > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He > also > > > contributed > > > much to Streaming Semantics, including the non-determinism problem and > > the > > > message > > > ordering problem. > > > > > > Please join me in congratulating Lincoln for becoming a Flink > committer! > > > > > > Cheers, > > > Jark Wu > > > > > >
[jira] [Created] (FLINK-30636) Typo fix: "to to" -> "to"
Gunnar Morling created FLINK-30636: -- Summary: Typo fix: "to to" -> "to" Key: FLINK-30636 URL: https://issues.apache.org/jira/browse/FLINK-30636 Project: Flink Issue Type: Technical Debt Reporter: Gunnar Morling There's a surprising number of occurrences of "to to" in JavaDoc and the like, where it actually should just be "to". -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! On Wed, Jan 11, 2023 at 10:17 AM Shuo Cheng wrote: > Congratulations, Lincoln! > > On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new > Flink > > committer. > > > > Lincoln Lee has been a long-term Flink contributor since 2017. He mainly > > works on Flink > > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry Async > > I/O), FLIP-234 ( > > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He also > > contributed > > much to Streaming Semantics, including the non-determinism problem and > the > > message > > ordering problem. > > > > Please join me in congratulating Lincoln for becoming a Flink committer! > > > > Cheers, > > Jark Wu > > >
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
Hi Martijn, I think we should have a reasonable time window to wait for the discussion response. Many proposals will halt if we have to wait for every reply indefinitely. Chesnay raised some concerns in the thread one month ago and no objections or feedback after that. Hang replied to all the questions and others also think the concerns were addressed. Hang also announced waiting for the final discussion feedback on 26 Dec 2022, but no feedback was received from Chesnay before the vote started on 4 Jan 2023. So I think the discussion has been settled. I think we shouldn't block a FLIP indefinitely. Either having a clear waiting time or continuing the development of the FLIP. In the end, we still have a chance to revisit the FLIP before the 1.17 release if Chesnay comes back to the community. Best, Jark On Wed, 11 Jan 2023 at 16:17, Martijn Visser wrote: > -1 (binding) currently: I don't think this should have gone to a vote yet > given that Chesnay deliberately mentioned that he would vote against it > as-is. The discussion should have been settled first. > > On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > > > +1 (binding) > > > > Thanks, > > Zhu > > > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > > > +1 (binding) > > > > > > Best, > > > Jark > > > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > > > Thanks for the FLIP! > > > > > > > > +1 (binding) > > > > > > > > Best, > > > > Qingsheng > > > > > > > > On Wed, Jan 4, 2023 at 10:08 AM Hang Ruan > > wrote: > > > > > > > >> Hi all, > > > >> > > > >> Thanks for all the feedback so far. > > > >> Based on the discussion[1], we have come to a consensus, so I would > > like to > > > >> start a vote on FLIP-274: Introduce metric group for > > > >> OperatorCoordinator[2]. > > > >> > > > >> The vote will last for at least 72 hours (Jan 7th at 11:00 GMT) > unless > > > >> there is an objection or insufficient votes. > > > >> > > > >> [1] > https://lists.apache.org/thread/63m9w60rndqnrqvgb6qosvt2bcbww53k > > > >> [2] > > > >> > > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-274%3A+Introduce+metric+group+for+OperatorCoordinator > > > >> > > > >> Best, > > > >> Hang > > > >> > > > > > >
[jira] [Created] (FLINK-30635) Fail to recover for row-format when writing to OSS using FileSystemTable connector
luoyuxia created FLINK-30635: Summary: Fail to recover for row-format when writing to OSS using FileSystemTable connector Key: FLINK-30635 URL: https://issues.apache.org/jira/browse/FLINK-30635 Project: Flink Issue Type: Bug Components: Connectors / FileSystem Reporter: luoyuxia -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30634) HiveTableSourceITCase.setupSourceDatabaseAndData fails with InvalidObjectException
Matthias Pohl created FLINK-30634: - Summary: HiveTableSourceITCase.setupSourceDatabaseAndData fails with InvalidObjectException Key: FLINK-30634 URL: https://issues.apache.org/jira/browse/FLINK-30634 Project: Flink Issue Type: Bug Components: Connectors / Hive Affects Versions: 1.16.0 Reporter: Matthias Pohl We have multiple failures in HiveTableSourceITCase https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44692=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=d04c9862-880c-52f5-574b-a7a79fef8e0f=27933 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [ANNOUNCE] New Apache Flink Committer - Lincoln Lee
Congratulations, Lincoln! On Tue, Jan 10, 2023 at 11:52 AM Jark Wu wrote: > Hi everyone, > > On behalf of the PMC, I'm very happy to announce Lincoln Lee as a new Flink > committer. > > Lincoln Lee has been a long-term Flink contributor since 2017. He mainly > works on Flink > SQL parts and drives several important FLIPs, e.g., FLIP-232 (Retry Async > I/O), FLIP-234 ( > Retryable Lookup Join), FLIP-260 (TableFunction Finish). Besides, He also > contributed > much to Streaming Semantics, including the non-determinism problem and the > message > ordering problem. > > Please join me in congratulating Lincoln for becoming a Flink committer! > > Cheers, > Jark Wu >
[jira] [Created] (FLINK-30633) Update AWS SDKv2 to v2.19.14
Danny Cranmer created FLINK-30633: - Summary: Update AWS SDKv2 to v2.19.14 Key: FLINK-30633 URL: https://issues.apache.org/jira/browse/FLINK-30633 Project: Flink Issue Type: Improvement Components: Connectors / AWS, Connectors / DynamoDB, Connectors / Firehose, Connectors / Kinesis Reporter: Danny Cranmer Fix For: 1.17.0, 1.16.1, 1.15.4, aws-connector-4.1.0, aws-connector-3.1.0 Update AWS SDK to latest version since it fixes a bug: - https://issues.apache.org/jira/browse/FLINK-30304 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30632) Introduce DataType for table store
Jingsong Lee created FLINK-30632: Summary: Introduce DataType for table store Key: FLINK-30632 URL: https://issues.apache.org/jira/browse/FLINK-30632 Project: Flink Issue Type: Sub-task Components: Table Store Reporter: Jingsong Lee Assignee: Jingsong Lee Fix For: table-store-0.4.0 Introduce table store own DataType to decouple Flink SQL LogicalType. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30631) Limit the max number of subpartitons consumed by each downstream task
Lijie Wang created FLINK-30631: -- Summary: Limit the max number of subpartitons consumed by each downstream task Key: FLINK-30631 URL: https://issues.apache.org/jira/browse/FLINK-30631 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Reporter: Lijie Wang Fix For: 1.17.0 In the current implementation(FLINK-25035), when the upstream vertex parallelism is much greater than the downstream vertex parallelism, it may lead to a large number of channels in the downstream tasks(for example, A -> B, all to all edge, max parallelism is 1000. If parallelism of A is 1000, parallelism of B is decided to be 1, then the only subtask of B will consume 1000 * 1000 subpartitions), resulting in a large overhead for processing channels. In this ticket, we temporarily address this issue by limiting the max number of subpartitons consumed by each downstream task. The ultimate solution should be to support single channel consume multiple subpartitons. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30630) Python: BatchModeDataStreamTests.test_keyed_map failed
Matthias Pohl created FLINK-30630:
-
Summary: Python: BatchModeDataStreamTests.test_keyed_map failed
Key: FLINK-30630
URL: https://issues.apache.org/jira/browse/FLINK-30630
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.15.3
Reporter: Matthias Pohl
{{BatchModeDataStreamTests.test_keyed_map}] failed in 1.15:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44691=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30629) ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat is unstable
Xintong Song created FLINK-30629:
Summary: ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat
is unstable
Key: FLINK-30629
URL: https://issues.apache.org/jira/browse/FLINK-30629
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.17.0
Reporter: Xintong Song
Fix For: 1.17.0
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44690=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=10819
{code:java}
Jan 11 04:32:39 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time
elapsed: 21.02 s <<< FAILURE! - in org.apache.flink.client.ClientHeartbeatTest
Jan 11 04:32:39 [ERROR]
org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat
Time elapsed: 9.157 s <<< ERROR!
Jan 11 04:32:39 java.lang.IllegalStateException: MiniCluster is not yet running
or has already been shut down.
Jan 11 04:32:39 at
org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
Jan 11 04:32:39 at
org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:1044)
Jan 11 04:32:39 at
org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:917)
Jan 11 04:32:39 at
org.apache.flink.runtime.minicluster.MiniCluster.getJobStatus(MiniCluster.java:841)
Jan 11 04:32:39 at
org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobStatus(MiniClusterJobClient.java:91)
Jan 11 04:32:39 at
org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat(ClientHeartbeatTest.java:79)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [VOTE] FLIP-274: Introduce metric group for OperatorCoordinator
-1 (binding) currently: I don't think this should have gone to a vote yet given that Chesnay deliberately mentioned that he would vote against it as-is. The discussion should have been settled first. On Tue, Jan 10, 2023 at 10:51 AM Zhu Zhu wrote: > +1 (binding) > > Thanks, > Zhu > > Jark Wu 于2023年1月10日周二 17:43写道： > > > > +1 (binding) > > > > Best, > > Jark > > > > > 2023年1月10日 12:02，Qingsheng Ren 写道： > > > > > > Thanks for the FLIP! > > > > > > +1 (binding) > > > > > > Best, > > > Qingsheng > > > > > > On Wed, Jan 4, 2023 at 10:08 AM Hang Ruan > wrote: > > > > > >> Hi all, > > >> > > >> Thanks for all the feedback so far. > > >> Based on the discussion[1], we have come to a consensus, so I would > like to > > >> start a vote on FLIP-274: Introduce metric group for > > >> OperatorCoordinator[2]. > > >> > > >> The vote will last for at least 72 hours (Jan 7th at 11:00 GMT) unless > > >> there is an objection or insufficient votes. > > >> > > >> [1] https://lists.apache.org/thread/63m9w60rndqnrqvgb6qosvt2bcbww53k > > >> [2] > > >> > > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-274%3A+Introduce+metric+group+for+OperatorCoordinator > > >> > > >> Best, > > >> Hang > > >> > > >
Re: [DISCUSS] FLIP-274 : Introduce metric group for OperatorCoordinator
Hi Hang, I'm a bit surprised that this has gone to a vote, given that Chesnay deliberately mentioned that he would vote against it as-is. I would expect that before going to a vote, he has had the opportunity to participate in this discussion. Best regards, Martijn On Tue, Jan 3, 2023 at 12:53 PM Jark Wu wrote: > Hi Dong, > > Regarding “SplitEnumeratorContext#metricGroup”, my only concern is that > this is a core interface for the FLIP. It’s hard to tell how sources use > metric group without mentioning this interface. Even if this is an existing > API, I think it’s worth introducing the interface again and declaring that > we will implement the interface instead of a no-op method in this FLIP. > > Anyway, this is a minor problem and shouldn’t block this FLIP. I’m +1 to > start a vote. > > Best, > Jark > > > > 2023年1月3日 10:03，Hang Ruan 写道： > > > > Hi, Jark and Dong, > > > > Thanks for your comments. Sorry for my late reply. > > > > For suggestion 1, I plan to implement the SplitEnumeratorMetricGroup in > > another issue, and it is not contained in this FLIP. I will add some > > description about this part. > > For suggestion 2, changes about OperatorCoordinator#metricGroup has > already > > been documented in the proposed change section. > > > > Best, > > Hang > > > > Dong Lin 于2023年1月1日周日 09:45写道： > > > >> Let me chime-in and add comments regarding the public interface section. > >> Please see my comments inline. > >> > >> On Thu, Dec 29, 2022 at 6:08 PM Jark Wu wrote: > >> > >>> Hi Hang, > >>> > >>> Thanks for driving this discussion. I think this is a very useful > feature > >>> for connectors. > >>> > >>> The FLIP looks quite good to me, and I just have two suggestions. > >>> > >>> 1. In the "Public Interface" section, mention that the implementation > >>> behavior of "SplitEnumeratorContext#metricGroup" is changed from > >> returning > >>> null to returning a concrete SplitEnumeratorMetricGroup instance. Even > >>> though the API is already there, the behavior change can also be > >> considered > >>> a public change. > >>> > >> > >> SplitEnumeratorContext#metricGroup is an interface and this FLIP does > not > >> seem to change its semantics/behavior. The FLIP does change the > >> implementation/behavior of SourceCoordinatorContext#metricGroup, which > is > >> marked @Internal. > >> > >> Thus it might seem a bit weird to add in the public interface section > >> saying that we change the interface SplitEnumeratorContext#metricGroup > from > >> returning null to non-null object. > >> > >> 2. Mention the newly added interface of > "OperatorCoordinator#metricGroup" > >>> in the "Proposed Changes" section or "Public Interface" section. As the > >>> FLIP said, OperatorCoordinator is widely used in many connectors. > Though > >> it > >>> is still an @Internal API, I think it is worth mentioning the change in > >> the > >>> FLIP. > >> > >> > >> Since OperatorCoordinator is an internal API, it seems reasonable to > >> explain it in the proposed change section. The FLIP seems to have > >> documented this in point 5 of the proposed change section. > >> > >> BTW, if we think there are @internal classes that are important enough > to > >> be added in the public interface section, it might be useful to > explicitly > >> discuss this topic and document it in the "*What are the "public > >> interfaces" of the project*" in this > >> < > >> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals > >>> > >> wiki. > >> > >> > >>> Best, > >>> Jark > >>> > >>> > >>> On Mon, 26 Dec 2022 at 18:06, Hang Ruan > wrote: > >>> > Hi, thanks for the feedback, Zhu Zhu and Qingsheng. > After combining everyone's comments, the main concerns and > >> corresponding > adjustments are as follows. > > Q1: Common metrics are not quite useful. > numEventsIn and numEventsOut counters will be removed from the > OperatorCoordinatorMetricGroup. These common metrics do not provide > >>> enough > information for users. The users are more willing to get the number of > events of the specified type instead of the total number. And this > >> metric > is calculated differently. The implementation could register the > metric > >>> by > themselves. > > Q2: This FLIP is overly complicated. > This FLIP will become concise after these modifications. > OperatorCoordinatorMetricGroup has already been introduced into Flink > >> by > FLIP-179< > > > >>> > >> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-179%3A+Expose+Standardized+Operator+Metrics > > . > And > this FLIP will not change it. This FLIP only provides a new metric > >> option > and a new metric group scope. The changes in proposed changes provide > >> the > details about the modifications for the internal classes, which might > >>> make > it look complicated. > > Thanks for all the comments again. If there are no further
