Re: Question about ZooKeeper HA new structure [FLINK-22636]

2021-09-03 Thread Till Rohrmann
rojects/flink/flink-docs-master/docs/ops/upgrading/#upgrading-the-flink-framework-version > > > > On Fri, Sep 3, 2021 at 9:49 AM Till Rohrmann wrote: > > > Hi Juha, > > > > Flink does not give backwards compatibility wrt to its internal data > > structures

Re: Question about ZooKeeper HA new structure [FLINK-22636]

2021-09-03 Thread Till Rohrmann
Hi Juha, Flink does not give backwards compatibility wrt to its internal data structures. The recommended way is to stop the jobs with a savepoint and then resume these jobs on the new Flink cluster. Failing over the processes with a new version is not guaranteed to work atm. I hope this answers

[jira] [Created] (FLINK-24133) PartitionRequestClientFactoryTest.testThrowsWhenNetworkFailure fails on Azure

2021-09-02 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-24133: - Summary: PartitionRequestClientFactoryTest.testThrowsWhenNetworkFailure fails on Azure Key: FLINK-24133 URL: https://issues.apache.org/jira/browse/FLINK-24133

[jira] [Created] (FLINK-24129) TopicRangeTest.rangeCreationHaveALimitedScope fails on Azure

2021-09-02 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-24129: - Summary: TopicRangeTest.rangeCreationHaveALimitedScope fails on Azure Key: FLINK-24129 URL: https://issues.apache.org/jira/browse/FLINK-24129 Project: Flink

Re: [DISCUSS] Automated architectural tests

2021-09-02 Thread Till Rohrmann
If it is possible to automate these kinds of checks, then I am all for it because everything else breaks eventually. So +1 for this proposal. I don't have experience with what tools are available, though. I would like to add a rule that every test class extends directly or indirectly TestLogger

Re: Do we still maintain travis build system?

2021-09-01 Thread Till Rohrmann
; On 30/08/2021 10:42, Till Rohrmann wrote: > > I think Flink 1.10.x used Travis. So I agree with Tison's proposal. +1 > for > > removing the "@flinkbot run travis" from the command documentation. > > > > cc @Chesnay Schepler > > > > Cheer

Re: [ANNOUNCE] Apache Flink Stateful Functions 3.1.0 released

2021-09-01 Thread Till Rohrmann
Great news! Thanks a lot for all your work on the new release :-) Cheers, Till On Wed, Sep 1, 2021 at 9:07 AM Johannes Moser wrote: > Congratulations, great job.  > > On 31.08.2021, at 17:09, Igal Shilman wrote: > > The Apache Flink community is very happy to announce the release of Apache >

Re: Do we still maintain travis build system?

2021-08-30 Thread Till Rohrmann
I think Flink 1.10.x used Travis. So I agree with Tison's proposal. +1 for removing the "@flinkbot run travis" from the command documentation. cc @Chesnay Schepler Cheers, Till On Sun, Aug 29, 2021 at 4:48 AM tison wrote: > Hi, > > I can still see "@flinkbot run travis" in flinkbot's toast

[jira] [Created] (FLINK-24038) DispatcherResourceManagerComponent fails to deregister application if no leading ResourceManager

2021-08-28 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-24038: - Summary: DispatcherResourceManagerComponent fails to deregister application if no leading ResourceManager Key: FLINK-24038 URL: https://issues.apache.org/jira/browse/FLINK

[jira] [Created] (FLINK-24015) Dispatcher does not log JobMaster initialization error on info level

2021-08-27 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-24015: - Summary: Dispatcher does not log JobMaster initialization error on info level Key: FLINK-24015 URL: https://issues.apache.org/jira/browse/FLINK-24015 Project

Re: 【Announce】Zeppelin 0.10.0 is released, Flink on Zeppelin Improved

2021-08-26 Thread Till Rohrmann
Cool, thanks for letting us know Jeff. Hopefully, many users use Zeppelin together with Flink. Cheers, Till On Thu, Aug 26, 2021 at 4:47 AM Leonard Xu wrote: > Thanks Jeff for the great work ! > > Best, > Leonard > > 在 2021年8月25日,22:48,Jeff Zhang 写道: > > Hi Flink users, > > We (Zeppelin

[jira] [Created] (FLINK-23965) E2E do not execute locally on MacOS

2021-08-25 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23965: - Summary: E2E do not execute locally on MacOS Key: FLINK-23965 URL: https://issues.apache.org/jira/browse/FLINK-23965 Project: Flink Issue Type: Bug

Re: Flink 1.14 Weekly 2021-08-24

2021-08-25 Thread Till Rohrmann
Thanks for the update Joe! Cheers, Till On Tue, Aug 24, 2021 at 3:30 PM Johannes Moser wrote: > Dear Apache Flink community, > > Today we moved from the two week sync rhythm to a one week one as we > entered the stabilisation phase last week. > > *Status check:* > All features got the state of

[jira] [Created] (FLINK-23947) DefaultDispatcherRunner does not log when the Dispatcher gains and loses leadership

2021-08-24 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23947: - Summary: DefaultDispatcherRunner does not log when the Dispatcher gains and loses leadership Key: FLINK-23947 URL: https://issues.apache.org/jira/browse/FLINK-23947

[jira] [Created] (FLINK-23946) Application mode fails fatally when being shut down

2021-08-24 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23946: - Summary: Application mode fails fatally when being shut down Key: FLINK-23946 URL: https://issues.apache.org/jira/browse/FLINK-23946 Project: Flink Issue

Re: Scala 2.12 End to End Tests

2021-08-24 Thread Till Rohrmann
default > > > > On Sat, 21 Aug 2021 at 13:43, Chesnay Schepler > wrote: > > > > > I'm with Till that we should switch to 2.12 by default . > > > > > > On 21/08/2021 11:12, Till Rohrmann wrote: > > > > Hi Danny, > > > &g

[jira] [Created] (FLINK-23932) KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue hangs on AzureCase

2021-08-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23932: - Summary: KafkaTableITKafkaTableITCase.testKafkaSourceSinkWithKeyAndPartialValue hangs on AzureCase Key: FLINK-23932 URL: https://issues.apache.org/jira/browse/FLINK-23932

Re: Projection pushdown for metadata columns

2021-08-23 Thread Till Rohrmann
Does this also affect Flink 1.14.0? If yes, do we want to fix this issue for the upcoming release? If yes, then please make this issue a blocker or at least critical. Cheers, Till On Mon, Aug 23, 2021 at 8:39 AM Ingo Bürk wrote: > Thanks Timo for the confirmation. I've also raised

[jira] [Created] (FLINK-23916) Add documentation for how to configure the CheckpointFailureManager

2021-08-23 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23916: - Summary: Add documentation for how to configure the CheckpointFailureManager Key: FLINK-23916 URL: https://issues.apache.org/jira/browse/FLINK-23916 Project: Flink

[jira] [Created] (FLINK-23906) Increase akka.ask.timeout for tests using the MiniCluster

2021-08-21 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23906: - Summary: Increase akka.ask.timeout for tests using the MiniCluster Key: FLINK-23906 URL: https://issues.apache.org/jira/browse/FLINK-23906 Project: Flink

Re: Scala 2.12 End to End Tests

2021-08-21 Thread Till Rohrmann
Hi Danny, I think in the nightly builds we do run the e2e with Scala 2.12 [1]. The way it is configured is via the PROFILE env variable if I am not mistaken. Independent of this we might wanna start a discussion whether we don't want to switch to Scala 2.12. per default. [1]

Re: [DISCUSS] FLIP-171: Async Sink

2021-08-20 Thread Till Rohrmann
> On 16.07.21, 18:28, "Till Rohrmann" wrote: > > CAUTION: This email originated from outside of the organization. Do > not click links or open attachments unless you can confirm the sender and > know the content is safe. > > > > Sure, thanks for the p

Re: [DISCUSS] Backport HybridSource to 1.13.x

2021-08-20 Thread Till Rohrmann
Hi everyone, if the HybridSource is self contained, then I don't see a lot of harm in doing the backport. We should probably explicitly state in the docs that a certain minor version is required for this source to be available because our docs don't distinguish between minor versions and having

Re: [DISCUSS] Merge FLINK-23757 after feature freeze

2021-08-18 Thread Till Rohrmann
@Dian Fu could you assess how involved this change is? If the change is not very involved and the risk is limited, then I'd be in favour of merging it because feature parity of APIs is quite important for our users. Cheers, Till On Wed, Aug 18, 2021 at 1:46 PM Ingo Bürk wrote: > Hello dev, >

Re: Configuring flink-python in IntelliJ

2021-08-18 Thread Till Rohrmann
Thanks for starting this discussion Ingo. I guess that Dian can probably answer this question. I am big +1 for updating our documentation for how to set up the IDE since this problem will probably be encountered a couple of times. Cheers, Till On Wed, Aug 18, 2021 at 11:03 AM Ingo Bürk wrote:

Re: [DISCUSS] Merging FLINK-21867 after feature freeze

2021-08-17 Thread Till Rohrmann
One addition: The change only affects the job-exceptions component of Flink's web ui and is, thus, quite isolated. Cheers, Till On Tue, Aug 17, 2021 at 4:42 PM David Morávek wrote: > Hi, > > We have a small UI change [1][2] that we'd like to get merged into 1.14 > [3]. The change allows

[jira] [Created] (FLINK-23785) SinkITCase.testMetrics fails with ConcurrentModification on Azure

2021-08-15 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23785: - Summary: SinkITCase.testMetrics fails with ConcurrentModification on Azure Key: FLINK-23785 URL: https://issues.apache.org/jira/browse/FLINK-23785 Project: Flink

[jira] [Created] (FLINK-23778) UpsertKafkaTableITCase.testKafkaSourceSinkWithKeyAndFullValue hangs on Azure

2021-08-15 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23778: - Summary: UpsertKafkaTableITCase.testKafkaSourceSinkWithKeyAndFullValue hangs on Azure Key: FLINK-23778 URL: https://issues.apache.org/jira/browse/FLINK-23778

Re: [ANNOUNCE] Apache Flink 1.11.4 released

2021-08-10 Thread Till Rohrmann
This is great news. Thanks a lot for being our release manager Godfrey and also to everyone who made this release possible. Cheers, Till On Tue, Aug 10, 2021 at 11:09 AM godfrey he wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.11.4, which is the

Re: [ANNOUNCE] Apache Flink 1.12.5 released

2021-08-10 Thread Till Rohrmann
This is great news. Thanks a lot for being our release manager Jingsong and also to everyone who made this release possible. Cheers, Till On Tue, Aug 10, 2021 at 10:57 AM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.5, which is

Re: Incompatible RAW types in Table API

2021-08-09 Thread Till Rohrmann
Hi Dominik, I am pulling in Timo who might know more about this. Cheers, Till On Mon, Aug 9, 2021 at 3:21 PM Dominik Wosiński wrote: > Hey all, > > I think I've hit some weird issue in Flink TypeInformation generation. I > have the following code: > > val stream: DataStream[Event] = ... >

Re: [ANNOUNCE] Apache Flink 1.13.2 released

2021-08-09 Thread Till Rohrmann
Thanks Yun Tang for being our release manager and the great work! Also thanks a lot to everyone who contributed to this release. Cheers, Till On Mon, Aug 9, 2021 at 9:48 AM Yu Li wrote: > Thanks Yun Tang for being our release manager and everyone else who made > the release possible! > > Best

Re: [VOTE] FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-09 Thread Till Rohrmann
+1 (binding) Cheers, Till On Thu, Aug 5, 2021 at 9:09 PM Arvid Heise wrote: > Dear devs, > > I'd like to open a vote on FLIP-180: Adjust StreamStatus and Idleness > definition [1] which was discussed in this thread [2]. > The vote will be open for at least 72 hours unless there is an objection

Re: Looking for Maintainers for Flink on YARN

2021-08-05 Thread Till Rohrmann
Great to hear from everybody. I've created an invite for Thu, Aug 11, 9am CEST and invited Gabor and Yangze. Let me know if somebody else wants to join as well. I'll add her then to the calendar invite. Cheers, Till On Thu, Aug 5, 2021 at 3:50 AM Yangze Guo wrote: > Thanks Konstantin for

Re: [FLINK-23555] needs your attention

2021-08-05 Thread Till Rohrmann
Hi, At the moment many community members are busy with finishing their work for the upcoming feature freeze. This can cause slow responsiveness on the JIRA tickets. Once the feature freeze is past, people will surely get back to you. Cheers, Till On Mon, Aug 2, 2021 at 8:43 AM 卫博文 wrote: >

Re: About [FLINK-22405] Support fixed-lengh chars in the LeadLag built-in function

2021-08-05 Thread Till Rohrmann
Hi Liwei, Thanks for helping the community. At the moment most of the committers are busy with finishing their work for the upcoming feature freeze. This might cause some delay in responses. We'll get back to your PR as soon as someone has capacity. cc Jark and Timo. Cheers, Till On Sun, Aug

Re: [DISCUSS] FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-05 Thread Till Rohrmann
> 3. To be sure, is it correct that in case of a dynamic assignment and > there > > is temporarily no data, that scenario 2 is applicable? > > > > 4. Does WatermarkStrategy#withIdleness currently cover scenario 2, 3 and > > the one from my 3rd question? (edited) >

[jira] [Created] (FLINK-23637) MeasurementInfoProviderTest.testNormalizingTags fails on Azure

2021-08-05 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23637: - Summary: MeasurementInfoProviderTest.testNormalizingTags fails on Azure Key: FLINK-23637 URL: https://issues.apache.org/jira/browse/FLINK-23637 Project: Flink

Re: [DISCUSS] FLIP-180: Adjust StreamStatus and Idleness definition

2021-07-23 Thread Till Rohrmann
eir own idleness > definition. Let's see what other devs think. I'm pinging the once that have > been most involved in the discussion: @Stephan Ewen > @Till > Rohrmann @Dawid Wysakowicz > . > > The flip mentions a 'watermarkstatus' package for the WatermarkStatus > &

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-23 Thread Till Rohrmann
ble to implement their > > target logic by explicitly add a flag regarding finished, and perhaps > have > > different logic if this part of states are referred to. Besides, this > case would > > only happen on rescaling or topology change, which embedded some kind >

Re: [VOTE] FLIP-183: Dynamic buffer size adjustment

2021-07-22 Thread Till Rohrmann
+1 (binding) @Anton it is usually a good practice to start a new mailing list thread for the vote. It should refer to the discussion thread and have a subject line of the form "[VOTE] FLIP-183: Dynamic buffer size adjustment". Next time, let's do it like this. Cheers, Till On Wed, Jul 21, 2021

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-22 Thread Till Rohrmann
ould explicitly requires the task to wait for a > checkpoint triggered after finish() > method is called for all the operators ? We could be able to achieve this > target by maintaining > some state inside the task. > > Checkpointing from a single subtask / UnionListSta

Re: [DISCUSS] FLIP-185: Shorter heartbeat timeout and interval default values

2021-07-22 Thread Till Rohrmann
erstand that normally, as user code is not a JVM-blocking activity such > as a GC, it should have no impact on heartbeats, but from experience, it > really does) > > > > Cheers, > > Arnaud > > > > > > *De :* Gen Luo > *Envoyé :* jeudi 22 juillet 2021 05:46 > *

Re: Flink 1.14. Bi-weekly 2021-07-20

2021-07-21 Thread Till Rohrmann
Thanks for the update, Joe, Xintong and Dawid. This is very helpful. Cheers, Till On Wed, Jul 21, 2021 at 9:27 AM Johannes Moser wrote: > Dear Flink Community, > > here's the latest update from the bi-weekly. > Short summary: still a lot of stuff is going on, the feature freeze has > been >

Re: [DISCUSS] Releasing Flink 1.11.4

2021-07-21 Thread Till Rohrmann
Great, thanks Godfrey. Cheers, Till On Wed, Jul 21, 2021 at 7:42 AM godfrey he wrote: > Hi Till, > > Sorry for the late reply. The previous period, I focused on another urgent > work, > and suspended the releasing work. I've recently restarted it. > > Best, > Godfrey &g

Re: [DISCUSS] FLIP-185: Shorter heartbeat timeout and interval default values

2021-07-21 Thread Till Rohrmann
about long pauses of unresponsiveness of Flink. [1] https://issues.apache.org/jira/browse/FLINK-23216 [2] https://issues.apache.org/jira/browse/FLINK-22483 Cheers, Till On Wed, Jul 21, 2021 at 4:58 AM Yang Wang wrote: > Thanks @Till Rohrmann for starting this discussion > > First

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-19 Thread Till Rohrmann
> >> > >> >> > >> >> > >> >> > -- >> >> > From:Piotr Nowojski >> >> > Send Time:2021 Jul. 16 (Fri.) 13:48 >> >> > To:dev >> >&

Re: Introduction email

2021-07-19 Thread Till Rohrmann
Hi Srini, Welcome to the Flink community :-) Great to hear what you are planning to do with Flink at LinkedIn. I think sharing this is very motivational for the community and also gives context for what you are focusing on. Looking forward to working with you and improving Flink. Cheers, Till

[DISCUSS] FLIP-185: Shorter heartbeat timeout and interval default values

2021-07-16 Thread Till Rohrmann
Hi everyone, Since Flink 1.5 we have the same heartbeat timeout and interval default values that are defined as heartbeat.timeout: 50s and heartbeat.interval: 10s. These values were mainly chosen to compensate for lengthy GC pauses and blocking operations that were executed in the main threads of

Re: [DISCUSS] FLIP-171: Async Sink

2021-07-16 Thread Till Rohrmann
inesis-171/src/main/java/software/amazon/flink/connectors/AmazonKinesisDataStreamSink.java > > From: Till Rohrmann > Date: Friday, 16. July 2021 at 18:10 > To: Piotr Nowojski > Cc: Steffen Hausmann , "dev@flink.apache.org" < > dev@flink.apache.org>, Arvid Heise > Su

Re: [DISCUSS] FLIP-171: Async Sink

2021-07-16 Thread Till Rohrmann
t;> I’m happy to take your guidance on this. I need to think through your >> proposals and I’ll follow-up on Monday with some more context so that we >> can close the discussion on these details. But for now, I’ll close the vote. >> >> Thanks, Steffen >> >&

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-16 Thread Till Rohrmann
ndependent but parallel mechanisms. With event time, > they are basically the same. > 2) it probably should for the sake of processing time windows. > > Here you are touching the bit of the current design that I like the least. > We basically have now three different ways of conveying very

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-16 Thread Till Rohrmann
r to flush it's internal > state > c) finish(), used for example by ContinuousFileReaderOperator > > It's a bit messy and I'm not sure if this should be strengthened out? Each > one of those has a little bit different semantic/meaning, but at the same > time they are very s

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-16 Thread Till Rohrmann
t operators `endInput()` and > `finish()` are actually the very same thing. > > Piotrek > > czw., 15 lip 2021 o 16:47 Till Rohrmann napisał(a): > > > Thanks for updating the FLIP. Based on the new section about > > stop-with-savepoint [--drain] I got two other question

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-16 Thread Till Rohrmann
ry segment size. In this way we > can > > leave any already filled in buffers on the sender side untouched and we > do > > not need to partition/slice them before sending them down, making at > least > > the initial version even simpler. This way we also do not need to > > di

[jira] [Created] (FLINK-23403) Decrease default values for heartbeat timeout and interval

2021-07-15 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23403: - Summary: Decrease default values for heartbeat timeout and interval Key: FLINK-23403 URL: https://issues.apache.org/jira/browse/FLINK-23403 Project: Flink

Re: [DISCUSS] FLIP-183: Dynamic buffer size adjustment

2021-07-15 Thread Till Rohrmann
Hi everyone, Thanks a lot for creating this FLIP Anton and Piotr. I think it looks like a very promising solution for speeding up our checkpoints and being able to create them more reliably. Following up on Steven's question: I assume that buffer sizes are only changed for newly assigned

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-15 Thread Till Rohrmann
upon calling finish, for example? Cheers, Till On Thu, Jul 15, 2021 at 10:15 AM Till Rohrmann wrote: > Thanks a lot for your answers and clarifications Yun. > > 1+2) Agreed, this can be a future improvement if this becomes a problem. > > 3) Great, this will help a lot with underst

Re: [VOTE] Release 1.12.5, release candidate #1

2021-07-15 Thread Till Rohrmann
Great, thanks a lot Jingsong! Cheers, Till On Thu, Jul 15, 2021 at 5:00 AM Jingsong Li wrote: > Hi Till and Flinkers working for the FLINK-23233, > > Thanks for the effort! > > RC2 is on the way. > > Best, > Jingsong > > On Tue, Jul 13, 2021 at 8:35 PM Till Rohrma

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-15 Thread Till Rohrmann
Thanks a lot for your answers and clarifications Yun. 1+2) Agreed, this can be a future improvement if this becomes a problem. 3) Great, this will help a lot with understanding the FLIP. Cheers, Till On Wed, Jul 14, 2021 at 5:41 PM Yun Gao wrote: > Hi Till, > > Very thanks for the review and

Re: [RESULT][VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-07-14 Thread Till Rohrmann
Hi everyone, I am a bit late to the voting party but let me ask three questions: 1) Why do we execute the trigger plan computation in the main thread if we cannot guarantee that all tasks are still running when triggering the checkpoint? Couldn't we do the computation in a different thread in

Re: [DISCUSS] Address deprecation warnings when upgrading dependencies

2021-07-14 Thread Till Rohrmann
I think this suggestion makes a lot of sense, Stephan. +1 for fixing deprecation warnings when bumping/changing dependencies. I actually found myself recently, whenever touching a test class, replacing Junit's assertThat with Hamcrest's version which felt quite tedious. Cheers, Till On Tue, Jul

Re: [DISCUSS] Releasing Flink 1.11.4

2021-07-13 Thread Till Rohrmann
Hi Godfrey, Are you continuing with the 1.11.4 release process? Cheers, Till On Tue, Jul 6, 2021 at 1:15 PM Chesnay Schepler wrote: > Since 1.11.4 is about releasing the commits we already have merged > between 1.11.3 and 1.13.0, I would suggest to not add additional fixes. > > On 06/07/2021

Re: [VOTE] Release 1.13.2, release candidate #1

2021-07-13 Thread Till Rohrmann
could cancel > this RC and wait for that problem resolved. > > Best > Yun Tang > ____ > From: Till Rohrmann > Sent: Wednesday, July 7, 2021 17:21 > To: dev > Cc: Xintong Song > Subject: Re: [VOTE] Release 1.13.2, release candidate #1 > &

Re: [VOTE] Release 1.12.5, release candidate #1

2021-07-13 Thread Till Rohrmann
is RC. That should also give us the chance to fix > FLINK-23233. > > Best, > Jingsong > > On Wed, Jul 7, 2021 at 5:29 PM Till Rohrmann wrote: > > > Hi folks, are we sure that FLINK-23233 [1] does not affect the 1.12 > release > > branch. I think this problem

Re: [VOTE] FLIP-181: Custom netty HTTP request inbound/outbound handlers

2021-07-12 Thread Till Rohrmann
Thanks for starting the vote Marton. I have two comments: * I would suggest that the interfaces return Optional or at least have a @Nullable annotation in order to make the contract explicit. * The test plan should contain tests for the general infrastructure which should live in Flink. We

Re: Job Recovery Time on TM Lost

2021-07-09 Thread Till Rohrmann
gt;>> flink version, we optimize it by task's report and jobmaster's probe. When >>> a task fails because of the connection, it reports to the jobmaster. The >>> jobmaster will try to confirm the liveness of the unconnected >>> taskmanager for certain times by confi

Re: [ANNOUNCE] New Apache Flink Committer - Yuan Mei

2021-07-07 Thread Till Rohrmann
Congratulations Yuan. Cheers, Till On Wed, Jul 7, 2021 at 7:22 PM Yu Li wrote: > Hi all, > > On behalf of the PMC, I’m very happy to announce Yuan Mei as a new Flink > committer. > > Yuan has been an active contributor for more than two years, with code > contributions on multiple components

Re: [DISCUSS] Lifecycle of ShuffleMaster and its Relationship with JobMaster and PartitionTracker

2021-07-07 Thread Till Rohrmann
One quick comment: When developing the ShuffleService abstraction we also thought that different jobs might want to use different ShuffleServices depending on their workload (e.g. batch vs. streaming workload). So ideally, the chosen solution here can also support this use case eventually.

Re: [ANNOUNCE] New PMC member: Guowei Ma

2021-07-07 Thread Till Rohrmann
Congratulations, Guowei! Cheers, Till On Wed, Jul 7, 2021 at 9:41 AM Roman Khachatryan wrote: > Congratulations! > > Regards, > Roman > > On Wed, Jul 7, 2021 at 8:24 AM Rui Li wrote: > > > > Congratulations Guowei! > > > > On Wed, Jul 7, 2021 at 1:01 PM Benchao Li wrote: > > > > >

Re: [ANNOUNCE] New Apache Flink Committer - Yang Wang

2021-07-07 Thread Till Rohrmann
Congratulations, Yang! Cheers, Till On Wed, Jul 7, 2021 at 9:41 AM Roman Khachatryan wrote: > Congrats! > > Regards, > Roman > > > On Wed, Jul 7, 2021 at 8:28 AM Qingsheng Ren wrote: > > > > Congratulations Yang! > > > > -- > > Best Regards, > > > > Qingsheng Ren > > Email: renqs...@gmail.com

Re: [VOTE] Release 1.12.5, release candidate #1

2021-07-07 Thread Till Rohrmann
Hi folks, are we sure that FLINK-23233 [1] does not affect the 1.12 release branch. I think this problem was introduced with FLINK-21996 [2] which is part of release-1.12. Hence, we might either fix this problem and cancel this RC or we need to create a fast 1.12.6 afterwards. [1]

Re: [VOTE] Release 1.13.2, release candidate #1

2021-07-07 Thread Till Rohrmann
I think FLINK-23233 sounds like a very serious problem to me. We either continue releasing 1.13.2 and do an immediate 1.13.3 afterwards or we cancel this RC and fix the problem and then resume the 1.13.2 release. Cheers, Till On Wed, Jul 7, 2021 at 5:49 AM JING ZHANG wrote: > Thanks Xintong

[jira] [Created] (FLINK-23297) Upgrade Protobuf to 3.17.3

2021-07-07 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23297: - Summary: Upgrade Protobuf to 3.17.3 Key: FLINK-23297 URL: https://issues.apache.org/jira/browse/FLINK-23297 Project: Flink Issue Type: Sub-task

Re: Job Recovery Time on TM Lost

2021-07-06 Thread Till Rohrmann
ould be good > enough. But since I'm not experienced with an unstable environment, I can't > tell whether that's also enough for it. > > On Mon, Jul 5, 2021 at 5:59 PM Till Rohrmann wrote: > >> I think for RPC communication there are retry strategies used by the >> underly

Re: Job Recovery Time on TM Lost

2021-07-05 Thread Till Rohrmann
> fault tolerance of heartbeat, unless we also introduce some retry > strategies for netty connections. > > > On Fri, Jul 2, 2021 at 9:34 PM Till Rohrmann wrote: > >> Could you share the full logs with us for the second experiment, Lu? I >> cannot tell from th

Re: Job Recovery Time on TM Lost

2021-07-02 Thread Till Rohrmann
;>> Thanks TIll and Yang for help! Also Thanks Till for a quick fix! >>>> >>>> I did another test yesterday. In this test, I intentionally throw >>>> exception from the source operator: >>>> ``` >>>> if (runtimeContext.getIndexOfTh

Re: Job Recovery Time on TM Lost

2021-07-01 Thread Till Rohrmann
nt. > As a result, Flink will not deploy tasks onto the dead TaskManagers. > > > I think most of the time cost in Phase 1 might be cancelling the tasks on > the dead TaskManagers. > > > Best, > Yang > > > Till Rohrmann 于2021年7月1日周四 下午4:49写道: > >&g

[jira] [Created] (FLINK-23209) Timeout heartbeat if the heartbeat target is no longer reachable

2021-07-01 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23209: - Summary: Timeout heartbeat if the heartbeat target is no longer reachable Key: FLINK-23209 URL: https://issues.apache.org/jira/browse/FLINK-23209 Project: Flink

Re: Job Recovery Time on TM Lost

2021-07-01 Thread Till Rohrmann
The analysis of Gen is correct. Flink currently uses its heartbeat as the primary means to detect dead TaskManagers. This means that Flink will take at least `heartbeat.timeout` time before the system recovers. Even if the cancellation happens fast (e.g. by having configured a low

[jira] [Created] (FLINK-23202) RpcService should fail result futures if messages could not be sent

2021-07-01 Thread Till Rohrmann (Jira)
Till Rohrmann created FLINK-23202: - Summary: RpcService should fail result futures if messages could not be sent Key: FLINK-23202 URL: https://issues.apache.org/jira/browse/FLINK-23202 Project: Flink

Re: [DISCUSS] Feedback Collection Jira Bot

2021-06-30 Thread Till Rohrmann
t; > > > stale-unassigned > > > > > > > > > role. > > > > > > > > > > > > > > > > > > * We change the time intervals as follows, accepting > reality > > a > > > > bit > > > > > > more &

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-30 Thread Till Rohrmann
> > > > liujiangang > > > > > > > > Jingsong Li 于2021年6月29日周二 上午10:31写道: > > > > > > > > > +1 Thanks Xintong for the update! > > > > > > > > > > Best, > > > > > Jingsong > > >

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-28 Thread Till Rohrmann
> https://docs.google.com/document/d/1uUbxbgbGErBXtmEjhwVhBWG3i6nhQ0LXs96OlntEYnU/edit?usp=sharing > > > > > > [2] > > > > https://cwiki.apache.org/confluence/display/FLINK/Merging+Pull+Requests > > > > > > > > > > > > On Fri, Jun

Re: Flink 1.14. Bi-weekly 2021-06-22

2021-06-28 Thread Till Rohrmann
Thanks a lot for the update, Joe. This is very helpful! Cheers, Till On Mon, Jun 28, 2021 at 10:10 AM Xintong Song wrote: > Thanks for the update, Joe. > > Thank you~ > > Xintong Song > > > On Mon, Jun 28, 2021 at 3:54 PM Johannes Moser > wrote: > > > Hello, > > > > Last Tuesday was our

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-25 Thread Till Rohrmann
good to > > me. > > > > > > Minor: I'm still a bit worried about the commits merged before we fix > > > the unstable tests can also break those tests. Instead of letting the > > > assigners keep a look at all potentially related commits, they can > > > m

Re: [DISCUSS] Releasing Flink 1.12.5

2021-06-24 Thread Till Rohrmann
Thanks for volunteering as our release manager, Jingsong. From my perspective, we are good to go for the 1.12.5 release. Cheers, Till On Thu, Jun 24, 2021 at 11:53 AM Jingsong Li wrote: > Dear devs, > > As discussed in [1], I would volunteer as the release manager for 1.12.5 > and kick off the

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-24 Thread Till Rohrmann
nd found there are some > > > > detailed problems that need to be discussed in a step further: > > > > 1. Report the test failures in the JIRA. > > > > 2. Set a deadline to find out the root cause and solve the failure > for > > > the > > > > ne

Re: Change in accumutors semantics with jobClient

2021-06-23 Thread Till Rohrmann
ithub.com/apache/flink/pull/16256 > > Cheers, > > Etienne > > On 21/06/2021 11:39, Till Rohrmann wrote: > > Thanks for bringing this to the dev ML Etienne. Could you maybe update > the > > release notes for Flink 1.13 [1] to include this change? That way it &

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-23 Thread Till Rohrmann
Guo wrote: > +1 for appending this to community guidelines for merging PRs. > > @Till Rohrmann > I agree that with this approach unstable tests will not block other > commit merges. However, it might be hard to prevent merging commits > that are related to those tests and shoul

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-22 Thread Till Rohrmann
at 12:00 PM Arvid Heise wrote: > I think this is overall a good idea. So +1 from my side. > However, I'd like to put a higher priority on infrastructure then, in > particular docker image/artifact caches. > > On Tue, Jun 22, 2021 at 11:50 AM Till Rohrmann > wrote: > &g

Re: [DISCUSS] Dashboard/HistoryServer authentication

2021-06-22 Thread Till Rohrmann
ious question, please correct me if there > > is > > > > >> reason to believe it is gaining adoption) > > > > >> > > > > >> The point about Kerberos being independent of infrastructure is a > > good > > > > one > > &g

Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2

2021-06-22 Thread Till Rohrmann
Thanks a lot, Yun, Godfrey, and Jingsong for being our release managers! Creating these bug-fix releases will be super helpful for our users. Cheers, Till On Tue, Jun 22, 2021 at 9:59 AM Piotr Nowojski wrote: > Thanks for volunteering. > > A quick update about FLINK-23011. It turned out to be

Re: [DISCUSS] Do not merge PRs with "unrelated" test failures.

2021-06-22 Thread Till Rohrmann
Thanks for bringing this topic to our attention Xintong. I think your proposal makes a lot of sense and we should follow it. It will give us confidence that our changes are working and it might be a good incentive to quickly fix build instabilities. Hence, +1. Cheers, Till On Tue, Jun 22, 2021

Re: [DISCUSS] Drop Mesos in 1.14

2021-06-22 Thread Till Rohrmann
I'd be ok with dropping support for Mesos if it helps us to clear our dependencies in the flink-runtime module. If we do it, then we should probably update our documentation with a pointer to the latest Flink version that supports Mesos in case of users strictly need Mesos. Cheers, Till On Tue,

Re: [DISCUSS] FLIP-171: Async Sink

2021-06-22 Thread Till Rohrmann
Adding the InterruptException to the write method would make it explicit that the write call can block but must react to interruptions (e.g. when Flink wants to cancel the operation). I think this makes the contract a bit clearer. I think starting simple and then extending the API as we see the

Re: Re: [DISCUSS] FLIP-169: DataStream API for Fine-Grained Resource Requirements

2021-06-21 Thread Till Rohrmann
I would be more in favor of what Zhu Zhu proposed to throw an exception with a meaningful and understandable explanation that also includes how to resolve this problem. I do understand the reasoning behind automatically switching the edge types in order to make things easier to use but a) this can

Re: [DISCUSS] Releasing Flink 1.11.4, 1.12.5, 1.13.2

2021-06-21 Thread Till Rohrmann
Thanks for starting this discussion Dawid. We have collected a couple of fixes for the different releases: #fixes: 1.11.4: 72 1.12.5: 35 1.13.2: 49 which in my opinion warrants new bugfix releases. Note that we intend to do another 1.11 release because of the seriousness of the FLINK-22815 [1]

Re: Change in accumutors semantics with jobClient

2021-06-21 Thread Till Rohrmann
Thanks for bringing this to the dev ML Etienne. Could you maybe update the release notes for Flink 1.13 [1] to include this change? That way it might be a bit more prominent. I think the change needs to go into the release-1.13 and master branch. [1]

Re: [DISCUSS] Dashboard/HistoryServer authentication

2021-06-17 Thread Till Rohrmann
I left some comments in the Google document. It would be great if someone from the community with security experience could also take a look at it. Maybe Eron you have an opinion on the topic. Cheers, Till On Thu, Jun 17, 2021 at 6:57 PM Till Rohrmann wrote: > Hi Gabor, > > I have

<    1   2   3   4   5   6   7   8   9   10   >