Re: [VOTE] Deprecating Mesos support
+1 On Wed, Apr 14, 2021 at 9:46 AM Matthias Pohl wrote: > Hi everyone, > After the discussion of deprecating the Mesos support went on for a while > now in [1] and considering that we already plan to retire it mid-term [2], > I want to start a vote on deprecating the Mesos support in the Flink > documentation as a next step. Ideally, we could even add this change to > 1.13. > > Please vote +1 to approve, or -1 with a comment. The vote will be open at > least until Friday, Mar 16, 2021. > > Best, > Matthias > > [1] > > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Remove-Mesos-support-td45974.html > [2] https://flink.apache.org/roadmap.html#feature-radar > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Feedback Collection Jira Bot
Hi everyone, After some offline conversations, I think, it makes sense to already open this thread now in order to collect feedback and suggestions around the Jira Bot. The following two changes I will do right away: * increase "stale-assigned.stale-days" to 14 days (Marta, Stephan, Nico have provided feedback that this is too aggressive). * exclude Sub-Tasks from all rules except the "stale-assigned" rule (I think, this was just an oversight in the original discussion.) Keep it coming. Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Drop Scala Shell
Thanks for bringing this up. +1 for dropping the Scala Shell or moving it out of Apache Flink if there is interest in the community to develop it further. On Thu, Sep 16, 2021 at 5:26 PM Seth Wiesman wrote: > +1 > > The scala shell requires time and expertise that the contributors cannot > provide. It will be great to let the broader community have the opportunity > to foster and mature the scala shell outside the constraints imposed by the > ASF. > > On Thu, Sep 16, 2021 at 9:02 AM Jeff Zhang wrote: > > > Hi Martijn, > > > > It is a pity that the flink community doesn't have resources to maintain > > scala shell which is useful for interactive user experience. We > > （Zeppelin）didn't use flink scala shell module directly, instead we > > customized the flink scala shell. So we are still interested in scala > shell > > module, for us if no one want to maintain scala shell, it would be better > > to have it as flink 3rd party library, maybe in flink-packages.org > > > > > > Martijn Visser 于2021年9月16日周四 下午7:21写道： > > > > > Hi everyone, > > > > > > As was discussed quite a while ago, the community has planned to drop > > > support for Scala 2.11 [1]. I'm proposing to also drop the Scala Shell, > > > which was briefly discussed in the email thread for dropping Scala 2.11 > > > [2]. > > > > > > The Scala Shell doesn't work on Scala 2.12 [3] and there hasn't been > much > > > traction to get this working and it looks like the Flink community / > > > committers don't want to maintain it anymore. Alternatively, if there's > > > still someone interested in the Scala Shell, it could also be moved to > a > > > 3rd party repository. Removing the Scala Shell would/could help in > > getting > > > Scala 2.11 dropped quicker. > > > > > > Let me know your thoughts. > > > > > > Best regards, > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-20845 > > > [2] > > > > > > > > > https://lists.apache.org/thread.html/ra817c5b54e3de48d80e5b4e0ae67941d387ee25cf9779f5ae37d0486%40%3Cdev.flink.apache.org%3E > > > [3] https://issues.apache.org/jira/browse/FLINK-10911 > > > > > > > > > Martijn Visser | Product Manager > > > > > > mart...@ververica.com > > > > > > <https://www.ververica.com/> > > > > > > > > > Follow us @VervericaData > > > > > > -- > > > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > > Conference > > > > > > Stream Processing | Event Driven | Real Time > > > > > > > > > -- > > Best Regards > > > > Jeff Zhang > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Looking for Maintainers for Flink on YARN
Dear community, We are looking for community members, who would like to maintain Flink's YARN support going forward. So far, this has been handled by teams at Ververica & Alibaba. The focus of these teams has shifted over the past months so that we only have little time left for this topic. Still, we think, it is important to maintain high quality support for Flink on YARN. What does "Maintaining Flink on YARN" mean? There are no known bigger efforts outstanding. We are mainly talking about addressing "test-stability" issues, bugs, version upgrades, community contributions & smaller feature requests. The prioritization of these would be up to the future maintainers, except "test-stability" issues which are important to address for overall productivity. If a group of community members forms itself, we are happy to give an introduction to relevant pieces of the code base, principles, assumptions, ... and hand over open threads. If you would like to take on this responsibility or can join this effort in a supporting role, please reach out! Cheers, Konstantin for the Deployment & Coordination Team at Ververica -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Change Default Jira Priority from "Major" to "Minor"
Thanks, Martijn.
I was not aware of the definition in Jira. I don't think we can actually
change those, because they are shared by many projects. It's not ideal
though. For the same reason, we can probably not *rename* Minor to Normal.
It would also affect a lot of other projects. What we would have to do is:
add "Normal" to FLINK, remove "Minor" from FLINK and migrate all "Minor"
tickets to "Normal". That would be possible, I think.
What do others think?
On Thu, Jul 29, 2021 at 7:21 AM Martijn Visser
wrote:
> Hi,
>
> Thanks for the thorough explanation. I was basing my information on the
> info you see you click on the question mark behind 'Priority' when you
> create a ticket [1].
> Given the documented ticket priorities, I would vote for renaming Minor to
> Normal and set that as the default priority.
>
> Thanks,
>
> Martijn
>
> [1]
>
> https://issues.apache.org/jira/secure/ShowConstantsHelp.jspa?decorator=popup#PriorityLevels
>
> On Wed, 28 Jul 2021 at 16:06, Konstantin Knauf wrote:
>
> > Hi everyone,
> >
> > well that escalated quickly :) Let me share some more background. This is
> > our current definition for the priorities (from
> > https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process).
> >
> > *Blocker* - infrastructure failures, bugs that block us from releasing
> >
> > *Critical* - test instabilities, security related issues, important
> > components is non-functional for important use case
> >
> > *Major (default)* - typical feature requests, bug that affects some use
> > cases
> >
> > *Minor* - nice to have feature requests, wishes, not necessarily under
> > active development or discussion, catch all
> >
> > *Not a Priority *- stale feature request or bug report
> >
> > Tickets (except Not a Priority) need an assignee or an active public
> > discussion otherwise their priority is slowly reduced up to "Not a
> > Priority" automatically by the flink-jira-bot
> > <https://github.com/apache/flink-jira-bot>.
> >
> > The time intervals until they prioritized are configured in
> > https://github.com/apache/flink-jira-bot/blob/master/config.yaml:
> >
> > Blocker: 8 days without assignee or activity
> > Critical 21 days without assignee or activity
> > Major: 67 days without assignee or activity
> > Minor: 187 days without assignee or activity
> >
> > So, the big jump is between Major and Minor, which is why I proposed to
> set
> > the default to Minor.
> >
> > When you propose new priorities ("Normal") I propose you also share how
> it
> > would fit in/change this framework, i.e.
> >
> > * What's the definition?
> > * Does it change other definitions? Do we need to migrate tickets?
> > * What would be the Jira Bot configuration?
> >
> > I know, this is a lot to ask, but otherwise I fear we might talk past
> each
> > other.
> >
> > Cheers,
> >
> > Konstantin
> >
> >
> >
> > On Tue, Jul 27, 2021 at 1:35 PM Martijn Visser
> > wrote:
> >
> > > Hi,
> > >
> > > I would introduce a new priority between Major and Minor. I think
> there's
> > > still value for having a Minor priority, especially for tickets where
> > > something is being reported but there's a workable workaround
> available.
> > >
> > > With regards to Yun's comment, I think that's a follow-up question on
> > what
> > > to do with tickets that are already in the system that are registered
> as
> > > "Major". There are currently 4 Blockers, 29 Critical, 938 Major, 2340
> > Minor
> > > and 0 Not a Priority open tickets. I don't think there's a bulk rule
> that
> > > can be applied to, for example, move all Major to a Normal state. I do
> > > think this will balance itself out over time if you would introduce
> > > "Normal" as a new default priority and the ones who can change Jira
> > tickets
> > > also check if the right priority is set whenever they work on a ticket.
> > >
> > > Best regards,
> > >
> > > Martijn
> > >
> > > On Tue, 27 Jul 2021 at 12:40, Yun Tang wrote:
> > >
> > > > Hi Konstantin,
> > > >
> > > > How about rename "Major" to "Normal"? We already have higher critical
> > and
> > > > blocker priorities, and I personally usually treat current "major" as
> > > > "normal" priority.
> > > >
> >
Re: Security Vulnerabilities with Flink OpenJDK Docker Image
Hi Daniel, sorry for the late reply and thanks for the report. We'll look into this and get back to you. Cheers, Konstantin On Tue, Jun 15, 2021 at 4:33 AM Daniel Moore wrote: > Hello All, > > We have been implementing a solution using the Flink image from > https://github.com/apache/flink-docker/blob/master/1.13/scala_2.12-java11-debian/Dockerfile > and it got flagged by our image repository for 3 major security > vulnerabilities: > > CVE-2017-8804 > CVE-2019-25013 > CVE-2021-33574 > > All of these stem from the `glibc` packages in the `openjdk:11-jre` image. > > We have a working image based on building Flink using the Amazon Corretto > image - > https://github.com/corretto/corretto-docker/blob/88df29474df6fc3f3f19daa8c5515d934f706cd0/11/jdk/al2/Dockerfile. > This works although there are some issues related to linking > `libjemalloc`. Before we fully test this new image we wanted to reach out > to the community for insight on the following questions: > > 1. Are these vulnerabilities captured in an issue yet? > 2. If so, when could we except a new official image that contains the > Debian fixes for these issues? > 3. If not, how can we help contribute to a solution? > 4. Are there officially supported non-Debian based Flink images? > > We appreciate the insights and look forward to working with the community > on a solution. > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Looking for Maintainers for Flink on YARN
Hi everyone Thank you Marton & team for stepping up and thanks, Xintong, for offering your continued support. I'd like to leave this thread "open" for a bit longer so that others can continue to chime in. In the meantime, I would already like to propose two dates for an informal kick-off: * Thu, Aug 11, 9am CEST * Fri, Aug 12, 9am CEST Would any of these dates work for you? As an agenda we propose something along the lines of: * Introductions * Architecture Introduction by Till * Handover of Ongoing Tickets * Open Questions As I am out of office for a month starting tomorrow, Till will take this thread over from hereon. Best, Konstantin Cheers, Konstantin On Mon, Aug 2, 2021 at 9:36 AM Gabor Somogyi wrote: > Hi All, > > Just arrived back from holiday and seen this thread. > I'm working with Marton and you can ping me directly on jira/PR in need. > > BR, > G > > > On Mon, Aug 2, 2021 at 5:02 AM Xintong Song wrote: > > > Thanks Konstantin for bringing this up, and thanks Marton and your team > for > > volunteering. > > > > @Marton, > > Our team at Alibaba will keep helping with the Flink on YARN maintenance. > > However, as Konstaintin said, the time our team dedicated to this will > > probably be less than it used to be. Your offer of help is significant > and > > greatly appreciated. Please feel free to reach out to us if you need any > > help on this. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Thu, Jul 29, 2021 at 5:04 PM Márton Balassi > > > wrote: > > > > > Hi Konstantin, > > > > > > Thank you for raising this topic, our development team at Cloudera > would > > be > > > happy to step up to address this responsibility. > > > > > > Best, > > > Marton > > > > > > On Thu, Jul 29, 2021 at 10:15 AM Konstantin Knauf > > > wrote: > > > > > > > Dear community, > > > > > > > > We are looking for community members, who would like to maintain > > Flink's > > > > YARN support going forward. So far, this has been handled by teams at > > > > Ververica & Alibaba. The focus of these teams has shifted over the > past > > > > months so that we only have little time left for this topic. Still, > we > > > > think, it is important to maintain high quality support for Flink on > > > YARN. > > > > > > > > What does "Maintaining Flink on YARN" mean? There are no known bigger > > > > efforts outstanding. We are mainly talking about addressing > > > > "test-stability" issues, bugs, version upgrades, community > > contributions > > > & > > > > smaller feature requests. The prioritization of these would be up to > > the > > > > future maintainers, except "test-stability" issues which are > important > > to > > > > address for overall productivity. > > > > > > > > If a group of community members forms itself, we are happy to give an > > > > introduction to relevant pieces of the code base, principles, > > > assumptions, > > > > ... and hand over open threads. > > > > > > > > If you would like to take on this responsibility or can join this > > effort > > > in > > > > a supporting role, please reach out! > > > > > > > > Cheers, > > > > > > > > Konstantin > > > > for the Deployment & Coordination Team at Ververica > > > > > > > > -- > > > > > > > > Konstantin Knauf > > > > > > > > https://twitter.com/snntrable > > > > > > > > https://github.com/knaufk > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] Apache Flink 1.13.3 released
Thank you, Chesnay & Martijn, for managing this release! On Thu, Oct 21, 2021 at 10:29 AM Chesnay Schepler wrote: > The Apache Flink community is very happy to announce the release of > Apache Flink 1.13.3, which is the third bugfix release for the Apache > Flink 1.13 series. > > Apache Flink® is an open-source stream processing framework for > distributed, high-performing, always-available, and accurate data > streaming applications. > > The release is available for download at: > https://flink.apache.org/downloads.html > > Please check out the release blog post for an overview of the > improvements for this bugfix release: > https://flink.apache.org/news/2021/10/19/release-1.13.3.html > > The full release notes are available in Jira: > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12350329 > > We would like to thank all contributors of the Apache Flink community > who made this release possible! > > Regards, > Chesnay > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Should we drop Row SerializationSchema/DeserializationSchema?
+1 for deprecating and then dropping them. On Thu, Oct 21, 2021 at 3:31 PM Timo Walther wrote: > Hi Francesco, > > thanks for starting this discussion. It is definitely time to clean up > more connectors and formats that were used for the old planner but are > actually not intended for the DataStream API. > > +1 for deprecating and dropping the mentioned formats. Users can either > use Table API or implement a custom > SerializationSchema/DeserializationSchema according to their needs. It > is actually not that complicated to add Jackson and configure the > ObjectMapper for reading JSON/CSV. > > Regards, > Timo > > > On 18.10.21 17:42, Francesco Guardiani wrote: > > Hi all, > > In flink-avro, flink-csv and flink-json we have implementations of > > SerializationSchema/DeserializationSchema for the > org.apache.flink.types.Row > > type. In particular, I'm referring to: > > > > - org.apache.flink.formats.json.JsonRowSerializationSchema > > - org.apache.flink.formats.json.JsonRowDeserializationSchema > > - org.apache.flink.formats.avro.AvroRowSerializationSchema > > - org.apache.flink.formats.avro.AvroRowDeserializationSchema > > - org.apache.flink.formats.csv.CsvRowDeserializationSchema > > - org.apache.flink.formats.csv.CsvRowSerializationSchema > > > > These classes were used in the old table planner, but now the table > planner > > doesn't use the Row type internally anymore, so these classes are unused > > from the flink-table packages. > > > > Because these classes are exposed (some have @PublicEvolving annotation) > > there might be some users out there using them when using the DataStream > > APIs, for example to convert an input stream of JSON from Kafka to a Row > > instance. > > > > Do you have any opinions about deprecating these classes in 1.15 and then > > drop them in 1.16? Or are you using them? If yes, can you describe your > use > > case? > > > > Thank you, > > > > FG > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Creating an external connector repository
Thank you, Arvid & team, for working on this. I would also favor one connector repository under the ASF. This will already force us to provide better tools and more stable APIs, which connectors developed outside of Apache Flink will benefit from, too. Besides simplifying the formal release process for connectors, I believe, we can also be more liberal with Committership for connector maintainers. I expect that this setup can scale better than the current one, but it doesn't scale super well either. In addition, there is still the ASF barrier to contributions/releases. So, we might have more connectors in this repository than we have in Apache Flink right now, but not all connectors will end up in this repository. For those "external" connectors, we should still aim to improve visibility, documentation and tooling. It feels like such a hybrid approach might be the only option given competing requirements. Thanks, Konstnatin On Mon, Oct 18, 2021 at 10:22 PM Thomas Weise wrote: > Thanks for initiating this discussion. > > There are definitely a few things that are not optimal with our > current management of connectors. I would not necessarily characterize > it as a "mess" though. As the points raised so far show, it isn't easy > to find a solution that balances competing requirements and leads to a > net improvement. > > It would be great if we can find a setup that allows for connectors to > be released independently of core Flink and that each connector can be > released separately. Flink already has separate releases > (flink-shaded), so that by itself isn't a new thing. Per-connector > releases would need to allow for more frequent releases (without the > baggage that a full Flink release comes with). > > Separate releases would only make sense if the core Flink surface is > fairly stable though. As evident from Iceberg (and also Beam), that's > not the case currently. We should probably focus on addressing the > stability first, before splitting code. A success criteria could be > that we are able to build Iceberg and Beam against multiple Flink > versions w/o the need to change code. The goal would be that no > connector breaks when we make changes to Flink core. Until that's the > case, code separation creates a setup where 1+1 or N+1 repositories > need to move lock step. > > Regarding some connectors being more important for Flink than others: > That's a fact. Flink w/o Kafka connector (and few others) isn't > viable. Testability of Flink was already brought up, can we really > certify a Flink core release without Kafka connector? Maybe those > connectors that are used in Flink e2e tests to validate functionality > of core Flink should not be broken out? > > Finally, I think that the connectors that move into separate repos > should remain part of the Apache Flink project. Larger organizations > tend to approve the use of and contribution to open source at the > project level. Sometimes it is everything ASF. More often it is > "Apache Foo". It would be fatal to end up with a patchwork of projects > with potentially different licenses and governance to arrive at a > working Flink setup. This may mean we prioritize usability over > developer convenience, if that's in the best interest of Flink as a > whole. > > Thanks, > Thomas > > > > On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler > wrote: > > > > Generally, the issues are reproducibility and control. > > > > Stuffs completely broken on the Flink side for a week? Well then so are > > the connector repos. > > (As-is) You can't go back to a previous version of the snapshot. Which > > also means that checking out older commits can be problematic because > > you'd still work against the latest snapshots, and they not be > > compatible with each other. > > > > > > On 18/10/2021 15:22, Arvid Heise wrote: > > > I was actually betting on snapshots versions. What are the limits? > > > Obviously, we can only do a release of a 1.15 connector after 1.15 is > > > release. > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Creating an external connector repository
t; >> released separately. Flink already has separate releases > > >> (flink-shaded), so that by itself isn't a new thing. Per-connector > > >> releases would need to allow for more frequent releases (without the > > >> baggage that a full Flink release comes with). > > >> > > >> Separate releases would only make sense if the core Flink surface is > > >> fairly stable though. As evident from Iceberg (and also Beam), that's > > >> not the case currently. We should probably focus on addressing the > > >> stability first, before splitting code. A success criteria could be > > >> that we are able to build Iceberg and Beam against multiple Flink > > >> versions w/o the need to change code. The goal would be that no > > >> connector breaks when we make changes to Flink core. Until that's the > > >> case, code separation creates a setup where 1+1 or N+1 repositories > > >> need to move lock step. > > >> > > >> Regarding some connectors being more important for Flink than others: > > >> That's a fact. Flink w/o Kafka connector (and few others) isn't > > >> viable. Testability of Flink was already brought up, can we really > > >> certify a Flink core release without Kafka connector? Maybe those > > >> connectors that are used in Flink e2e tests to validate functionality > > >> of core Flink should not be broken out? > > >> > > >> Finally, I think that the connectors that move into separate repos > > >> should remain part of the Apache Flink project. Larger organizations > > >> tend to approve the use of and contribution to open source at the > > >> project level. Sometimes it is everything ASF. More often it is > > >> "Apache Foo". It would be fatal to end up with a patchwork of projects > > >> with potentially different licenses and governance to arrive at a > > >> working Flink setup. This may mean we prioritize usability over > > >> developer convenience, if that's in the best interest of Flink as a > > >> whole. > > >> > > >> Thanks, > > >> Thomas > > >> > > >> > > >> > > >> On Mon, Oct 18, 2021 at 6:59 AM Chesnay Schepler > > >> wrote: > > >>> Generally, the issues are reproducibility and control. > > >>> > > >>> Stuffs completely broken on the Flink side for a week? Well then so > are > > >>> the connector repos. > > >>> (As-is) You can't go back to a previous version of the snapshot. > Which > > >>> also means that checking out older commits can be problematic because > > >>> you'd still work against the latest snapshots, and they not be > > >>> compatible with each other. > > >>> > > >>> > > >>> On 18/10/2021 15:22, Arvid Heise wrote: > > >>>> I was actually betting on snapshots versions. What are the limits? > > >>>> Obviously, we can only do a release of a 1.15 connector after 1.15 > is > > >>>> release. > > >>> > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Releasing Flink 1.13.3
Hi Martijn, Thanks for starting the discussion. +1 for a soon Flink 1.13.3. IMHO we don't need to block the release on FLINK-23519 (an improvement), FLINK-21853 (seems like a Minor issue with a test) or FLINK-23946 (depends on https://issues.apache.org/jira/browse/FLINK-23946, which is not started yet). This would leave FLINK-24315, which I can't judge at all. Cheers, Konstantin On Wed, Sep 22, 2021 at 1:34 PM Martijn Visser wrote: > Hi all, > > I would like to start discussing releasing Flink 1.13.3. There are > currently 138 tickets already resolved for 1.13.3, a few important fixes > are: > > * https://issues.apache.org/jira/browse/FLINK-24347 (KafkaSource cannot > checkpoint if the parallelism is higher than the partition number) > * https://issues.apache.org/jira/browse/FLINK-24303 (SourceCoordinator > exception may fail Session Cluster) > * https://issues.apache.org/jira/browse/FLINK-24277 (Offset commit should > be disabled if consumer group ID is not specified in KafkaSource) > * https://issues.apache.org/jira/browse/FLINK-23802 (Reduce > ReadTimeoutExceptions for Kinesis Consumer) > > There are currently 4 tickets with fix version 1.13.3 that are in progress: > > * https://issues.apache.org/jira/browse/FLINK-24315 (Cannot rebuild > watcher > thread while the K8S API server is unavailable) > * https://issues.apache.org/jira/browse/FLINK-23946 (Application mode > fails > fatally when being shut down) > * https://issues.apache.org/jira/browse/FLINK-23519 (Aggregate State > Backend Latency by State Level) > * https://issues.apache.org/jira/browse/FLINK-21853 (Running HA per-job > cluster (rocks, non-incremental) end-to-end test could not finished in 900 > seconds) > > Can Yangze, David, Yun and Till give an update on the status for those? > > Are there any other open tickets that we should wait for? Is there a PMC > member who would like to manage the release? > > Best regards, > > Martijn Visser | Product Manager > > mart...@ververica.com > > <https://www.ververica.com/> > > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Deprecate Java 8 support
+1 On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler wrote: > Hello, > > after recent discussions on the dev > <https://lists.apache.org/thread/0fwo7nwzy51gck4vxhyfnbnttd4jycpx> and > user <https://lists.apache.org/thread/2jnj5xrokphxhokpo5w3p6w7z0pkl9gx> > mailing list to deprecate Java 8 support, with a general consensus in > favor of it, I would now like tod o a formal vote. > > The deprecation would entail a notification to our users to encourage > migrating to Java 11, and various efforts on our side to prepare a > migration to Java 11, like updating some e2e tests to actually run on > Java 11, performance benchmarking etc. . > > There is no set date for the removal of Java 8 support. > > We'll use the usual minimum 72h vote duration, with committers having > binding votes. > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)
ery poor. We are also trying to solve this problem. > > The PatternProcessorManager keeps all the active PatternProcessor in > memory. For scenarios that they want to have their own pattern for each > user_id, IMO, is it possible to reduce the fine-grained pattern of > PatternProcessor to solve the performance problem of the Pattern Matching, > for example, a pattern corresponds to a group of users? The scenarios > mentioned above need to be solved by case by case. > > Best, > Nicholas Jiang > > On 2021/12/17 11:43:10 yue ma wrote: > > Glad to see the Community's progress in Flink CEP. After reading this > Flip, > > I have few questions, would you please take a look ? > > > > 1. About Pattern Updating. If we use PatternProcessoerDiscoverer to > update > > the rules, will it increase the load of JM? For example, if the user > wants > > the updated rule to take effect immediately,, which means that we need to > > set a shorter check interval or there is another scenario when users > > rarely update the pattern, will the PatternProcessoerDiscoverer be in > most > > of the time Do useless checks ? Will a lazy update mode could be used, > > which the pattern only be updated when triggered by the user, and do > > nothing at other times ? > > > > 2. I still have some confusion about how Key Generating Opertator and > > CepOperator (Pattern Matching & Processing Operator) work together. If > > there are N PatternProcessors, will the Key Generating Opertator > generate N > > keyedStreams, and then N CepOperator would process each Key separately ? > Or > > every CepOperator Task would process all patterns, if so, does the key > type > > in each PatternProcessor need to be the same ? > > > > 3. Maybe need to pay attention to it when implementing it .If some > Pattern > > has been removed or updateed ,will the partially matched results in > > StateBackend would be clean up or We rely on state ttl to clean up these > > expired states. > > > > 4. Will the PatternProcessorManager keep all the active PatternProcessor > in > > memory ? We have also Support Multiple Rule and Dynamic Rule Changing . > > But we are facing such a problem, some users’ usage scenarios are that > they > > want to have their own pattern for each user_id, which means that there > > could be thousands of patterns, which would make the performance of > Pattern > > Matching very poor. We are also trying to solve this problem. > > > > Yunfeng Zhou 于2021年12月10日周五 19:16写道： > > > > > Hi all， > > > > > > I'm opening this thread to propose the design to support multiple rule > & > > > dynamic rule changing in the Flink-CEP project, as described in > FLIP-200 > > > [1] > > > . > > > > > > Currently Flink CEP only supports having a single pattern inside a > > > CepOperator and does not support changing the pattern dynamically. In > order > > > to reduce resource consumption and to experience shorter downtime > during > > > pattern updates, there is a growing need in the production environment > that > > > expects CEP to support having multiple patterns in one operator and to > > > support dynamically changing them. Therefore I propose to add certain > > > infrastructure as described in FLIP-200 to support these > functionalities. > > > > > > Please feel free to reply to this email thread. Looking forward to your > > > feedback! > > > > > > [1] > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195730308 > > > > > > Best regards, > > > > > > Yunfeng > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] GHA migration roadmap
m 1 machine (for stragglers) > > >> - Azure cron jobs are disabled, but kept around in case we need to > > revert > > >> - Timeframe: 1-2 weeks > > >> > > >> *Phase 5: *Removal of Azure CI leftovers > > >> - Only after we are satisfied that GHA is stable (at least 1 month > after > > >> the switch, can be longer) > > >> - Green GHA build is required from now on > > >> - Stale PRs that don't have a GHA run will have to trigger a new one > > (but > > >> they would most likely have to rebase anyway...) > > >> - (old) FlinkCIBot is disabled > > >> - Azure yamls are deleted > > >> - Azure runners are removed from machines > > >> > > >> > > >> Timing-wise, the full switch to GHA should happen during a quiet time, > > far > > >> away from a release. The remaining phases shouldn't have much impact, > > but > > >> right before a release is not a good moment, of course. > > >> Please give us your thoughts and point out anything we missed or that > > >> doesn't seem to make sense! > > >> > > >> Best, > > >> Nico > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Releasing Flink 1.14.3
t; >> > >>> > > > > > >> >> > >>> Best regards, > > > > > >> >> > >>> > > > > > >> >> > >>> Martijn > > > > > >> >> > >>> > > > > > >> >> > >>> > > > > > >> >> > >>> On Fri, 26 Nov 2021 at 12:08, Chesnay Schepler < > > > > > >> ches...@apache.org> > > > > > >> >> wrote: > > > > > >> >> > >>>> > > > > > >> >> > >>>> FLINK-25022: I will open a PR later today, and it > > should be > > > > > >> easy to > > > > > >> >> > >>>> backport. > > > > > >> >> > >>>> FLINK-25027: Unlikely to make it for 1.14.1; I also > > wouldn't > > > > > >> >> consider it > > > > > >> >> > >>>> a blocker > > > > > >> >> > >>>> > > > > > >> >> > >>>> On 24/11/2021 19:40, Martijn Visser wrote: > > > > > >> >> > >>>> > Hi all, > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > I would like to start a discussion on releasing > Flink > > 1.14.1. > > > > > >> >> Flink 1.14 > > > > > >> >> > >>>> > was released on the 29th of September [1] and so far > > 107 > > > > > >> issues > > > > > >> >> have been > > > > > >> >> > >>>> > resolved, including multiple blockers and critical > > priorities > > > > > >> >> [2]. > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > There are currently 169 open tickets which contain a > > > > > >> fixVersion > > > > > >> >> for 1.14.1 > > > > > >> >> > >>>> > [3]. I'm including the ones that are currently > marked > > as > > > > > >> >> critical or a > > > > > >> >> > >>>> > blocker to verify if these should be included in > > Flink 1.14.1. > > > > > >> >> It would be > > > > > >> >> > >>>> > great if those that are assigned or working on one > or > > more of > > > > > >> >> these tickets > > > > > >> >> > >>>> > can give an update on its status. > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-24543 > - > > > > > >> Zookeeper > > > > > >> >> connection > > > > > >> >> > >>>> > issue causes inconsistent state in Flink -> I think > > this > > > > > >> depends > > > > > >> >> on the > > > > > >> >> > >>>> > outcome of dropping Zookeeper 3.4 as was proposed on > > the Dev > > > > > >> >> mailing list > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-25027 > > - Allow > > > > > >> GC > > > > > >> >> of a > > > > > >> >> > >>>> > finished job's JobMaster before the slot timeout is > > reached > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-25022 > - > > > > > >> >> ClassLoader leak with > > > > > >> >> > >>>> > ThreadLocals on the JM when submitting a job through > > the REST > > > > > >> API > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-24789 > - > > > > > >> >> IllegalStateException > > > > > >> >> > >>>> > with CheckpointCleaner being closed already > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-24328 > > - Long > > > > > >> term > > > > > >> >> fix for > > > > > >> >> > >>>> > receiving new buffer size before network reader > > configured -> > > > > > >> >> I'm not sure > > > > > >> >> > >>>> > if this would end up in Flink 1.14.1, I think it's > > more likely > > > > > >> >> that it > > > > > >> >> > >>>> > would be Flink 1.15. Anton/Dawid, could you confirm > > this? > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-23946 > - > > > > > >> >> Application mode > > > > > >> >> > >>>> > fails fatally when being shut down -> This depends > on > > > > > >> >> > >>>> > https://issues.apache.org/jira/browse/FLINK-24038 > > and I don't > > > > > >> >> see much > > > > > >> >> > >>>> > happening there, so I also expect that this would > > move to > > > > > >> Flink > > > > > >> >> 1.15. > > > > > >> >> > >>>> > David, could you confirm? > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-22113 > - > > > > > >> UniqueKey > > > > > >> >> constraint > > > > > >> >> > >>>> > is lost with multiple sources join in SQL > > > > > >> >> > >>>> > * https://issues.apache.org/jira/browse/FLINK-21788 > > - Throw > > > > > >> >> > >>>> > PartitionNotFoundException if the partition file has > > been lost > > > > > >> >> for blocking > > > > > >> >> > >>>> > shuffle -> I'm also expecting that this would move > to > > Flink > > > > > >> >> 1.15, can you > > > > > >> >> > >>>> > confirm Yingjie ? > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > There are quite some other tickets that I've > excluded > > from > > > > > >> this > > > > > >> >> list, > > > > > >> >> > >>>> > because they are either test instabilities or are > not > > > > > >> depending > > > > > >> >> on a Flink > > > > > >> >> > >>>> > release to be resolved. > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Note: there are quite a few test instabilities in > the > > list and > > > > > >> >> help on > > > > > >> >> > >>>> > those is always appreciated. You can check all > > unassigned > > > > > >> tickets > > > > > >> >> > >>>> > instabilities in Jira [4]. > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Are there any other open tickets that we should wait > > for? Is > > > > > >> >> there a PMC > > > > > >> >> > >>>> > member who would like to manage the release? I'm > more > > than > > > > > >> happy > > > > > >> >> to help > > > > > >> >> > >>>> > with monitoring the status of the tickets. > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Best regards, > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Martijn > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > [1] > > > > > >> https://flink.apache.org/news/2021/09/29/release-1.14.0.html > > > > > >> >> > >>>> > [2] > > > > > >> >> > >>>> > > > > > > >> >> > > > > > >> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > > > > >> >> > >>>> > [3] > > > > > >> >> > >>>> > > > > > > >> >> > > > > > >> > > > https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > [4] > > > > > >> >> > >>>> > > > > > > >> >> > > > > > >> > > > https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Martijn Visser | Product Manager > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > mart...@ververica.com > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > <https://www.ververica.com/> > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Follow us @VervericaData > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > -- > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Join Flink Forward <https://flink-forward.org/> - > > The Apache > > > > > >> >> Flink > > > > > >> >> > >>>> > Conference > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > Stream Processing | Event Driven | Real Time > > > > > >> >> > >>>> > > > > > > >> >> > >>>> > > > > > >> >> > >> > > > > > >> >> > >> > > > > > >> >> > >> -- > > > > > >> >> > >> Marios > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> -- > > > > > >> >> Best, Jingsong Lee > > > > > >> >> > > > > > >> > > > > > > >> > > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Releasing Flink 1.14.3
fore this > > >> > > >> is > > >> > > >> >> merged. > > >> > > >> >> > >>> * https://issues.apache.org/jira/browse/FLINK-25022 - > > >> > > >> ClassLoader > > >> > > >> >> leak with ThreadLocals on the JM when submitting a job > through > > >> the > > >> > > >> REST API > > >> > > >> >> -> @Chesnay Schepler has provided a PR, so I suspect it > would > > >> also just > > >> > > >> >> take a couple of days before this is merged. > > >> > > >> >> > >>> > > >> > > >> >> > >>> Is there anyone who can help me with creating the > actual > > >> release > > >> > > >> >> when these tickets are resolved > > >> > > >> >> > >>> > > >> > > >> >> > >>> Best regards, > > >> > > >> >> > >>> > > >> > > >> >> > >>> Martijn > > >> > > >> >> > >>> > > >> > > >> >> > >>> > > >> > > >> >> > >>> On Fri, 26 Nov 2021 at 12:08, Chesnay Schepler < > > >> > > >> ches...@apache.org> > > >> > > >> >> wrote: > > >> > > >> >> > >>>> > > >> > > >> >> > >>>> FLINK-25022: I will open a PR later today, and it > > should > > >> be > > >> > > >> easy to > > >> > > >> >> > >>>> backport. > > >> > > >> >> > >>>> FLINK-25027: Unlikely to make it for 1.14.1; I also > > >> wouldn't > > >> > > >> >> consider it > > >> > > >> >> > >>>> a blocker > > >> > > >> >> > >>>> > > >> > > >> >> > >>>> On 24/11/2021 19:40, Martijn Visser wrote: > > >> > > >> >> > >>>> > Hi all, > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > I would like to start a discussion on releasing > Flink > > >> 1.14.1. > > >> > > >> >> Flink 1.14 > > >> > > >> >> > >>>> > was released on the 29th of September [1] and so > far > > >> 107 > > >> > > >> issues > > >> > > >> >> have been > > >> > > >> >> > >>>> > resolved, including multiple blockers and critical > > >> priorities > > >> > > >> >> [2]. > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > There are currently 169 open tickets which contain > a > > >> > > >> fixVersion > > >> > > >> >> for 1.14.1 > > >> > > >> >> > >>>> > [3]. I'm including the ones that are currently > marked > > >> as > > >> > > >> >> critical or a > > >> > > >> >> > >>>> > blocker to verify if these should be included in > > Flink > > >> 1.14.1. > > >> > > >> >> It would be > > >> > > >> >> > >>>> > great if those that are assigned or working on one > or > > >> more of > > >> > > >> >> these tickets > > >> > > >> >> > >>>> > can give an update on its status. > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-24543 > > - > > >> > > >> Zookeeper > > >> > > >> >> connection > > >> > > >> >> > >>>> > issue causes inconsistent state in Flink -> I think > > >> this > > >> > > >> depends > > >> > > >> >> on the > > >> > > >> >> > >>>> > outcome of dropping Zookeeper 3.4 as was proposed > on > > >> the Dev > > >> > > >> >> mailing list > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-25027 > > - > > >> Allow > > >> > > >> GC > > >> > > >> >> of a > > >> > > >> >> > >>>> > finished job's JobMaster before the slot timeout is > > >> reached > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-25022 > > - > > >> > > >> >> ClassLoader leak with > > >> > > >> >> > >>>> > ThreadLocals on the JM when submitting a job > through > > >> the REST > > >> > > >> API > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-24789 > > - > > >> > > >> >> IllegalStateException > > >> > > >> >> > >>>> > with CheckpointCleaner being closed already > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-24328 > > - > > >> Long > > >> > > >> term > > >> > > >> >> fix for > > >> > > >> >> > >>>> > receiving new buffer size before network reader > > >> configured -> > > >> > > >> >> I'm not sure > > >> > > >> >> > >>>> > if this would end up in Flink 1.14.1, I think it's > > >> more likely > > >> > > >> >> that it > > >> > > >> >> > >>>> > would be Flink 1.15. Anton/Dawid, could you confirm > > >> this? > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-23946 > > - > > >> > > >> >> Application mode > > >> > > >> >> > >>>> > fails fatally when being shut down -> This depends > on > > >> > > >> >> > >>>> > https://issues.apache.org/jira/browse/FLINK-24038 > > and > > >> I don't > > >> > > >> >> see much > > >> > > >> >> > >>>> > happening there, so I also expect that this would > > move > > >> to > > >> > > >> Flink > > >> > > >> >> 1.15. > > >> > > >> >> > >>>> > David, could you confirm? > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-22113 > > - > > >> > > >> UniqueKey > > >> > > >> >> constraint > > >> > > >> >> > >>>> > is lost with multiple sources join in SQL > > >> > > >> >> > >>>> > * > https://issues.apache.org/jira/browse/FLINK-21788 > > - > > >> Throw > > >> > > >> >> > >>>> > PartitionNotFoundException if the partition file > has > > >> been lost > > >> > > >> >> for blocking > > >> > > >> >> > >>>> > shuffle -> I'm also expecting that this would move > to > > >> Flink > > >> > > >> >> 1.15, can you > > >> > > >> >> > >>>> > confirm Yingjie ? > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > There are quite some other tickets that I've > excluded > > >> from > > >> > > >> this > > >> > > >> >> list, > > >> > > >> >> > >>>> > because they are either test instabilities or are > not > > >> > > >> depending > > >> > > >> >> on a Flink > > >> > > >> >> > >>>> > release to be resolved. > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Note: there are quite a few test instabilities in > the > > >> list and > > >> > > >> >> help on > > >> > > >> >> > >>>> > those is always appreciated. You can check all > > >> unassigned > > >> > > >> tickets > > >> > > >> >> > >>>> > instabilities in Jira [4]. > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Are there any other open tickets that we should > wait > > >> for? Is > > >> > > >> >> there a PMC > > >> > > >> >> > >>>> > member who would like to manage the release? I'm > more > > >> than > > >> > > >> happy > > >> > > >> >> to help > > >> > > >> >> > >>>> > with monitoring the status of the tickets. > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Best regards, > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Martijn > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > [1] > > >> > > >> https://flink.apache.org/news/2021/09/29/release-1.14.0.html > > >> > > >> >> > >>>> > [2] > > >> > > >> >> > >>>> > > > >> > > >> >> > > >> > > >> > > >> > > > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > >> > > >> >> > >>>> > [3] > > >> > > >> >> > >>>> > > > >> > > >> >> > > >> > > >> > > >> > > > https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > [4] > > >> > > >> >> > >>>> > > > >> > > >> >> > > >> > > >> > > >> > > > https://issues.apache.org/jira/issues?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20fixVersion%20%3D%201.14.1%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20priority%20DESC%2C%20created%20DESC > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Martijn Visser | Product Manager > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > mart...@ververica.com > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > <https://www.ververica.com/> > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Follow us @VervericaData > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > -- > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Join Flink Forward <https://flink-forward.org/> - > > The > > >> Apache > > >> > > >> >> Flink > > >> > > >> >> > >>>> > Conference > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > Stream Processing | Event Driven | Real Time > > >> > > >> >> > >>>> > > > >> > > >> >> > >>>> > > >> > > >> >> > >> > > >> > > >> >> > >> > > >> > > >> >> > >> -- > > >> > > >> >> > >> Marios > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> > > >> > > >> >> -- > > >> > > >> >> Best, Jingsong Lee > > >> > > >> >> > > >> > > >> > > > >> > > >> > > >> > > > > > >> > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)
ash the job, right? > > > > > > IMO, if for whatever reason the heartbeat timeout, it couldn't check > the > > > PatternProcessor consistency between the OperatorCoordinator and the > > > subtasks so that the job would be crashed. > > > > > > -- Konstantin: What I was concerned about is that we basically let > users > > > run a UserFunction in the OperatorCoordinator, which it does not seem > to > > > have been designed for. > > > > > > In general, we have reached an agreement on the design of this FLIP, > but > > > there are some concerns on the OperatorCoordinator, about whether > > basically > > > let users run a UserFunction in the OperatorCoordinator is designed for > > > OperatorCoordinator. We would like to invite Becket Qin who is the > author > > > of OperatorCoordinator to help us to answer this concern. > > > > > > Best, > > > Nicholas Jiang > > > > > > > > > On 2021/12/20 10:07:14 Martijn Visser wrote: > > > > Hi all, > > > > > > > > Really like the discussion on this topic moving forward. I really > think > > > > this feature will be much appreciated by the Flink users. What I > still > > > have > > > > left to answer/reply to: > > > > > > > > -- Good point. If for whatever reason the different taskmanagers > can't > > > get > > > > the latest rule, the Operator Coordinator could send a heartbeat to > all > > > > taskmanagers with the latest rules and check the heartbeat response > > from > > > > all the taskmanagers whether the latest rules of the taskmanager is > > equal > > > > to these of the Operator Coordinator. > > > > > > > > Just to be sure, this indeed would mean that if for whatever reason > the > > > > heartbeat timeout, it would crash the job, right? > > > > > > > > -- We have consided about the solution mentioned above. In this > > > solution, I > > > > have some questions about how to guarantee the consistency of the > rule > > > > between each TaskManager. By having a coodinator in the JobManager to > > > > centrally manage the latest rules, the latest rules of all > TaskManagers > > > are > > > > consistent with those of the JobManager, so as to avoid the > > > inconsistencies > > > > that may be encountered in the above solution. Can you introduce how > > this > > > > solution guarantees the consistency of the rules? > > > > > > > > The consistency that we could guarantee was based on how often each > > > > TaskManager would do a refresh and how often we would accept a > refresh > > to > > > > fail. We set the refresh time to a relatively short one (30 seconds) > > and > > > > maximum failures to 3. That meant that we could guarantee that rules > > > would > > > > be updated in < 2 minutes or else the job would crash. That was > > > sufficient > > > > for our use cases. This also really depends on how big your cluster > > is. I > > > > can imagine that if you have a large scale cluster that you want to > > run, > > > > you don't want to DDOS the backend system where you have your rules > > > stored. > > > > > > > > -- In summary, the current design is that JobManager tells all > > > TaskManagers > > > > the latest rules through OperatorCoodinator, and will initiate a > > > heartbeat > > > > to check whether the latest rules on each TaskManager are consistent. > > We > > > > will describe how to deal with the Failover scenario in more detail > on > > > FLIP. > > > > > > > > Thanks for that. I think having the JobManager tell the TaskManagers > > the > > > > applicable rules would indeed end up being the best design. > > > > > > > > -- about the concerns around consistency raised by Martijn: I think a > > lot > > > > of those can be mitigated by using an event time timestamp from which > > the > > > > rules take effect. The reprocessing scenario, for example, is covered > > by > > > > this. If a pattern processor should become active as soon as > possible, > > > > there will still be inconsistencies between Taskmanagers, but "as > soon > > as > > > > possible" is vague anyway, which is why I think that's ok. > > &g
Re: [DISCUSS] FLIP-203: Incremental savepoints
Hi Piotr, Thanks a lot for starting the discussion. Big +1. In my understanding, this FLIP introduces the snapshot format as a *really* user facing concept. IMO it is important that we document a) that it is not longer the checkpoint/savepoint characteristics that determines the kind of changes that a snapshots allows (user code, state schema evolution, topology changes), but now this becomes a property of the format regardless of whether this is a snapshots or a checkpoint b) the exact changes that each format allows (code, state schema, topology, state backend, max parallelism) In this context: will the native format support state schema evolution? If not, I am not sure, we can let the format default to native. Thanks, Konstantin On Mon, Dec 20, 2021 at 2:09 PM Piotr Nowojski wrote: > Hi devs, > > I would like to start a discussion about a previously announced follow up > of the FLIP-193 [1], namely allowing savepoints to be in native format and > incremental. The changes do not seem invasive. The full proposal is > written down as FLIP-203: Incremental savepoints [2]. Please take a look, > and let me know what you think. > > Best, > Piotrek > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-193%3A+Snapshots+ownership > [2] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Semantic > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-193: Snapshots ownership
Hi Dawid, Thanks for working on this FLIP. Clarifying the differences and guarantees around savepoints and checkpoints will make it easier and safer for users and downstream projects and platforms to work with them. +1 to the changing the current (undefined) behavior when recovering from retained checkpoints. Users can now choose between claiming and not claiming, which I think will make the current mixed behavior obsolete. Cheers, Konstantin On Fri, Nov 19, 2021 at 8:19 AM Dawid Wysakowicz wrote: > Hi devs, > > I'd like to bring up for a discussion a proposal to clean up ownership > of snapshots, both checkpoints and savepoints. > > The goal here is to make it clear who is responsible for deleting > checkpoints/savepoints files and when can that be done in a safe manner. > > Looking forward for your feedback! > > Best, > > Dawid > > [1] https://cwiki.apache.org/confluence/x/bIyqCw > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Improve the name and structure of job vertex and operator name for job
> improvement
> > >>>>>>>>>> on
> > >>>>>>>>>>>> name
> > >>>>>>>>>>>>>>> and structure of job vertex name, mainly to improve
> > >>>>>>> experience of
> > >>>>>>>>>>>>>> debugging
> > >>>>>>>>>>>>>>> and analyzing sql job at runtime.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> the main proposed changes including:
> > >>>>>>>>>>>>>>> 1. separate description and name for operator, so
> > >> that
> > >>>>>> we
> > >>>>>>> can
> > >>>>>>>>>> have
> > >>>>>>>>>>>>>> detailed
> > >>>>>>>>>>>>>>> info at description and shorter name, which could be
> > >>>>>> more
> > >>>>>>>>>> friendly
> > >>>>>>>>>>>> for
> > >>>>>>>>>>>>>>> external systems like logging/metrics without losing
> > >>>>>> useful
> > >>>>>>>>>>>> information.
> > >>>>>>>>>>>>>>> 2. introduce a tree-mode vertex description which
> > >> can
> > >>>>>> make
> > >>>>>>> the
> > >>>>>>>>>>>>>> description
> > >>>>>>>>>>>>>>> more readable and easier to understand
> > >>>>>>>>>>>>>>> 3. clean up and improve description for sql operator
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> here is an example with the changes for a sql job:
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> vertex name:
> > >>>>>>>>>>>>>>> GlobalGroupAggregate[52] -> (Calc[53] ->
> > >>>>>>> NotNullEnforcer[54] ->
> > >>>>>>>>>>> Sink:
> > >>>>>>>>>>>>>>> tb_ads_dwi_pub_hbd_spm_dtr_002_003[54], Calc[55] ->
> > >>>>>>>>>>>> NotNullEnforcer[56]
> > >>>>>>>>>>>>>> ->
> > >>>>>>>>>>>>>>> Sink: tb_ads_dwi_pub_hbd_spm_dtr_002_004[56])
> > >>>>>>>>>>>>>>> vertex description:
> > >>>>>>>>>>>>>>> [52]:GlobalGroupAggregate(groupBy=[stat_date,
> > >>>>>> spm_url_ab,
> > >>>>>>>>>> client],
> > >>>>>>>>>>>>>>> select=[stat_date, spm_url_ab, client,
> > >> COUNT(count1$0)
> > >>>>>> AS
> > >>>>>>>>>>>>>>> clk_cnt_app_mtr_001, COUNT(distinct$0 count$1) AS
> > >>>>>>>>>>> clk_uv_app_mtr_001,
> > >>>>>>>>>>>>>>> COUNT(count1$2) AS clk_cnt_app_mtr_002,
> > >> COUNT(distinct$0
> > >>>>>>> count$3)
> > >>>>>>>>>>> AS
> > >>>>>>>>>>>>>>> clk_uv_app_mtr_002, COUNT(count1$4) AS
> > >>>>>> clk_cnt_app_mtr_003,
> > >>>>>>>>>>>>>>> COUNT(distinct$0 count$5) AS clk_uv_app_mtr_003]) :-
> > >>>>>>>>>>>>>>> [53]:Calc(select=[CASE((client <> ''),
> > >>>>>> CONCAT_WS('\u0004',
> > >>>>>>>>>>>>>>> CONCAT(SUBSTRING(MD5(CONCAT(spm_url_ab, '12345')),
> > >> 1,
> > >>>>>> 4),
> > >>>>>>>>>> ':md5'),
> > >>>>>>>>>>>>>>> CONCAT(spm_url_ab, ':spmab'), '12345:app',
> > >>>>>> CONCAT(client,
> > >>>>>>>>>>> ':client'),
> > >>>>>>>>>>>>>>> CONCAT('ddd:', stat_date)),
> > >> null:VARCHAR(2147483647)) AS
> > >>>>>>> rowkey,
> > >>>>>>>>>>>>>>> clk_cnt_app_mtr_001 AS clk_cnt_app_dtr_001,
> > >>>>>>> clk_uv_app_mtr_001 AS
> > >>>>>>>>>>>>>>> clk_uv_app_dtr_001, clk_cnt_app_mtr_002 AS
> > >>>>>>> clk_cnt_app_dtr_002,
> > >>>>>>>>>>>>>>> clk_uv_app_mtr_002 AS clk_uv_app_dtr_002,
> > >>>>>>> clk_cnt_app_mtr_003 AS
> > >>>>>>>>>>>>>>> clk_cnt_app_dtr_003, clk_uv_app_mtr_003 AS
> > >>>>>>> clk_uv_app_dtr_003]) :
> > >>>>>>>>>>> +-
> > >>>>>>>>>>>>>>> [54]:NotNullEnforcer(fields=[rowkey]) : +-
> > >>>>>>>>>>>>>>>
> > >>
> >
> [54]:Sink(table=[default_catalog.default_database.tb_ads_dwi_pub_hbd_spm_dtr_002_003],
> > >>>>>>>>>>>>>>> fields=[rowkey, clk_cnt_app_dtr_001,
> > >> clk_uv_app_dtr_001,
> > >>>>>>>>>>>>>>> clk_cnt_app_dtr_002, clk_uv_app_dtr_002,
> > >>>>>>> clk_cnt_app_dtr_003,
> > >>>>>>>>>>>>>>> clk_uv_app_dtr_003]) +-
> > >> [55]:Calc(select=[CASE((client
> > >>>>>> <>
> > >>>>>>> ''),
> > >>>>>>>>>>>>>>> CONCAT_WS('\u0004',
> > >>>>>> CONCAT(SUBSTRING(MD5(CONCAT(spm_url_ab,
> > >>>>>>>>>>>> '12345')), 1,
> > >>>>>>>>>>>>>>> 4), ':md5'), CONCAT(spm_url_ab, ':spmab'),
> > >> '12345:app',
> > >>>>>>>>>>>> CONCAT('ddd:',
> > >>>>>>>>>>>>>>> stat_date), CONCAT(client, ':client')), (client =
> > >> ''),
> > >>>>>>>>>>>>>> CONCAT_WS('\u0004',
> > >>>>>>>>>>>>>>> CONCAT(SUBSTRING(MD5(CONCAT(spm_url_ab, '92459')),
> > >> 1,
> > >>>>>> 4),
> > >>>>>>>>>> ':md5'),
> > >>>>>>>>>>>>>>> CONCAT(spm_url_ab, ':spmab'), '92459:app',
> > >>>>>> CONCAT('ddd:',
> > >>>>>>>>>>>> stat_date)),
> > >>>>>>>>>>>>>>> null:VARCHAR(2147483647)) AS rowkey,
> > >>>>>> clk_cnt_app_mtr_001 AS
> > >>>>>>>>>>>>>>> clk_cnt_app_dtr_001, clk_uv_app_mtr_001 AS
> > >>>>>>> clk_uv_app_dtr_001,
> > >>>>>>>>>>>>>>> clk_cnt_app_mtr_002 AS clk_cnt_app_dtr_002,
> > >>>>>>> clk_uv_app_mtr_002 AS
> > >>>>>>>>>>>>>>> clk_uv_app_dtr_002, clk_cnt_app_mtr_003 AS
> > >>>>>>> clk_cnt_app_dtr_003,
> > >>>>>>>>>>>>>>> clk_uv_app_mtr_003 AS clk_uv_app_dtr_003]) +-
> > >>>>>>>>>>>>>>> [56]:NotNullEnforcer(fields=[rowkey]) +-
> > >>>>>>>>>>>>>>>
> > >>
> >
> [56]:Sink(table=[default_catalog.default_database.tb_ads_dwi_pub_hbd_spm_dtr_002_004],
> > >>>>>>>>>>>>>>> fields=[rowkey, clk_cnt_app_dtr_001,
> > >> clk_uv_app_dtr_001,
> > >>>>>>>>>>>>>>> clk_cnt_app_dtr_002, clk_uv_app_dtr_002,
> > >>>>>>> clk_cnt_app_dtr_003,
> > >>>>>>>>>>>>>>> clk_uv_app_dtr_003])
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> For more detail on the proposal:
> > >>>>>>>>>>>>>>>
> > >>
> >
> https://docs.google.com/document/d/1VUVJeHY_We09GY53-K2lETP3HUNZG9wMKyecFWk_Wxk
> > >>>>>>>>>>>>>>> <
> > >>
> >
> https://docs.google.com/document/d/1VUVJeHY_We09GY53-K2lETP3HUNZG9wMKyecFWk_Wxk/edit#
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Looking forward to your feedback, thanks.
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Bests
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>> Wenlong Lyu
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> >
> >
>
--
Konstantin Knauf
https://twitter.com/snntrable
https://github.com/knaufk
Re: [DISCUSS] FLIP-193: Snapshots ownership
hat support "fast" copy (ideally with latency numbers). > > Regards, > Roman > > On Mon, Nov 22, 2021 at 9:24 AM Yun Gao > > wrote: > > Hi, > > Very thanks Dawid for proposing the FLIP to clarify the ownership for the > states. +1 for the overall changes since it makes the behavior clear and > provide users a determined method to finally cleanup savepoints / retained > checkpoints. > > Regarding the changes to the public interface, it seems currently the changes > are all bound > to the savepoint, but from the FLIP it seems perhaps we might also need to > support the claim declaration > for retained checkpoints like in the cli side[1] ? If so, then might it be > better to change the option name > from `execution.savepoint.restore-mode` to something like > `execution.restore-mode`? > > Best, > Yun > > > [1] > https://nightlies.apache.org/flink/flink-docs-master/docs/ops/state/checkpoints/#resuming-from-a-retained-checkpoint > > > -- > From:Konstantin Knauf > > Send Time:2021 Nov. 19 (Fri.) 16:00 > To:dev > > Subject:Re: [DISCUSS] FLIP-193: Snapshots ownership > > Hi Dawid, > > Thanks for working on this FLIP. Clarifying the differences and > guarantees around savepoints and checkpoints will make it easier and safer > for users and downstream projects and platforms to work with them. > > +1 to the changing the current (undefined) behavior when recovering from > retained checkpoints. Users can now choose between claiming and not > claiming, which I think will make the current mixed behavior obsolete. > > Cheers, > > Konstantin > > On Fri, Nov 19, 2021 at 8:19 AM Dawid Wysakowicz > > wrote: > > Hi devs, > > I'd like to bring up for a discussion a proposal to clean up ownership > of snapshots, both checkpoints and savepoints. > > The goal here is to make it clear who is responsible for deleting > checkpoints/savepoints files and when can that be done in a safe manner. > > Looking forward for your feedback! > > Best, > > Dawid > > [1] https://cwiki.apache.org/confluence/x/bIyqCw > > > > -- > > Konstantin Knaufhttps://twitter.com/snntrablehttps://github.com/knaufk > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] FLIP-189: SQL Client Usability Improvements
+1 (binding) On Fri, Nov 5, 2021 at 4:03 PM Martijn Visser wrote: > +1 (non-binding). Looking forward! > > Best regards, > > Martijn > > On Fri, 5 Nov 2021 at 15:36, Timo Walther wrote: > > > +1 (binding) thanks for working on this. > > > > Regards, > > Timo > > > > > > On 05.11.21 10:14, Sergey Nuyanzin wrote: > > > Also there is a short demo showing some of the features mentioned in > this > > > FLIP. > > > It is available at https://asciinema.org/a/446247?speed=3.0 (It was > also > > > mentioned in [DISCUSS] thread) > > > > > > On Wed, Nov 3, 2021 at 11:04 PM Sergey Nuyanzin > > wrote: > > > > > >> > > >> Hi everyone, > > >> > > >> I would like to start a vote on FLIP-189: SQL Client Usability > > >> Improvements [1]. > > >> The FLIP was discussed in this thread [2]. > > >> FLIP-189 targets usability improvements of SQL Client such as parsing > > >> improvement, > > >> syntax highlighting, completion, prompts > > >> > > >> The vote will be open for at least 72 hours unless there is an > objection > > >> or not enough votes. > > >> > > >> [1] > > >> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements > > >> [2] https://lists.apache.org/thread/8d580jcqzpcbmfwqvhjso82hdd2x0461 > > >> > > >> -- > > >> Best, > > >> Sergey > > >> > > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk
Congratulations, Ingo! On Thu, Dec 2, 2021 at 7:13 PM Johannes Moser wrote: > What Martijn said, > > > On 02.12.2021, at 18:32, Martijn Visser wrote: > > > > Congrats Ingo, well deserved! > > > > Op do 2 dec. 2021 om 18:13 schreef Mika Naylor > > > >> Congratulations, Ingo! > >> > >> Best, > >> Mika > >> > >> On 02.12.2021 17:37, Matthias Pohl wrote: > >>> Congratulations, Ingo! Good job! :) > >>> > >>> On Thu, Dec 2, 2021 at 5:28 PM David Morávek wrote: > >>> > >>>> Congrats Ingo, well deserved ;) > >>>> > >>>> Best, > >>>> D. > >>>> > >>>> On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz < > dwysakow...@apache.org > >>> > >>>> wrote: > >>>> > >>>>> Congratulations Ingo! Happy to have you onboard as a committer! > >>>>> > >>>>> Best, > >>>>> > >>>>> Dawid > >>>>> > >>>>> On 02/12/2021 17:14, Francesco Guardiani wrote: > >>>>>> Congrats Ingo! > >>>>>> > >>>>>> On Thu, Dec 2, 2021 at 4:58 PM Nicolaus Weidner < > >>>>>> nicolaus.weid...@ververica.com> wrote: > >>>>>> > >>>>>>> Congrats Ingo! > >>>>>>> The PMC probably realized that it's simply too much work to review > >> and > >>>>>>> merge all your PRs, so now you can/have to do part of that work > >>>> yourself > >>>>>>> ;-) > >>>>>>> > >>>>>>> Best, > >>>>>>> Nico > >>>>>>> > >>>>>>> On Thu, Dec 2, 2021 at 4:50 PM Fabian Paul > >> wrote: > >>>>>>> > >>>>>>>> Thanks for always pushing Ingo. Congratulations! > >>>>>>>> > >>>>>>>> Best, > >>>>>>>> Fabian > >>>>>>>> > >>>>>>>> On Thu, Dec 2, 2021 at 4:24 PM Till Rohrmann < > >> trohrm...@apache.org> > >>>>>>> wrote: > >>>>>>>>> Hi everyone, > >>>>>>>>> > >>>>>>>>> On behalf of the PMC, I'm very happy to announce Ingo Bürk as a > >> new > >>>>>>> Flink > >>>>>>>>> committer. > >>>>>>>>> > >>>>>>>>> Ingo has started contributing to Flink since the beginning of > >> this > >>>>>>> year. > >>>>>>>> He > >>>>>>>>> worked mostly on SQL components. He has authored many PRs and > >> helped > >>>>>>>> review > >>>>>>>>> a lot of other PRs in this area. He actively reported issues and > >>>>> helped > >>>>>>>> our > >>>>>>>>> users on the MLs. His most notable contributions were Support SQL > >>>> 2016 > >>>>>>>> JSON > >>>>>>>>> functions in Flink SQL (FLIP-90), Register sources/sinks in Table > >>>> API > >>>>>>>>> (FLIP-129) and various other contributions in the SQL area. > >>>> Moreover, > >>>>>>> he > >>>>>>>> is > >>>>>>>>> one of the few people in our community who actually understands > >>>>> Flink's > >>>>>>>>> frontend. > >>>>>>>>> > >>>>>>>>> Please join me in congratulating Ingo for becoming a Flink > >>>> committer! > >>>>>>>>> > >>>>>>>>> Cheers, > >>>>>>>>> Till > >>>>> > >>>>> > >>>> > >> > > -- > > > > Martijn Visser | Product Manager > > > > mart...@ververica.com > > > > <https://www.ververica.com/> > > > > > > Follow us @VervericaData > > > > -- > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > Conference > > > > Stream Processing | Event Driven | Real Time > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl
Congratulations, Matthias! On Thu, Dec 2, 2021 at 8:03 PM Johannes Moser wrote: > Congrats Matthias. > > > On 02.12.2021, at 18:32, Martijn Visser wrote: > > > > Congratulations Matthias! > > > > Op do 2 dec. 2021 om 18:12 schreef Mika Naylor > > > >> Congratulations Matthias! > >> > >> Best, > >> Mika > >> > >> On 02.12.2021 17:27, David Morávek wrote: > >>> Congrats Matthias, well deserved ;) > >>> > >>> Best, > >>> D. > >>> > >>> On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz < > dwysakow...@apache.org> > >>> wrote: > >>> > >>>> Congratulations Matthias! Really well deserved! > >>>> > >>>> Best, > >>>> > >>>> Dawid > >>>> > >>>> On 02/12/2021 16:53, Nicolaus Weidner wrote: > >>>>> Congrats Matthias, well deserved! > >>>>> > >>>>> Best, > >>>>> Nico > >>>>> > >>>>> On Thu, Dec 2, 2021 at 4:48 PM Fabian Paul wrote: > >>>>> > >>>>>> Congrats and well deserved. > >>>>>> > >>>>>> Best, > >>>>>> Fabian > >>>>>> > >>>>>> On Thu, Dec 2, 2021 at 4:42 PM Ingo Bürk > wrote: > >>>>>>> Congrats, Matthias! > >>>>>>> > >>>>>>> On Thu, Dec 2, 2021 at 4:28 PM Till Rohrmann > > >>>>>> wrote: > >>>>>>>> Hi everyone, > >>>>>>>> > >>>>>>>> On behalf of the PMC, I'm very happy to announce Matthias Pohl as > a > >>>> new > >>>>>>>> Flink committer. > >>>>>>>> > >>>>>>>> Matthias has worked on Flink since August last year. He helped > >> review > >>>>>> a ton > >>>>>>>> of PRs. He worked on a variety of things but most notably the > >> tracking > >>>>>> and > >>>>>>>> reporting of concurrent exceptions, fixing HA bugs and deprecating > >> and > >>>>>>>> removing our Mesos support. He actively reports issues helping > >> Flink > >>>> to > >>>>>>>> improve and he is actively engaged in Flink's MLs. > >>>>>>>> > >>>>>>>> Please join me in congratulating Matthias for becoming a Flink > >>>>>> committer! > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>> > >>>> > >> > > -- > > > > Martijn Visser | Product Manager > > > > mart...@ververica.com > > > > <https://www.ververica.com/> > > > > > > Follow us @VervericaData > > > > -- > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > Conference > > > > Stream Processing | Event Driven | Real Time > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] FLIP-193: Snapshots ownership
Thanks, Dawid. +1 On Wed, Dec 1, 2021 at 1:23 PM Dawid Wysakowicz wrote: > Dear devs, > > I'd like to open a vote on FLIP-193: Snapshots ownership [1] which was > discussed in this thread [2]. > The vote will be open for at least 72 hours unless there is an objection or > not enough votes. > > Best, > > Dawid > > [1] https://cwiki.apache.org/confluence/x/bIyqCw > > [2] https://lists.apache.org/thread/zw2crf0c7t7t4cb5cwcwjpvsb3r1ovz2 > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-196: Source API stability guarantees
Hi Till, Thank you for picking up this topic. Explicit and consistent stability guarantees are important for our users as well as any downstream project of Apache Flink. Documenting the de-facto status quo and tackling existing inconsistencies sounds like a good first step. So, +1 from my side. Two questions for clarity: * Are you planning to implement the required tests via ArchUnit? * Fixing existing "test failures" is in the scope of this FLIP, correct? Cheers, Konstantin On Thu, Dec 2, 2021 at 3:48 PM Till Rohrmann wrote: > Hi everyone, > > I would like to start a discussion about what kind of API stability > guarantees we want to provide to our users. The Flink community already > agreed on some stability guarantees, but these guarantees were only > communicated via the ML and not properly documented [2]. Moreover, I've > seen more and more complaints about breaking changes (some rightful and > others not) on the ML recently [3] because we rarely mark our APIs as > stable. This motivated this FLIP. > > The proposal first concentrates on source API stability guarantees. Binary > stability guarantees are explicitly left out for a follow-up discussion. > > In essence, the proposal follows our current stability guarantees while > updating the guarantees for @PublicEvolving that we are already providing > w/o having stated them. Moreover, this FLIP proposes some guidelines for > determining the stability guarantees for an API object, how to evolve them > and some additional requirements for the return and parameter types of > methods. > > All in all, I hope that this proposal is more reflecting our current > understanding of stability guarantees than being controversial. One of the > outcomes of this FLIP should be to properly document these guarantees so > that it is easily discoverable and understandable for our users. Moreover, > I hope that we can provide more stable APIs our users can rely and build > upon. > > There will be a follow-up FLIP discussing the problem of how to make sure > that APIs become stable over time. > > Looking forward to your feedback. > > [1] https://cwiki.apache.org/confluence/x/IJeqCw > [2] https://lists.apache.org/thread/5jm25783oq5svyk7rr8g1gly2ooxqhjr > [3] https://lists.apache.org/thread/kzhfc3t6omzo2kyo8zj9yxoh8twq5fzr > > Cheers, > Till > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-197: API stability graduation process
Hi Till, This makes sense to me, in general. I am wondering two things: * is one release to promote enough or will it lead to a lot of exclusions? FLIP-27, for example, took much longer to stabilize, I believe. If we think, this is quite a strict rule, then in my experience, let's rather start with an easier rule that is broken less, then a strict rule that is broken from the start. * If I could choose whether we focus on this FLIP or another follow up of FLIP-196 like establishing stability guarantees for the REST API, I would choose the latter. Cheers, Konstantin On Thu, Dec 2, 2021 at 3:58 PM Till Rohrmann wrote: > Hi everyone, > > As promised, here is the follow-up FLIP [1] for discussing how we can > ensure that newly introduced APIs are being stabilized over time. This FLIP > is related to FLIP-196 [2]. > > The idea of FLIP-197 is to introduce an API graduation process that forces > us to increase the API stability guarantee unless there is a very good > reason not to do so. So the proposal is to reverse the process from opt-in > (increasing the stability guarantee explicitly) to opt-out (deciding that > an API cannot be graduated with a good reason). > > Since every process breaks if it is not automated, we propose a richer set > of API stability annotations that can capture enough information so that we > can implement a test that fails if we fail to follow the process. > > Looking forward to your feedback. > > Hopefully, we can provide our users a better experience when working with > Flink because we offer more stable APIs and make them available faster. > > [1] https://cwiki.apache.org/confluence/x/J5eqCw > [2] https://cwiki.apache.org/confluence/x/IJeqCw > > Cheers, > Till > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Deprecate MapR FS
+1 (what Seth said) On Thu, Dec 9, 2021 at 4:15 PM Seth Wiesman wrote: > +1 > > I actually thought we had already dropped this FS. If anyone is still > relying on it in production, the file system abstraction in Flink has been > incredibly stable over the years. They should be able to use the 1.14 MapR > FS with later versions of Flink. > > Seth > > On Wed, Dec 8, 2021 at 10:03 AM Martijn Visser > wrote: > >> Hi all, >> >> Flink supports multiple file systems [1] which includes MapR FS. MapR as >> a company doesn't exist anymore since 2019, the technology and intellectual >> property has been sold to Hewlett Packard. >> >> I don't think that there's anyone who's using MapR anymore and therefore >> I think it would be good to deprecate this for Flink 1.15 and then remove >> it in Flink 1.16. Removing this from Flink will slightly shrink the >> codebase and CI runtime. >> >> I'm also cross posting this to the User mailing list, in case there's >> still anyone who's using MapR. >> >> Best regards, >> >> Martijn >> >> [1] >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >> > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Immediate dedicated Flink releases for log4j vulnerability
coming in a few weeks, but releasing asap. > > > > > > >>> The mood I perceive in the industry is pretty much panicky > over > > > this, > > > > > > >> and I > > > > > > >>> expect we will see many requests for a patched release and > many > > > > > > >> discussions > > > > > > >>> why the workaround alone would not be enough due to certain > > > > > guidelines. > > > > > > >>> > > > > > > >>> I suggest that we preempt those discussions and create > releases > > > the > > > > > > >>> following way: > > > > > > >>> > > > > > > >>> - we take the latest already released versions from each > > release > > > > > > >> branch: > > > > > > >>> ==> 1.14.0, 1.13.3, 1.12.5, 1.11.4 > > > > > > >>> - we add a single commit to those that just updates the > log4j > > > > > > >> dependency > > > > > > >>> - we release those as 1.14.1, 1.13.4, 1.12.6, 1.11.5, etc. > > > > > > >>> - that way we don't need to do functional release tests, > > > because the > > > > > > >>> released code is identical to the previous release, except > for > > > the > > > > > > log4j > > > > > > >>> dependency > > > > > > >>> - we can then continue the work on the upcoming bugfix > > releases > > > as > > > > > > >>> planned, without high pressure > > > > > > >>> > > > > > > >>> I would suggest creating those RCs immediately and release > them > > > with > > > > > a > > > > > > >>> special voting period (24h or so). > > > > > > >>> > > > > > > >>> WDYT? > > > > > > >>> > > > > > > >>> Best, > > > > > > >>> Stephan > > > > > > >>> > > > > > > >>> [1] https://nvd.nist.gov/vuln/detail/CVE-2021-44228 > > > > > > >>> [2] https://flink.apache.org/2021/12/10/log4j-cve.html > > > > > > >>> > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Best, Jingsong Lee > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)
Thanks, Yufeng, for starting this discussion. I think this will be a very popular feature. I've seen a lot of users asking for this in the past. So, generally big +1. I think we should have a rough idea on how to expose this feature in the other APIs. Two ideas: 1. In order to make this more deterministic in case of reprocessing and out-of-orderness, I am wondering if we can add a timestamp to each rule that determines the start time from which the rule should be in effect. This can be an event or a processing time depending on the characteristics of the pipeline. The timestamp would default to Long.MIN_TIMESTAMP if not provided, which means effectively immediately. This could also be a follow up, if you think it will make the implementation too complicated initially. 2. I am wondering, if we should name Rule->DynamicPatternHolder or so and CEP.rule-> CEP.dynamicPatterns instead (other classes correspondingly)? Rule is quite ambiguous and DynamicPattern seems more descriptive to me. On Mon, Dec 13, 2021 at 4:30 AM Nicholas Jiang wrote: > Hi Martijn, > >IMO, in this FLIP, we only need to introduce the general design of the > Table API/SQL level. As for the design details, you can create a new FLIP. > And do we need to take into account the support for Batch mode if you > expand the MATCH_RECOGNIZE function? About the dynamic rule engine design, > do you have any comments? This core of the FLIP is about the multiple rule > and dynamic rule changing mechanism. > > Best, > Nicholas Jiang > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)
Hi Nicholas, I am not sure I understand your question about renaming. I think the most important member of the current Rule class is the Pattern, the KeySelector and PatternProcessFunction are more auxiliary if you will. That's why, I think, it would be ok to rename Rule to DynamicPatternHolder although it contains more than just a Pattern. Cheers, Konstantin On Mon, Dec 13, 2021 at 9:16 AM Nicholas Jiang wrote: > Hi Konstantin, > >Thanks for your feedback. The point that add a timestamp to each rule > that determines the start time from which the rule makes sense to me. At > present, The timestamp is current time at default, so no timestamp field > represents the start time from which the rule. And about the renaming rule, > your suggestion looks good to me and no any new concept introduces. But > does this introduce Rule concept or reuse the Pattern concept for the > DynamicPattern renaming? > > Best, > Nicholas Jiang > > On 2021/12/13 07:45:04 Konstantin Knauf wrote: > > Thanks, Yufeng, for starting this discussion. I think this will be a very > > popular feature. I've seen a lot of users asking for this in the past. > So, > > generally big +1. > > > > I think we should have a rough idea on how to expose this feature in the > > other APIs. > > > > Two ideas: > > > > 1. In order to make this more deterministic in case of reprocessing and > > out-of-orderness, I am wondering if we can add a timestamp to each rule > > that determines the start time from which the rule should be in effect. > > This can be an event or a processing time depending on the > characteristics > > of the pipeline. The timestamp would default to Long.MIN_TIMESTAMP if not > > provided, which means effectively immediately. This could also be a > follow > > up, if you think it will make the implementation too complicated > initially. > > > > 2. I am wondering, if we should name Rule->DynamicPatternHolder or so and > > CEP.rule-> CEP.dynamicPatterns instead (other classes correspondingly)? > > Rule is quite ambiguous and DynamicPattern seems more descriptive to me. > > > > On Mon, Dec 13, 2021 at 4:30 AM Nicholas Jiang > > > wrote: > > > > > Hi Martijn, > > > > > >IMO, in this FLIP, we only need to introduce the general design of > the > > > Table API/SQL level. As for the design details, you can create a new > FLIP. > > > And do we need to take into account the support for Batch mode if you > > > expand the MATCH_RECOGNIZE function? About the dynamic rule engine > design, > > > do you have any comments? This core of the FLIP is about the multiple > rule > > > and dynamic rule changing mechanism. > > > > > > Best, > > > Nicholas Jiang > > > > > > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)
Hi Nicholas, I understand that a Rule contains more than the Pattern. Still, I favor DynamicPattern[Holder] over Rule, because the term "Rule" does not exist in Flink's CEP implementation so far and "dynamic" seems to be the important bit here. Cheers, Konstantin On Tue, Dec 14, 2021 at 4:46 AM Nicholas Jiang wrote: > Hi DianFu, > > Thanks for your feedback of the FLIP. > > About the mentioned question for the `getLatestRules`, IMO, this > doesn't need to rename into `getRuleChanges` because this method is used > for getting the total amount of the latest rules which has been updated > once. > > About the CEP.rule method, the CEP.dynamicPattern renaming is > confusing for users. The dynamic pattern only creates the PatternStream not > the DataStream. From the concept, a dynamic pattern is also a pattern, not > contains the PatternProcessFunction. If renaming the CEP.rule into > CEP.dynamicPattern, the return value of the method couldn't include the > PatternProcessFunction, only returns the PatternStream. I think the > difference between the Rule and the Pattern is that Rule contains the > PatternProcessFunction, but the Pattern or DynamicPattern doesn't contain > the function. > > Best > Nicholas Jiang > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS][FLINK-24427] Hide Scala from table planner
Hi Francesco, Thanks for this summary. When would the follow ups (mentioned under out of scope) be done? I am asking because the intermediate state of the uber JAR described in the document seems a bit messy and I fear that users will stumble across that. For the old type system for UDFs: naive question as I am not involved in SQL much, is there an agreed upon deprecation/removal plan for the legacy type system yet? Cheers, Konstantin On Wed, Dec 8, 2021 at 6:02 PM Francesco Guardiani wrote: > Hi all, > In case you haven't seen, last week I published in the issue comments this > document to explain how we're proceeding to hide Scala from table planner: > > https://docs.google.com/document/d/12yDUCnvcwU2mODBKTHQ1xhfOq1ujYUrXltiN_rbhT34/edit?usp=sharing > > There is a section I've added yesterday which is particularly relevant, > because it explains the impact on the distribution. I strongly encourage > people to look at it. > > Once we perform all the changes, I'm gonna announce them on the user > mailing list as well, together with the package name changes already > brought in by #17897 <https://github.com/apache/flink/pull/17897> to > flink-parquet and flink-orc. > > Thanks, > FG > > -- > > Francesco Guardiani | Software Engineer > > france...@ververica.com > > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > > Ververica GmbH > > Registered at Amtsgericht Charlottenburg: HRB 158244 B > > Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, > Jinwei (Kevin) Zhang > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Change Default Jira Priority from "Major" to "Minor"
Quick Question to Martijn, Jingsong: do you propose to rename "Minor" to "Normal" or would you like to introduce a new priority between Major and Minor? On Tue, Jul 27, 2021 at 11:44 AM Jingsong Li wrote: > I agree with Martijn. > > My problem is just minor, which will make me a little disappointed. > > Best, > Jingsong > > > On Tue, Jul 27, 2021 at 5:32 PM Martijn Visser > wrote: > > > Hi, > > > > I personally would prefer to use "Normal" as a default priority because I > > think a lot of people's first reaction is that their reported problem is > > bigger than a minor loss of function [1], resulting in them choosing the > > next priority which is currently "Major". > > > > Best regards, > > > > Martijn > > > > [1] > > > > > https://issues.apache.org/jira/secure/ShowConstantsHelp.jspa?decorator=popup#PriorityLevels > > > > On Mon, 26 Jul 2021 at 11:54, Caizhi Weng wrote: > > > > > Hi Konstantin! > > > > > > Thanks for raising this up. From my point of view it is a reasonable > > > change. But I think it would be better to handle different types of > > tickets > > > respectively. For example, for bugs a default major seems to be better, > > > while for others the default shall be minor. > > > > > > Konstantin Knauf 于2021年7月26日周一 下午3:35写道： > > > > > > > Hi everyone, > > > > > > > > In [1] I proposed to change the default priority of our Jira project > to > > > > "Minor". Most tickets are opened with the default priority. Arguably > > the > > > > majority of these tickets fall into the "Minor" category instead of > > > "Major" > > > > according to the definition in [3] (and the implementation of the > Jira > > > bot > > > > [2]). Specifically, tickets in "Minor" stay untouched by the Jira bot > > > much > > > > longer than tickets in "Major". > > > > > > > > Since this is affecting every new ticket, I would like to collect a > bit > > > > more feedback. So, what do you think? > > > > > > > > Thanks, > > > > > > > > Konstantin > > > > > > > > [1] https://lists.apache.org/x/list.html?dev@flink.apache.org:lte=1M > : > > > > [2] https://github.com/apache/flink-jira-bot > > > > [3] > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process#FlinkJiraProcess-TicketsPriorities > > > > > > > > -- > > > > > > > > Konstantin Knauf > > > > > > > > https://twitter.com/snntrable > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > -- > Best, Jingsong Lee > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Change Default Jira Priority from "Major" to "Minor"
Hi everyone, In [1] I proposed to change the default priority of our Jira project to "Minor". Most tickets are opened with the default priority. Arguably the majority of these tickets fall into the "Minor" category instead of "Major" according to the definition in [3] (and the implementation of the Jira bot [2]). Specifically, tickets in "Minor" stay untouched by the Jira bot much longer than tickets in "Major". Since this is affecting every new ticket, I would like to collect a bit more feedback. So, what do you think? Thanks, Konstantin [1] https://lists.apache.org/x/list.html?dev@flink.apache.org:lte=1M: [2] https://github.com/apache/flink-jira-bot [3] https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process#FlinkJiraProcess-TicketsPriorities -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Change Default Jira Priority from "Major" to "Minor"
Hi everyone,
well that escalated quickly :) Let me share some more background. This is
our current definition for the priorities (from
https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process).
*Blocker* - infrastructure failures, bugs that block us from releasing
*Critical* - test instabilities, security related issues, important
components is non-functional for important use case
*Major (default)* - typical feature requests, bug that affects some use
cases
*Minor* - nice to have feature requests, wishes, not necessarily under
active development or discussion, catch all
*Not a Priority *- stale feature request or bug report
Tickets (except Not a Priority) need an assignee or an active public
discussion otherwise their priority is slowly reduced up to "Not a
Priority" automatically by the flink-jira-bot
<https://github.com/apache/flink-jira-bot>.
The time intervals until they prioritized are configured in
https://github.com/apache/flink-jira-bot/blob/master/config.yaml:
Blocker: 8 days without assignee or activity
Critical 21 days without assignee or activity
Major: 67 days without assignee or activity
Minor: 187 days without assignee or activity
So, the big jump is between Major and Minor, which is why I proposed to set
the default to Minor.
When you propose new priorities ("Normal") I propose you also share how it
would fit in/change this framework, i.e.
* What's the definition?
* Does it change other definitions? Do we need to migrate tickets?
* What would be the Jira Bot configuration?
I know, this is a lot to ask, but otherwise I fear we might talk past each
other.
Cheers,
Konstantin
On Tue, Jul 27, 2021 at 1:35 PM Martijn Visser
wrote:
> Hi,
>
> I would introduce a new priority between Major and Minor. I think there's
> still value for having a Minor priority, especially for tickets where
> something is being reported but there's a workable workaround available.
>
> With regards to Yun's comment, I think that's a follow-up question on what
> to do with tickets that are already in the system that are registered as
> "Major". There are currently 4 Blockers, 29 Critical, 938 Major, 2340 Minor
> and 0 Not a Priority open tickets. I don't think there's a bulk rule that
> can be applied to, for example, move all Major to a Normal state. I do
> think this will balance itself out over time if you would introduce
> "Normal" as a new default priority and the ones who can change Jira tickets
> also check if the right priority is set whenever they work on a ticket.
>
> Best regards,
>
> Martijn
>
> On Tue, 27 Jul 2021 at 12:40, Yun Tang wrote:
>
> > Hi Konstantin,
> >
> > How about rename "Major" to "Normal"? We already have higher critical and
> > blocker priorities, and I personally usually treat current "major" as
> > "normal" priority.
> >
> > Best,
> > Yun Tang
> >
> > From: Konstantin Knauf
> > Sent: Tuesday, July 27, 2021 18:03
> > To: dev
> > Subject: Re: [DISCUSS] Change Default Jira Priority from "Major" to
> "Minor"
> >
> > Quick Question to Martijn, Jingsong: do you propose to rename "Minor" to
> > "Normal" or would you like to introduce a new priority between Major and
> > Minor?
> >
> > On Tue, Jul 27, 2021 at 11:44 AM Jingsong Li
> > wrote:
> >
> > > I agree with Martijn.
> > >
> > > My problem is just minor, which will make me a little disappointed.
> > >
> > > Best,
> > > Jingsong
> > >
> > >
> > > On Tue, Jul 27, 2021 at 5:32 PM Martijn Visser
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I personally would prefer to use "Normal" as a default priority
> > because I
> > > > think a lot of people's first reaction is that their reported problem
> > is
> > > > bigger than a minor loss of function [1], resulting in them choosing
> > the
> > > > next priority which is currently "Major".
> > > >
> > > > Best regards,
> > > >
> > > > Martijn
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ShowConstantsHelp.jspa?decorator=popup#PriorityLevels
> > > >
> > > > On Mon, 26 Jul 2021 at 11:54, Caizhi Weng
> > wrote:
> > > >
> > > > > Hi Konstantin!
> > > > >
> > > > > Thanks for raising this up. From my point of view it is a
> reasonable
> > > > > change. But
Re: [VOTE] Create a separate sub project for FLIP-188: flink-store
+1 to a separate repository assuming this repository will still be part of Apache Flink (same PMC, Committers). I am not aware we have something like "sub-projects" officially. I share Till and Timo's concerns regarding "store". On Fri, Jan 7, 2022 at 9:59 AM Till Rohrmann wrote: > +1 for the separate project. > > I would agree that flink-store is not the best name. flink-storage > > flink-store but I would even more prefer a name that conveys that it has > something to do with table storage. > > Cheers, > Till > > On Fri, Jan 7, 2022 at 9:14 AM Timo Walther wrote: > > > +1 for the separate project > > > > But maybe use `flink-storage` instead of `flink-store`? > > > > I'm not a native speaker but store is defined as "A place where items > > may be purchased.". It almost sounds like the `flink-packages` project. > > > > Regards, > > Timo > > > > > > On 07.01.22 08:37, Jingsong Li wrote: > > > Hi everyone, > > > > > > I'd like to start a vote for create a separate sub project for > > > FLIP-188 [1]: `flink-store`. > > > > > > - If you agree with the name `flink-store`, please just +1 > > > - If you have a better suggestion, please write your suggestion, > > > followed by a reply that can +1 to the name that has appeared > > > - If you do not want it to be a subproject of flink, just -1 > > > > > > The vote will be open for at least 72 hours unless there is an > > > objection or not enough votes. > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage > > > > > > Best, > > > Jingsong > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Use of JIRA fixVersion
Hi Thomas, I am also +1 to set fixVersion more conservatively, in particular for patch releases. I am not sure we can or should solve this in a "strict" way. I am happy to give contributors and contributing teams a bit of freedom in how they use fixVersion for planning, but giving more concrete guidance than what we currently have in [1] would be desirable, I believe. Cheers, Konstantin [1] https://cwiki.apache.org/confluence/display/FLINK/Flink+Jira+Process#FlinkJiraProcess-FixVersion/s : On Tue, Jan 11, 2022 at 8:44 PM Thomas Weise wrote: > Hi, > > As part of preparing the 1.14.3 release, I observed that there were > around 200 JIRA issues with fixVersion 1.14.3 that were unresolved > (after blocking issues had been dealt with). Further cleanup resulted > in removing fixVersion 1.14.3 from most of these and we are left with > [1] - these are the tickets that rolled over to 1.14.4. > > The disassociated issues broadly fell into following categories: > > * test infrastructure / stability related (these can be found by label) > * stale tickets (can also be found by label) > * tickets w/o label that pertain to addition of features that don't > fit into or don't have to go into patch release > > I wanted to bring this up so that we can potentially come up with > better guidance for use of the fixVersion field, since it is important > for managing releases [2]. Manual cleanup as done in this case isn't > desirable. A few thoughts: > > * In most cases, it is not necessary to set fixVersion upfront. > Instead, we can set it when the issue is actually resolved, and set it > for all versions/branches for which a backport occured after the > changes are merged > * How to know where to backport? "Affect versions" seems to be the > right field to use for that purpose. While resolving an issue for > master it can guide backporting. > * What if an issue should block a release? The priority of the issue > should be blocker. Blockers are low cardinality and need to be fixed > before release. So that would be the case where fixVersion is set > upfront. > > Thanks, > Thomas > > [1] > https://issues.apache.org/jira/issues/?jql=project%20%3D%20Flink%20and%20fixVersion%20%3D%201.14.4%20and%20resolution%20%3D%20Unresolved%20 > [2] > https://cwiki.apache.org/confluence/display/FLINK/Creating+a+Flink+Release > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Flink native k8s integration vs. operator
tegration is really >>>> suitable here. The direction that we're headed is with the standalone >>>> deployment on Kubernetes + the reactive mode (adaptive scheduler). >>>> > >>>> > In theory, if we want to build a really cloud (Kubernetes) native >>>> stream processor, deploying the pipeline should be as simple as deploying >>>> any other application. It should be also simple to integrate with CI & CD >>>> environment and the fast / frequent deploy philosophy. >>>> > >>>> > Let's see where we stand and where we can expand from there: >>>> > >>>> > a) Resource efficiency >>>> > >>>> > We already have the reactive mode in place. This allows you to add / >>>> remove task managers by adjusting the TM deployment (`kubectl scale ...`) >>>> and Flink will automatically react to the available resources. This is >>>> currently only supported with the Application Mode, that is limited to a >>>> single job (which should be enough for this kind of workload). >>>> > >>>> > The re-scaling logic is left completely up to the user and can be as >>>> simple as setting up a HPA (Horizontal Pod Autoscaler). I tend to think in >>>> the direction, that we might want to provide a custom k8s metrics server, >>>> that allows HPA to query the metrics from JM, to make this more flexible >>>> and easy to use. >>>> > >>>> > As this looks really great in theory, there are still some >>>> shortcomings that we're actively working on addressing. For this feature to >>>> be really widely adopted, we need to make the re-scaling experience as fast >>>> as possible, so we can re-scale often to react to the input rate. This >>>> could be currently a problem with large RocksDB states as this involves >>>> full re-balance of the state (splitting / merging RocksDB instances). The >>>> k8s operator approach has the same / even worse limitation as it involves >>>> taking a savepoint a re-building the state from it. >>>> > >>>> > b) Fast recovery >>>> > >>>> > This is IMO not as different from the native mode (although I'd have >>>> to check whether RM failover can reuse task managers). This involves >>>> frequent and fast checkpointing, local recovery (which is still not >>>> supported in reactive mode, but this will be hopefully addressed soon) and >>>> working directory efforts [4]. >>>> > >>>> > c) Application upgrades >>>> > >>>> > This is the topic I'm still struggling with a little. Historically >>>> this involves external lifecycle management (savepoint + submitting a new >>>> job). I think at the end of the day, with application mode on standalone >>>> k8s, it could be as simple as updating the docker image of the JM >>>> deployment. >>>> > >>>> > If I think about the simplest upgrade scenario, simple in-place >>>> restore from the latest checkpoint, it may be fairly simple to implement. >>>> What I'm struggling with are the more complex upgrade scenarios such as >>>> dual, blue / green deployment. >>>> > >>>> > >>>> > To sum this up, I'd really love if Flink could provide great out-of >>>> the box experience with standalone mode on k8s, that makes the experience >>>> as close to running / operating any other application as possible. >>>> > >>>> > I'd really appreciate to hear your thoughts on this topic. >>>> > >>>> > [1] >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-187%3A+Adaptive+Batch+Job+Scheduler >>>> > [2] https://github.com/flink-extended/flink-remote-shuffle >>>> > [3] >>>> https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/elastic_scaling/ >>>> > [4] >>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-198%3A+Working+directory+for+Flink+processes >>>> > >>>> > Best, >>>> > D. >>>> > >>>> > On Tue, Jan 4, 2022 at 12:44 AM Thomas Weise wrote: >>>> >> >>>> >> Hi, >>>> >> >>>> >> I was recently looking at the Flink native Kubernetes integration [1] >>>> >> to get an idea how it relates to existing operator based solutions >>>> >> [2], [3]. >>>> >> >>>> >> Part of the native integration's motivations was simplicity (no extra >>>> >> component to install), but arguably that is also a shortcoming. The >>>> >> k8s operator model can offer support for application lifecycle like >>>> >> upgrade and rescaling, as well as job submission without a Flink >>>> >> client. >>>> >> >>>> >> When using the Flink native integration it would still be necessary >>>> to >>>> >> provide that controller functionality. Is the idea to use the native >>>> >> integration for task manager resource allocation in tandem with an >>>> >> operator that provides the external controller functionality? If >>>> >> anyone using the Flink native integration can share experience, I >>>> >> would be curious to learn more about the specific setup and if there >>>> >> are plans to expand the k8s native integration capabilities. >>>> >> >>>> >> For example: >>>> >> >>>> >> * Application upgrade with features such as [4]. Since the job >>>> manager >>>> >> is part of the deployment it cannot orchestrate the deployment. It >>>> >> needs to be the responsibility of an external process. Has anyone >>>> >> contemplated adding such a component to Flink itself? >>>> >> >>>> >> * Rescaling: Theoretically a parallelism change could be performed >>>> w/o >>>> >> restart of the job manager pod. Hence, building blocks to trigger and >>>> >> apply rescaling could be part of Flink itself. Has this been explored >>>> >> further? >>>> >> >>>> >> Yang kindly pointed me to [5]. Is the recommendation/conclusion that >>>> >> when a k8s operator is already used, then let it be in charge of the >>>> >> task manager resource allocation? If so, what scenario was the native >>>> >> k8s integration originally intended for? >>>> >> >>>> >> Thanks, >>>> >> Thomas >>>> >> >>>> >> [1] >>>> https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/#deployment-modes >>>> >> [2] https://github.com/lyft/flinkk8soperator >>>> >> [3] https://github.com/spotify/flink-on-k8s-operator >>>> >> [4] >>>> https://github.com/lyft/flinkk8soperator/blob/master/docs/state_machine.md >>>> >> [5] https://lists.apache.org/thread/8cn99f6n8nhr07n5vqfo880tpm624s5d >>>> >>> -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime
Thanks, Wenhao. On Thu, Jan 13, 2022 at 4:19 PM Wenhao Ji wrote: > It seems that we have reached a consensus that the proposal will not > be implemented in Flink. I will mark the FLIP as discarded if there > are no objections. > > Thanks, everyone, for joining the discussion again! > > Wenhao > > > On Tue, Jan 11, 2022 at 11:12 PM Wenhao Ji wrote: > > > > Hi all, > > > > Yes, indeed. > > After I did some investigation on similar features provided by the > > Cloud platforms, I actually found several popular Clouds have already > > offered this. > > > > - AWS Kinesis: Setting the Application Logging Level [1], which is > > implemented by UpdateApplication API [2]. > > - Ververica: Logging & Metrics[3], by changing the template. > > - Alicloud: Configure job logs [4], which is quite similar to > > Ververica also by changing the template > > - Cloudera: Enabling Flink DEBUG logging[5], by changing the > > Configuration and triggering a restart > > > > It looks like this feature is not necessary. It has been developed in > > one way or another by so many platforms in the ecosystem. > > > > [1]: > https://docs.aws.amazon.com/kinesisanalytics/latest/java/cloudwatch-logs.html#cloudwatch-level > > [2]: > https://docs.aws.amazon.com/kinesisanalytics/latest/apiv2/API_UpdateApplication.html > > [3]: https://docs.ververica.com/platform_operations/logging_metrics.html > > [4]: > https://www.alibabacloud.com/help/doc-detail/173646.htm#title-1ay-hju-pka > > [5]: > https://docs.cloudera.com/csa/1.6.0/monitoring/topics/csa-ssb-enabling-debug-logging.html > > > > Thanks, > > Wenhao > > > > On Tue, Jan 11, 2022 at 8:24 PM Martijn Visser > wrote: > > > > > > Hi all, > > > > > > I agree with Konstantin, this feels like a problem that shouldn't be > solved > > > via Apache Flink but via the logging ecosystem itself. > > > > > > Best regards, > > > > > > Martijn > > > > > > On Tue, 11 Jan 2022 at 13:11, Konstantin Knauf > wrote: > > > > > > > I've now read over the discussion on the ticket, and I am personally > not in > > > > favor of adding this functionality to Flink via the REST API or Web > UI. I > > > > believe that changing the logging configuration via the existing > > > > configuration files (log4j or logback) is good enough, to justify not > > > > increasing the scope of Flink in that direction. As you specifically > > > > mention YARN: doesn't Cloudera's Hadoop platform, for example, offer > means > > > > to manage the configuration files for all worker nodes from a central > > > > configuration management system? It overall feels like we are trying > to > > > > solve a problem in Apache Flink that is already solved in its > ecosystem and > > > > increases the scope of the project without adding core value. I also > expect > > > > that over time the exposed logging configuration options would > become more > > > > and more complex. > > > > > > > > I am curious to hear what others think. > > > > > > > > On Tue, Jan 11, 2022 at 10:34 AM Chesnay Schepler < > ches...@apache.org> > > > > wrote: > > > > > > > > > Reloading the config from the filesystem is already enabled by > default; > > > > > that was one of the things that made us switch to Log4j 2. > > > > > > > > > > The core point of contention w.r.t. this topic is whether having > the > > > > > admin ssh into the machine is too inconvenient. > > > > > > > > > > Personally I still think that the the current capabilities are > > > > > sufficient, and I do not want us to rely on internals of the > logging > > > > > backends in production code. > > > > > > > > > > On 10/01/2022 17:26, Konstantin Knauf wrote: > > > > > > Thank you for starting the discussion. Being able to change the > logging > > > > > > level at runtime is very valuable in my experience. > > > > > > > > > > > > Instead of introducing our own API (and eventually even > persistence), > > > > > could > > > > > > we just periodically reload the log4j or logback configuration > from the > > > > > > environment/filesystem? I only quickly googled the topic and > [1,2] > > > > > suggest > > > > > &
Re: [DISCUSS] Move Flink website to privacy friendly Analytics solution
Hi Martijn, I think this is a great initiative. Thank you for pursuing this. It allows us to a) generate better insights into the usage of Apache Flink and its documentation as shown in the video a) do this in a privacy preserving way and c) act as a role model for other Apache projects on this matter Big +1. I am happy to help, if I can. Cheers, Konstantin On Fri, Jan 14, 2022 at 11:21 AM Martijn Visser wrote: > Hi everyone, > > The Flink website currently uses Google Analytics to track how visitors of > the website are interacting with it. It provides insights into which > documentation pages are visited, how users are using the website (what's > the cycle of pages they visit before exiting the page), if they are > downloading Flink etc. However, the Apache Software Foundation discourages > using Google Analytics [1] unless meeting certain requirements. The Flink > website currently does not meet those requirements. > > I do believe that it's useful to understand what parts of a website are > important to users, what features are most frequently read up on, where > they get lost in the docs, etc. so we can better understand how users use > the system, the website, and the docs and where to focus improvements next. > > I would like to move the Flink website from Google Analytics to an > alternative as soon as possible for Flink. I would be in favour of opening > up insights to this data for everyone too, it's public data anyway. > > For the past couple of months, I've been engaging in a conversation with > ASF Legal and ASF Infra about setting up a privacy-friendly alternative for > Google Analytics for all ASF projects via the priv...@apache.org mailing > list (I can't find a public web archive link for this unfortunately). As > part of that discussion, I've done a test with the open source and > self-hosted version of Matomo [2], taking a look at the privacy > implications and the functionality that this tool offers. You can watch a > recording of that experiment [3] and view the test setup I've used [4]. > > The current status is that ASF Legal, ASF Infra and I have agreed to take > the next step on this project. This step means that: > > * I set up Matomo on a VM provided by ASF Infra > * A new DNS name is created (either https://analytics.apache.org/ or > https://matomo.analytics.apache.org/) by ASF Infra > * The Flink website is adjusted to remove the tracking from Google > Analytics and include the necessary Javascript to allow tracking of the > Flink website and documentation in Matomo > > If this test would be successful, ASF Infra would take over the hosting of > this solution and provide it to all ASF projects. > > I would like to understand from the Flink community: > > 1. Do you think this is a good idea? > > 2. If yes, I need a couple of PMCs for requesting a VM from Apache Infra > [5] > > Best regards, > > Martijn > https://twitter.com/MartijnVisser82 > > [1] https://privacy.apache.org/faq/committers.html > [2] https://matomo.org/ > [3] > > https://drive.google.com/file/d/1yomYhLoyrzBW620bpn_dROiwyvSCzuvt/view?usp=sharing > [4] https://github.com/MartijnVisser/matomo-analytics > [5] https://infra.apache.org/vm-for-project.html > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Future of Per-Job Mode
Hi everyone, I would like to discuss and understand if the benefits of having Per-Job Mode in Apache Flink outweigh its drawbacks. *# Background: Flink's Deployment Modes* Flink currently has three deployment modes. They differ in the following dimensions: * main() method executed on Jobmanager or Client * dependencies shipped by client or bundled with all nodes * number of jobs per cluster & relationship between job and cluster lifecycle* (supported resource providers) ## Application Mode * main() method executed on Jobmanager * dependencies already need to be available on all nodes * dedicated cluster for all jobs executed from the same main()-method (Note: applications with more than one job, currently still significant limitations like missing high-availability). Technically, a session cluster dedicated to all jobs submitted from the same main() method. * supported by standalone, native kubernetes, YARN ## Session Mode * main() method executed in client * dependencies are distributed from and by the client to all nodes * cluster is shared by multiple jobs submitted from different clients, independent lifecycle * supported by standalone, Native Kubernetes, YARN ## Per-Job Mode * main() method executed in client * dependencies are distributed from and by the client to all nodes * dedicated cluster for a single job * supported by YARN only *# Reasons to Keep** There are use cases where you might need the combination of a single job per cluster, but main() method execution in the client. This combination is only supported by per-job mode. * It currently exists. Existing users will need to migrate to either session or application mode. *# Reasons to Drop** With Per-Job Mode and Application Mode we have two modes that for most users probably do the same thing. Specifically, for those users that don't care where the main() method is executed and want to submit a single job per cluster. Having two ways to do the same thing is confusing. * Per-Job Mode is only supported by YARN anyway. If we keep it, we should work towards support in Kubernetes and Standalone, too, to reduce special casing. * Dropping per-job mode would reduce complexity in the code and allow us to dedicate more resources to the other two deployment modes. * I believe with session mode and application mode we have to easily distinguishable and understandable deployment modes that cover Flink's use cases: * session mode: olap-style, interactive jobs/queries, short lived batch jobs, very small jobs, traditional cluster-centric deployment mode (fits the "Hadoop world") * application mode: long-running streaming jobs, large scale & heterogenous jobs (resource isolation!), application-centric deployment mode (fits the "Kubernetes world") *# Call to Action* * Do you use per-job mode? If so, why & would you be able to migrate to one of the other methods? * Am I missing any pros/cons? * Are you in favor of dropping per-job mode midterm? Cheers and thank you, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Flink native k8s integration vs. operator
Hi Thomas, Yes, I was referring to a separate repository under Apache Flink. Cheers, Konstantin On Thu, Jan 13, 2022 at 6:19 AM Thomas Weise wrote: > Hi everyone, > > Thanks for the feedback and discussion. A few additional thoughts: > > [Konstantin] > With respect to common lifecycle management operations: > these features are > > not available (within Apache Flink) for any of the other resource > providers > > (YARN, Standalone) either. From this perspective, I wouldn't consider > this > > a shortcoming of the Kubernetes integration. > > I think time and evolution of the ecosystem are factors to consider as > well. The state and usage of Flink was much different when YARN > integration was novel. Expectations are different today and the > lifecycle functionality provided by an operator may as well be > considered essential to support the concept of a Flink application on > k8s. After few years learning from operator experience outside of > Flink it might be a good time to fill the gap. > > [Konstantin] > I still believe that we should keep this focus on low > > level composable building blocks (like Jobs and Snapshots) in Apache > Flink > > to make it easy for everyone to build fitting higher level abstractions > > like a FlinkApplication Custom Resource on top of it. > > I completely agree that it is important that the basic functions of > Flink are solid and continued focus is necessary. Thanks for sharing > the pointers, these are great improvements. At the same time, > ecosystem, contributor base and user spectrum are growing. There have > been significant additions in many areas of Flink including connectors > and higher level abstractions like statefun, SQL and Python. It's also > evident from additional repositories/subprojects that we have in Flink > today. > > [Konstantin] > Having said this, if others in the community have the > capacity to push and > > *maintain* a somewhat minimal "reference" Kubernetes Operator for Apache > > Flink, I don't see any blockers. If or when this happens, I'd see some > > clear benefits of using a separate repository (easier independent > > versioning and releases, different build system & tooling (go, I > assume)). > > Naturally different contributors to the project have different focus. > Let's find out if there is strong enough interest to take this on and > strong enough commitment to maintain. As I see it, there is a > tremendous amount of internal investment going into operationalizing > Flink within many companies. Improvements to the operational side of > Flink like the operator would complement Flink nicely. I assume that > you are referring to a separate repository within Apache Flink, which > would give it the chance to achieve better sustainability than the > existing external operator efforts. There is also the fact that some > organizations which are heavily invested in operationalizing Flink are > allowing contributing to Apache Flink itself but less so to arbitrary > github projects. Regarding the tooling, it could well turn out that > Java is a good alternative given the ecosystem focus and that there is > an opportunity for reuse in certain aspects (metrics, logging etc.). > > [Yang] > I think Xintong has given a strong point why we introduced > the native K8s integration, which is active resource management. > > I have a concrete example for this in the production. When a K8s node is > down, the standalone K8s deployment will take longer > > recovery time based on the K8s eviction time(IIRC, default is 5 > minutes). For the native K8s integration, Flink RM could be aware of the > > TM heartbeat lost and allocate a new one timely. > > Thanks for sharing this, we should evaluate it as part of a proposal. > If we can optimize recovery or scaling with active resource management > then perhaps it is worth to support it through the operator. > Previously mentioned operators all rely on the standalone model. > > Cheers, > Thomas > > On Wed, Jan 12, 2022 at 3:21 AM Konstantin Knauf > wrote: > > > > cc dev@ > > > > Hi Thomas, Hi everyone, > > > > Thank you for starting this discussion and sorry for chiming in late. > > > > I agree with Thomas' and David's assessment of Flink's "Native Kubernetes > > Integration", in particular, it does actually not integrate well with the > > Kubernetes ecosystem despite being called "native" (tooling, security > > concerns). > > > > With respect to common lifecycle management operations: these features > are > > not available (within Apache Flink) for any of the other resource > providers > > (YARN, Standalone
Re: [DISCUSS] FLIP-203: Incremental savepoints
Hi Piotr, would it be possible to provide a table that shows the compatibility guarantees provided by the different snapshots going forward? Like type of change (Topology. State Schema, Parallelism, ..) in one dimension, and type of snapshot as the other dimension. Based on that, it would be easier to discuss those guarantees, I believe. Cheers, Konstantin On Mon, Jan 3, 2022 at 9:11 AM David Morávek wrote: > Hi Piotr, > > does this mean that we need to keep the checkpoints compatible across minor > versions? Or can we say, that the minor version upgrades are only > guaranteed with canonical savepoints? > > My concern is especially if we'd want to change layout of the checkpoint. > > D. > > > > On Wed, Dec 29, 2021 at 5:19 AM Yu Li wrote: > > > Thanks for the proposal Piotr! Overall I'm +1 for the idea, and below are > > my two cents: > > > > 1. How about adding a "Term Definition" section and clarify what "native > > format" (the "native" data persistence format of the current state > backend) > > and "canonical format" (the "uniform" format that supports switching > state > > backends) means? > > > > 2. IIUC, currently the FLIP proposes to only support incremental > savepoint > > with native format, and there's no plan to add such support for canonical > > format, right? If so, how about writing this down explicitly in the FLIP > > doc, maybe in a "Limitations" section, plus the fact that > > `HashMapStateBackend` cannot support incremental savepoint before > FLIP-151 > > is done? (side note: @Roman just a kindly reminder, that please take > > FLIP-203 into account when implementing FLIP-151) > > > > 3. How about changing the description of "the default configuration of > the > > checkpoints will be used to determine whether the savepoint should be > > incremental or not" to something like "the `state.backend.incremental` > > setting now denotes the type of native format snapshot and will take > effect > > for both checkpoint and savepoint (with native type)", to prevent concept > > confusion between checkpoint and savepoint? > > > > 4. How about putting the notes of behavior change (the default type of > > savepoint will be changed to `native` in the future, and by then the > taken > > savepoint cannot be used to switch state backends by default) to a more > > obvious place, for example moving from the "CLI" section to the > > "Compatibility" section? (although it will only happen in 1.16 release > > based on the proposed plan) > > > > And all above suggestions apply for our user-facing document after the > FLIP > > is (partially or completely, accordingly) done, if taken (smile). > > > > Best Regards, > > Yu > > > > > > On Tue, 21 Dec 2021 at 22:23, Seth Wiesman wrote: > > > > > >> AFAIK state schema evolution should work both for native and > canonical > > > >> savepoints. > > > > > > Schema evolution does technically work for both formats, it happens > after > > > the code paths have been unified, but the community has up until this > > point > > > considered that an unsupported feature. From my perspective making this > > > supported could be as simple as adding test coverage but that's an > active > > > decision we'd need to make. > > > > > > On Tue, Dec 21, 2021 at 7:43 AM Piotr Nowojski > > > wrote: > > > > > > > Hi Konstantin, > > > > > > > > > In this context: will the native format support state schema > > evolution? > > > > If > > > > > not, I am not sure, we can let the format default to native. > > > > > > > > AFAIK state schema evolution should work both for native and > canonical > > > > savepoints. > > > > > > > > Regarding what is/will be supported we will document as part of this > > > > FLIP-203. But it's not as simple as just the difference between > native > > > and > > > > canonical formats. > > > > > > > > Best, Piotrek > > > > > > > > pon., 20 gru 2021 o 14:28 Konstantin Knauf > > > napisał(a): > > > > > > > > > Hi Piotr, > > > > > > > > > > Thanks a lot for starting the discussion. Big +1. > > > > > > > > > > In my understanding, this FLIP introduces the snapshot format as a > > > > *really* > > > > >
Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime
I've now read over the discussion on the ticket, and I am personally not in favor of adding this functionality to Flink via the REST API or Web UI. I believe that changing the logging configuration via the existing configuration files (log4j or logback) is good enough, to justify not increasing the scope of Flink in that direction. As you specifically mention YARN: doesn't Cloudera's Hadoop platform, for example, offer means to manage the configuration files for all worker nodes from a central configuration management system? It overall feels like we are trying to solve a problem in Apache Flink that is already solved in its ecosystem and increases the scope of the project without adding core value. I also expect that over time the exposed logging configuration options would become more and more complex. I am curious to hear what others think. On Tue, Jan 11, 2022 at 10:34 AM Chesnay Schepler wrote: > Reloading the config from the filesystem is already enabled by default; > that was one of the things that made us switch to Log4j 2. > > The core point of contention w.r.t. this topic is whether having the > admin ssh into the machine is too inconvenient. > > Personally I still think that the the current capabilities are > sufficient, and I do not want us to rely on internals of the logging > backends in production code. > > On 10/01/2022 17:26, Konstantin Knauf wrote: > > Thank you for starting the discussion. Being able to change the logging > > level at runtime is very valuable in my experience. > > > > Instead of introducing our own API (and eventually even persistence), > could > > we just periodically reload the log4j or logback configuration from the > > environment/filesystem? I only quickly googled the topic and [1,2] > suggest > > that this might be possible? > > > > [1] https://stackoverflow.com/a/16216956/6422562? > > [2] https://logback.qos.ch/manual/configuration.html#autoScan > > > > > > > > > > > > On Mon, Jan 10, 2022 at 5:10 PM Wenhao Ji > wrote: > > > >> Hi everyone, > >> > >> Hope you enjoyed the Holiday Season. > >> > >> I would like to start the discussion on the improvement purpose > >> FLIP-210 [1] which aims to provide a way to change log levels at > >> runtime to simplify issues and bugs detection as reported in the > >> ticket FLINK-16478 [2]. > >> Firstly, thanks Xingxing Di and xiaodao for their previous effort. The > >> FLIP I drafted is largely influenced by their previous designs [3][4]. > >> Although we have reached some agreements under the jira comments about > >> the scope of this feature, we still have the following questions > >> listed below ready to be discussed in this thread. > >> > >> ## Question 1 > >> > >>> Creating as custom DSL and implementing it for several logging backend > >> sounds like quite a maintenance burden. Extensions to the DSL, and > >> supported backends, could become quite an effort. (by Chesnay Schepler) > >> > >> I tried to design the API of the logging backend to stay away from the > >> details of implementations but I did not find any slf4j-specific API > >> that is available to change the log level of a logger. So what I did > >> is to introduce another kind of abstraction on top of the slf4j / > >> log4j / logback so that we will not depend on the logging provider's > >> api directly. It will be convenient for us to adopt any other logging > >> providers. Please see the "Logging Abstraction" section. > >> > >> ## Question 2 > >> > >>> Do we know whether other systems support this kind of feature? If yes, > >> how do they solve it for different logging backends? (by Till Rohrmann) > >> > >> I investigated several Java frameworks including Spark, Storm, and > >> Spring Boot. Here is what I found. > >> Spark & Storm directly depend on the log4j implementations, which > >> means they do not support any other slf4j implementation at all. They > >> simply call the log4j api directly. (see SparkContext.scala#L381 [5], > >> Utils.scala#L2443 [6] in Spark, and LogConfigManager.java#L144 [7] in > >> Storm). They are pretty different from what Flink provides. > >> However, I found Spring Boot has implemented what we are interested > >> in. Just as Flink, Spring boot also supports many slf4j > >> implementations. Users are not limited to log4j. They have the ability > >> to declare different logging frameworks by importing certain > >> dependencies. After that spring will decid
Re: [DISCUSS] FLIP-210: Change logging level dynamically at runtime
Thank you for starting the discussion. Being able to change the logging level at runtime is very valuable in my experience. Instead of introducing our own API (and eventually even persistence), could we just periodically reload the log4j or logback configuration from the environment/filesystem? I only quickly googled the topic and [1,2] suggest that this might be possible? [1] https://stackoverflow.com/a/16216956/6422562? [2] https://logback.qos.ch/manual/configuration.html#autoScan On Mon, Jan 10, 2022 at 5:10 PM Wenhao Ji wrote: > Hi everyone, > > Hope you enjoyed the Holiday Season. > > I would like to start the discussion on the improvement purpose > FLIP-210 [1] which aims to provide a way to change log levels at > runtime to simplify issues and bugs detection as reported in the > ticket FLINK-16478 [2]. > Firstly, thanks Xingxing Di and xiaodao for their previous effort. The > FLIP I drafted is largely influenced by their previous designs [3][4]. > Although we have reached some agreements under the jira comments about > the scope of this feature, we still have the following questions > listed below ready to be discussed in this thread. > > ## Question 1 > > > Creating as custom DSL and implementing it for several logging backend > sounds like quite a maintenance burden. Extensions to the DSL, and > supported backends, could become quite an effort. (by Chesnay Schepler) > > I tried to design the API of the logging backend to stay away from the > details of implementations but I did not find any slf4j-specific API > that is available to change the log level of a logger. So what I did > is to introduce another kind of abstraction on top of the slf4j / > log4j / logback so that we will not depend on the logging provider's > api directly. It will be convenient for us to adopt any other logging > providers. Please see the "Logging Abstraction" section. > > ## Question 2 > > > Do we know whether other systems support this kind of feature? If yes, > how do they solve it for different logging backends? (by Till Rohrmann) > > I investigated several Java frameworks including Spark, Storm, and > Spring Boot. Here is what I found. > Spark & Storm directly depend on the log4j implementations, which > means they do not support any other slf4j implementation at all. They > simply call the log4j api directly. (see SparkContext.scala#L381 [5], > Utils.scala#L2443 [6] in Spark, and LogConfigManager.java#L144 [7] in > Storm). They are pretty different from what Flink provides. > However, I found Spring Boot has implemented what we are interested > in. Just as Flink, Spring boot also supports many slf4j > implementations. Users are not limited to log4j. They have the ability > to declare different logging frameworks by importing certain > dependencies. After that spring will decide the activated one by > scanning its classpath and context. (see LoggingSystem.java#L164 [8] > and LoggersEndpoint.java#L99 [9]) > > ## Question 3 > > Besides the questions raised in the jira comments, I also find another > thing that has not been discussed. Considering this feature as an MVP, > do we need to introduce a HighAvailabilityService to store the log > settings so that they can be synced to newly-joined task managers and > also job manager followers for consistency? This issue is included in > the "Limitations" section in the flip. > > Finally, thanks for your time for joining this discussion and > reviewing this FLIP. I would appreciate it if you could have any > comments or suggestions on this. > > > [1]: > https://cwiki.apache.org/confluence/display/FLINK/FLIP-210%3A+Change+logging+level+dynamically+at+runtime > [2]: https://issues.apache.org/jira/browse/FLINK-16478 > [3]: > https://docs.google.com/document/d/1Q02VSSBzlZaZzvxuChIo1uinw8KDQsyTZUut6_IDErY > [4]: > https://docs.google.com/document/d/19AyuTHeERP6JKmtHYnCdBw29LnZpRkbTS7K12q4OfbA > [5]: > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/SparkContext.scala#L381 > [6]: > https://github.com/apache/spark/blob/11596b3b17b5e0f54e104cd49b1397c33c34719d/core/src/main/scala/org/apache/spark/util/Utils.scala#L2433 > [7]: > https://github.com/apache/storm/blob/3f96c249cbc17ce062491bfbb39d484e241ab168/storm-client/src/jvm/org/apache/storm/daemon/worker/LogConfigManager.java#L144 > [8]: > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/logging/LoggingSystem.java#L164 > [9]: > https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator/src/main/java/org/springframework/boot/actuate/logging/LoggersEndpoint.java#L99 > > Thanks, > Wenhao > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Add libatomic1 to flink image
Moving dev@ to bcc, adding user@ Hi Julius, the recommended approach would be to build your own Docker images from the official images along the lines of FROM apache/flink:1.14.3 RUN apt-get install -y libatomic1 Cheers, Konstantin On Fri, Mar 11, 2022 at 11:07 AM Almeida, Julius wrote: > Hi Developers, > > I wish to add a new library to existing flink docker image since it’s a > dependency requirement in my processor. > Is it possible to add? > > apt-get install libatomic1 > > Thanks, > Julius > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Release 1.14.4, release candidate #1
+1 (binding) - checked signatures and checksums of binaries OK - build from sources - checked for suspicious changes in pom.xml between 1.14.3 and 1.14.4 - ran examples/python/table/batch/word_count.py in a local mini cluster - ran examples/python/table/batch/word_count.py against standalone session cluster On Wed, Mar 9, 2022 at 3:07 PM Timo Walther wrote: > +1 (binding) > > - I scanned the commit diff and affected files. > - I could not find major API changes or otherwise problematic changes. > - The biggest changes I could spot were around Kubernetes (see > FLINK-20830). > > Thanks for taking care of this Konstantin. > > Timo > > > Am 02.03.22 um 10:04 schrieb Yun Tang: > > +1 non-binding > > > > - Checked the signatures for pending release artifacts. > > - Download the pre-built flink-dist of both scala_2.11 and scala_2.12 > versions and run them locally with the StateMachine example. > > - Reviewed the flink-web PR > > > > Best > > Yun Tang > > > > From: Seth Wiesman > > Sent: Monday, February 28, 2022 23:02 > > To: dev > > Subject: Re: [VOTE] Release 1.14.4, release candidate #1 > > > > +1 non-binding > > > > - built from source > > - checked hashes and signatures > > - started locally and deployed wordcount / stopped with savepoint / > > restarted > > - reviewed announcement post > > > > Thanks for managing the release! > > > > Seth > > > > On Fri, Feb 25, 2022 at 7:30 AM Konstantin Knauf > wrote: > > > >> Hi everyone, > >> > >> Please review and vote on the release candidate #1 for the version > 1.14.4, > >> as follows: > >> [ ] +1, Approve the release > >> [ ] -1, Do not approve the release (please provide specific comments) > >> > >> The complete staging area is available for your review, which > includes: > >> * JIRA release notes [1], > >> * the official Apache source release and binary convenience releases to > be > >> deployed to dist.apache.org [2], which are signed with the key with > >> fingerprint 8C3FB007FE60 DEFA [3], > >> * all artifacts to be deployed to the Maven Central Repository [4], > >> * source code tag "release-1.14.4-rc1" [5], > >> * website pull request listing the new release and adding announcement > blog > >> post [6]. > >> > >> The vote will be open for at least 72 hours. It is adopted by majority > >> approval, with at least 3 PMC affirmative votes. > >> > >> Thanks, > >> Konstantin > >> > >> [1] > >> > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231 > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.4-rc1/ > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1487/ > >> [5] https://github.com/apache/flink/tree/release-1.14.4-rc1 > >> [6] https://github.com/apache/flink-web/pull/510 > >> > >> -- > >> > >> Konstantin Knauf > >> > >> https://twitter.com/snntrable > >> > >> https://github.com/knaufk > >> > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[RESULT] [VOTE] Release 1.14.4, release candidate #1
Hi everyone, I am pleased to announce that we have unanimously approved this release candidate: There are 5 approving votes, 3 of which are binding: - Seth Wiesman (non-binding) - Yun Tang (non-binding) - Yun Gao (binding) - Timo Walther (binding) - Konstantin Knauf (binding) There are no disapproving votes. Thank you for verifying the release candidate. I will now proceed to finalize the release and announce it once everything is published. Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Preview release for Flink Kubernetes Operator
Hi everyone, can we mark all the APIs as experimental/alpha so that it is clear that these can be broken in future releases for now? I think this would be very important given the early stage of the project. We want to be able to address shortcomings without worrying too much about backwards compatibility at this stage, I believe. Cheers, Konstantin On Sun, Mar 13, 2022 at 7:48 AM Yang Wang wrote: > Thanks Gyula for starting this discussion. > > Given that the core functionality is closing to stable, I am in favor of > having the MVP release at the end of March. > The first release will help us to collect more feedbacks from the users. > Also it is a good chance to let the users know that the community is trying > to maintain an official Kubernetes operator :) > I hope that the companies could build their own production streaming > platform on top of the flink-kubernetes-operator in the future. > > FYI: @Wenjun Min is still working hard on supporting the Session Job in > Flink Kubernetes operator, It will be great if we could include it in the > first release. > And I believe we have enough time. > > Moreover, I agree with you that we need to invest more time in the > documentation, e2e tests, helm install optimization, logging, > etc. before the release. > > > Best, > Yang > > > Gyula Fóra 于2022年3月12日周六 01:10写道： > > > Hi Team! > > > > I would like to discuss the timeline for the initial preview/milestone > > release of the flink-kubernetes-operator > > <https://github.com/apache/flink-kubernetes-operator> project. > > > > The last few weeks we have been working very hard with the community to > > stabilize the initial feature set and I think we have made great > progress. > > While we are still far from a production ready-state, a preview release > > will give us the opportunity to reach more people and gather much needed > > input to take this project to the next level. > > > > There are still a couple missing features that we need to iron out and we > > need to make sure we have proper documentation but after that I think it > > would be a good time for the preview release. > > > > I propose to aim for the first release candidate around the 25-27th of > > March after which we should dedicate a few days for some extensive > testing > > and bugfixing. > > > > What do you think? > > > > Gyula > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] New PMC member: Yuan Mei
Congratulations, Yuan! On Mon, Mar 14, 2022 at 11:29 AM Jing Zhang wrote: > Congratulations, Yuan! > > Best, > Jing Zhang > > Jing Ge 于2022年3月14日周一 18:15写道： > > > Congrats! Very well deserved! > > > > Best, > > Jing > > > > On Mon, Mar 14, 2022 at 10:34 AM Piotr Nowojski > > wrote: > > > > > Congratulations :) > > > > > > pon., 14 mar 2022 o 09:59 Yun Tang napisał(a): > > > > > > > Congratulations, Yuan! > > > > > > > > Best, > > > > Yun Tang > > > > > > > > From: Zakelly Lan > > > > Sent: Monday, March 14, 2022 16:55 > > > > To: dev@flink.apache.org > > > > Subject: Re: [ANNOUNCE] New PMC member: Yuan Mei > > > > > > > > Congratulations, Yuan! > > > > > > > > Best, > > > > Zakelly > > > > > > > > On Mon, Mar 14, 2022 at 4:49 PM Johannes Moser > > > wrote: > > > > > > > > > Congrats Yuan. > > > > > > > > > > > On 14.03.2022, at 09:45, Arvid Heise wrote: > > > > > > > > > > > > Congratulations and well deserved! > > > > > > > > > > > > On Mon, Mar 14, 2022 at 9:30 AM Matthias Pohl > > > > > wrote: > > > > > > > > > > > >> Congratulations, Yuan. > > > > > >> > > > > > >> On Mon, Mar 14, 2022 at 9:25 AM Shuo Cheng > > > > wrote: > > > > > >> > > > > > >>> Congratulations, Yuan! > > > > > >>> > > > > > >>> On Mon, Mar 14, 2022 at 4:22 PM Anton Kalashnikov < > > > > kaa@yandex.com> > > > > > >>> wrote: > > > > > >>> > > > > > >>>> Congratulations, Yuan! > > > > > >>>> > > > > > >>>> -- > > > > > >>>> > > > > > >>>> Best regards, > > > > > >>>> Anton Kalashnikov > > > > > >>>> > > > > > >>>> 14.03.2022 09:13, Leonard Xu пишет: > > > > > >>>>> Congratulations Yuan! > > > > > >>>>> > > > > > >>>>> Best, > > > > > >>>>> Leonard > > > > > >>>>> > > > > > >>>>>> 2022年3月14日 下午4:09，Yangze Guo 写道： > > > > > >>>>>> > > > > > >>>>>> Congratulations! > > > > > >>>>>> > > > > > >>>>>> Best, > > > > > >>>>>> Yangze Guo > > > > > >>>>>> > > > > > >>>>>> On Mon, Mar 14, 2022 at 4:08 PM Martijn Visser < > > > > > >>>> martijnvis...@apache.org> wrote: > > > > > >>>>>>> Congratulations Yuan! > > > > > >>>>>>> > > > > > >>>>>>> On Mon, 14 Mar 2022 at 09:02, Yu Li > > wrote: > > > > > >>>>>>> > > > > > >>>>>>>> Hi all! > > > > > >>>>>>>> > > > > > >>>>>>>> I'm very happy to announce that Yuan Mei has joined the > > Flink > > > > PMC! > > > > > >>>>>>>> > > > > > >>>>>>>> Yuan is helping the community a lot with creating and > > > validating > > > > > >>>> releases, > > > > > >>>>>>>> contributing to FLIP discussions and good code > contributions > > > to > > > > > >> the > > > > > >>>>>>>> state backend and related components. > > > > > >>>>>>>> > > > > > >>>>>>>> Congratulations and welcome, Yuan! > > > > > >>>>>>>> > > > > > >>>>>>>> Best Regards, > > > > > >>>>>>>> Yu (On behalf of the Apache Flink PMC) > > > > > >>>>>>>> > > > > > >>>> -- > > > > > >>>> > > > > > >>>> Best regards, > > > > > >>>> Anton Kalashnikov > > > > > >>>> > > > > > >>>> > > > > > >> > > > > > > > > > > > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] Apache Flink 1.1.4.4 released
Hi Xingbo, you are totally right. Thank you for noticing. This also affected Flink 1.13.6, the other release I was recently managing. I simply skipped a step in the release guide. It should be fixed now. Could you double-check? Cheers, Konstantin On Wed, Mar 16, 2022 at 4:07 AM Xingbo Huang wrote: > Thanks a lot for being our release manager Konstantin and everyone who > contributed. I have a question about pyflink. I see that there are no > corresponding wheel packages uploaded on pypi, only the source package is > uploaded. Is there something wrong with building the wheel packages? > > Best, > Xingbo > > Leonard Xu 于2022年3月16日周三 01:02写道： > >> Thanks a lot for being our release manager Konstantin and everyone who >> involved! >> >> Best, >> Leonard >> >> 2022年3月15日 下午9:34，Martijn Visser 写道： >> >> Thank you Konstantin and everyone who contributed! >> >> >> -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[ANNOUNCE] Apache Flink 1.1.4.4 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.14.4, which is the third bugfix release for the Apache Flink 1.14 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2022/03/11/release-1.14.4.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Jark, I think you are raising a very good point. I think we need an application mode for SQL that would work along the lines of executing a SQL script (incl. init scripts) located in a particular directory in the Docker Image. Details to be discussed. Do you think Zeppelin/SQL CLI could work with such a mode for non-interactive queries (interactive queries would use a session cluster)? Best, Konstantin On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > Hi David, > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > jobs. So the session cluster is not a fit mode. > > Best, > Jark > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > Hi Jark, can you please elaborate about the current need of the per-job > > mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't > > these a natural fit for the session cluster? > > > > D. > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > Hi Konstantin, > > > > > > I'm not very familiar with the implementation of per-job mode and > > > application mode. > > > But is there any instruction for users abou how to migrate > platforms/jobs > > > to application mode? > > > IIUC, the biggest difference between the two modes is where the main() > > > method is executed. > > > However, SQL jobs are not jar applications and don't have the main() > > > method. > > > For example, SQL CLI submits SQL jobs by invoking > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > > mode? > > > > > > Best, > > > Jark > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > wrote: > > > > > > > Hi everyone, > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > these limitations of per-job mode on YARN. It seems that there is a > > > general > > > > agreement to deprecate per-job mode and to drop it once the > limitations > > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > > > Thanks again, > > > > > > > > Konstantin > > > > > > > > > > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > wrote: > > > > > > > > > Hi Yang, > > > > > > > > > > Thank you for the clarification. In general I think we will have > time > > > to > > > > > experiment with this until it will be removed totally and migrate > our > > > > > solution to use application mode. > > > > > > > > > > Regards, > > > > > F > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > Hi all, > > > > > > > > > > > > I remember the application mode was initially named "cluster > mode". > > > As > > > > a > > > > > > contrast, the per-job mode is the "client mode". > > > > > > So I believe application mode should cover all the > functionalities > > of > > > > > > per-job except where we are running the user main code. > > > > > > In the containerized or the Kubernetes world, the application > mode > > is > > > > > more > > > > > > native and easy to use since all the Flink and user > > > > > > jars are bundled in the image. I am also in favor of deprecating > > and > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > IIRC, the YARN application mode could ship user jars and > > dependencies > > > > via > > > > > > "yarn.ship-files" config option. The only > > > > > > limitation is that we could not ship and load the user > dependencies > > > > with > > > > > > user classloader, not the parent classloader. > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > directory > > > > > > automatically. > > > > > > > > > > > > > > >
Re: [DISCUSS] FLIP-212: Introduce Flink Kubernetes Operator
t; > > > > > > > > > > > > > > > On Tue, Feb 1, 2022 at 2:33 PM Márton > > > Balassi < > > > > > > > > > > > > > > > > > balassi.mar...@gmail.com> > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Hi team, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thank you for the great feedback, > > Thomas > > > has > > > > > > > > updated > > > > > > > > > > the > &g
Re: [DISCUSS] Release Flink 1.14.4
Hi Qingsheng, Hi everyone, Thanks for the update. Sounds good to move FLINK-26018 to Flink 1.14.5. Since FLINK-26018 is the only Critical bug left, I will create an RC 1 soon. Best, Konstantin On Fri, Feb 25, 2022 at 9:49 AM Qingsheng Ren wrote: > Thanks Konstantin for the effort on the release! > > For FLINK-26018 [1] I share the same opinion with Fabian that it looks > like we need a new feature to completely fix the issue. Considering we > don’t have enough capacity to resolve it quickly for now, is it OK to > postpone this issue to the next release? > > Best, > > Qingsheng Ren > > [1] https://issues.apache.org/jira/browse/FLINK-26018 > > > On Feb 11, 2022, at 5:29 PM, Konstantin Knauf wrote: > > > > Hi everyone, > > > > what do you think about a timely Flink 1.14.4 in order to release the fix > > for https://issues.apache.org/jira/browse/FLINK-25732. Currently, the > > landing page of Flink Web User Interface is not working when there are > > standby Jobmanagers. > > > > In addition, I propose to wait for > > https://issues.apache.org/jira/browse/FLINK-26018 to be resolved. > > > > I can volunteer as release manager. > > > > Cheers, > > > > Konstantin > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Release 1.13.6, release candidate #1
+1 (binding) - checked signatures and checksums of binaries OK - build from sources - run TopSpeedWindowing.jar in standalone application mode - ran examples/python/table/batch/word_count.py in a local mini cluster - ran examples/python/table/batch/word_count.py against standalone session cluster On Tue, Feb 15, 2022 at 4:53 PM Anton Kalashnikov wrote: > +1 (non-binding) > > - signatures OK > - checksums OK > - tag OK > - all artifacts OK > - PR OK > > run examples > > -- > > Best regards, > Anton Kalashnikov > > > 15.02.2022 09:30, Dawid Wysakowicz пишет: > > +1 (binding) > > > > > > - signatures OK > > - checksums OK > > - tag OK > > - PR looks good > > > > - built from sources > > > > - run example > > > > - checked dependency version changes from 1.13.5 and notice files > > > > Best, > > > > Dawid > > > > On 05/02/2022 21:06, Konstantin Knauf wrote: > >> Hi everyone, > >> > >> Please review and vote on the release candidate #1 for the version > >> 1.13.6, > >> as follows: > >> [ ] +1, Approve the release > >> [ ] -1, Do not approve the release (please provide specific comments) > >> > >> The complete staging area is available for your review, which > >> includes: > >> * JIRA release notes [1], > >> * the official Apache source release and binary convenience releases > >> to be > >> deployed to dist.apache.org [2], which are signed with the key with > >> fingerprint 8C3FB007FE60 DEFA [3], > >> * all artifacts to be deployed to the Maven Central Repository [4], > >> * source code tag "release-1.2.3-rc3" [5], > >> * website pull request listing the new release and adding > >> announcement blog > >> post [6]. > >> > >> The vote will be open for at least 72 hours. It is adopted by majority > >> approval, with at least 3 PMC affirmative votes. > >> > >> Thanks, > >> Konstantin > >> > >> [1] > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074 > >> > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.6-rc1/ > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1486/ > >> [5] https://github.com/apache/flink/tree/release-1.13.6-rc1 > >> [6] https://github.com/apache/flink-web/pull/505 > >> > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[RESULT] [VOTE] Release 1.13.6, release candidate #1
Hi everyone, I am pleased to announce that we have unanimously approved this release candidate: There are 4 approving votes, 3 of which are binding: - Chesnay Schepler (binding) - Dawid Wysakowicz (binding) - Anton Kalashnikov (non-binding) - Konstantin Knauf (binding) There are no disapproving votes. Thank you for verifying the release candidate. I will now (today or tomorrow) proceed to finalize the release and announce it once everything is published. Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Disable "Automated Checks / Review Progress" GitHub integration
+1 On Thu, Feb 17, 2022 at 1:11 PM Robert Metzger wrote: > Hi all, > > Some time ago, we added this "Automated Checks / Review Progress" [1] bot > to the Flink PRs. I'm not aware of anybody using it, and I'm also not sure > if it still works properly. > > Therefore, I propose to disable this bot. Please let me know if you > disagree, otherwise, I'll soon disable it. > > > Best, > Robert > > > [1]https://github.com/apache/flink/pull/18818#issuecomment-1042865516 > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[ANNOUNCE] Apache Flink 1.13.6 released
The Apache Flink community is very happy to announce the release of Apache Flink 1.13.6, which is the fifth bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for download at: https://flink.apache.org/downloads.html Please check out the release blog post for an overview of the improvements for this bugfix release: https://flink.apache.org/news/2022/02/09/release-1.13.6.html The full release notes are available in Jira: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074 We would like to thank all contributors of the Apache Flink community who made this release possible! Regards, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Plan to externalize connectors and versioning
+1 to the approach. I expect that we will encounter more questions and challenges as we go, but these are best discussed and addressed in the context of a specific connector like ElasticSearch. On Fri, Feb 18, 2022 at 2:54 PM Martijn Visser wrote: > Hi everyone, > > As a follow-up to earlier discussions [1] [2] to externalize the connectors > from the Flink repository, I would like to propose a plan to externalize > these connectors. The goal of this plan is to start with moving connectors > to its own repositories without introducing regressions for connector > developers. > > The plan is as follows: > > 1. A new repository is requested for a connector. > 2. The code for that connector is moved to its individual repository, > including the commit history > 3. Any first release made for a connector in an external connector > repository starts with major version 3, so 3.0.0. The reason for that is > that we want to decouple the Flink releases from a connector release. If we > would start with major version 2, it could cause some confusion because > people could think a Flink 2.0 has been released. This does mean that each > connector needs to have a compatibility matrix generated, stating which > version number of the connector is compatible with the correct Flink > version. > 4. The group id and artifact id for the connector will remain the same, > only the version is different. > 5. The connector dependencies on the Flink website are updated to point to > the newly released connector artifact. > 6. If a connector is moved, there is one release cycle where there will be > binary releases for that connector in both Flink core and from the > connector repository. This is to make Flink users who are upgrading > slightly easier. We will have to make a note in the release notes that a > connector has been moved and that a user should update any references from > the original connector artifact (from the Flink connector) to the new > connector artifact (from the external conenctor version) > > We propose to first try to execute this plan for the Elasticsearch > connector as follows: > > 1. We wait until the Flink 1.15 release branch is cut > 2. When that's done, the Elasticsearch code (including commit history) from > Flink's 1.15 release branch will be moved to the > flink-connector-elasticsearch main branch. > 3. When Flink 1.15 is released, we will also release an Elasticsearch > connector for the external connector repository with version 3.0.0. > 4. Bugfixes or improvements will be made first pointing to the external > connector repository and will be cherry-picked back to the release-1.15 > branch in the Flink core repository. > 5. The Elasticsearch code, test etc will be removed from the master branch > in the Flink core repository and dropped with Flink 1.16 > > Looking forward to your thoughts on this! > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > [1] https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm > [2] https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Plan to externalize connectors and versioning
> With regards to point 6, do you have an alternative? Would you prefer to > > remove a connector in a Flink patch release? > > > > Best regards, > > > > Martijn > > > > On Fri, 18 Feb 2022 at 16:17, Chesnay Schepler > wrote: > > > >> Why do you want to immediately do a release for the elasticsearch > >> connector? What does that provide us? > >> > >> I'd rather first have a fully working setup and integrated documentation > >> before thinking about releasing anything. > >> Once we have that we may be able to externalize all connectors within 1 > >> release cycle and do a clean switch; otherwise we end up with a bit of a > >> messy situation for users where some connectors use version scheme A and > >> others B. > >> > >> I also doubt the value of 6). They'll have to update the version anyway > >> (and discover at some point that the version scheme has changed), so I > >> don't see what this makes easier. > >> > >> On 18/02/2022 14:54, Martijn Visser wrote: > >>> Hi everyone, > >>> > >>> As a follow-up to earlier discussions [1] [2] to externalize the > >> connectors > >>> from the Flink repository, I would like to propose a plan to > externalize > >>> these connectors. The goal of this plan is to start with moving > >> connectors > >>> to its own repositories without introducing regressions for connector > >>> developers. > >>> > >>> The plan is as follows: > >>> > >>> 1. A new repository is requested for a connector. > >>> 2. The code for that connector is moved to its individual repository, > >>> including the commit history > >>> 3. Any first release made for a connector in an external connector > >>> repository starts with major version 3, so 3.0.0. The reason for that > is > >>> that we want to decouple the Flink releases from a connector release. > If > >> we > >>> would start with major version 2, it could cause some confusion because > >>> people could think a Flink 2.0 has been released. This does mean that > >> each > >>> connector needs to have a compatibility matrix generated, stating which > >>> version number of the connector is compatible with the correct Flink > >>> version. > >>> 4. The group id and artifact id for the connector will remain the same, > >>> only the version is different. > >>> 5. The connector dependencies on the Flink website are updated to point > >> to > >>> the newly released connector artifact. > >>> 6. If a connector is moved, there is one release cycle where there will > >> be > >>> binary releases for that connector in both Flink core and from the > >>> connector repository. This is to make Flink users who are upgrading > >>> slightly easier. We will have to make a note in the release notes that > a > >>> connector has been moved and that a user should update any references > >> from > >>> the original connector artifact (from the Flink connector) to the new > >>> connector artifact (from the external conenctor version) > >>> > >>> We propose to first try to execute this plan for the Elasticsearch > >>> connector as follows: > >>> > >>> 1. We wait until the Flink 1.15 release branch is cut > >>> 2. When that's done, the Elasticsearch code (including commit history) > >> from > >>> Flink's 1.15 release branch will be moved to the > >>> flink-connector-elasticsearch main branch. > >>> 3. When Flink 1.15 is released, we will also release an Elasticsearch > >>> connector for the external connector repository with version 3.0.0. > >>> 4. Bugfixes or improvements will be made first pointing to the external > >>> connector repository and will be cherry-picked back to the release-1.15 > >>> branch in the Flink core repository. > >>> 5. The Elasticsearch code, test etc will be removed from the master > >> branch > >>> in the Flink core repository and dropped with Flink 1.16 > >>> > >>> Looking forward to your thoughts on this! > >>> > >>> Best regards, > >>> > >>> Martijn Visser > >>> https://twitter.com/MartijnVisser82 > >>> > >>> [1]https://lists.apache.org/thread/bywh947r2f5hfocxq598zhyh06zhksrm > >>> [2]https://lists.apache.org/thread/bk9f91o6wk66zdh353j1n7sfshh262tr > >>> > >> > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Release Flink 1.14.4
Hi everyone, We still have two more critical issues for Flink 1.14. I'd like to understand if it makes sense to push one of these to the next release. https://issues.apache.org/jira/browse/FLINK-24607 @Becket: this seems to be missing the backport, correct? https://issues.apache.org/jira/browse/FLINK-26018 @Qingsheng, Fabian, Piotr: I saw that Piotr and Fabian raised some general concerns about the solution. Does it make sense to wait for the resolution of this ticket for Flink 1.14.4? Thanks, Konstantin On Sat, Feb 12, 2022 at 12:42 PM Roman Khachatryan wrote: > Hi, > > +1 for the proposal. > > Thanks for volunteering Konstantin! > > Regards, > Roman > > > > On Fri, Feb 11, 2022 at 3:00 PM Till Rohrmann > wrote: > > > > +1 for a 1.14.4 release. The release-1.14 branch already includes 36 > fixed > > issues, some of them quite severe. > > > > Cheers, > > Till > > > > On Fri, Feb 11, 2022 at 1:23 PM Martijn Visser > > wrote: > > > > > Hi Konstantin, > > > > > > Thanks for opening the discussion. I think FLINK-25732 does indeed > warrant > > > a speedy Flink 1.14.4 release. I would indeed also like to include > > > FLINK-26018. > > > > > > Best regards, > > > > > > Martijn > > > > > > Op vr 11 feb. 2022 om 10:29 schreef Konstantin Knauf < > kna...@apache.org> > > > > > > > Hi everyone, > > > > > > > > what do you think about a timely Flink 1.14.4 in order to release > the fix > > > > for https://issues.apache.org/jira/browse/FLINK-25732. Currently, > the > > > > landing page of Flink Web User Interface is not working when there > are > > > > standby Jobmanagers. > > > > > > > > In addition, I propose to wait for > > > > https://issues.apache.org/jira/browse/FLINK-26018 to be resolved. > > > > > > > > I can volunteer as release manager. > > > > > > > > Cheers, > > > > > > > > Konstantin > > > > > > > > -- > > > > > > > > Konstantin Knauf > > > > > > > > https://twitter.com/snntrable > > > > > > > > https://github.com/knaufk > > > > > > > -- > > > > > > Martijn Visser | Product Manager > > > > > > mart...@ververica.com > > > > > > <https://www.ververica.com/> > > > > > > > > > Follow us @VervericaData > > > > > > -- > > > > > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > > > Conference > > > > > > Stream Processing | Event Driven | Real Time > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Enable scala formatting check
+1 I've never written any Scala in Flink, but this makes a lot of sense to me. Converging on a smaller set of tools and simplifying the build is always a good idea and the Community already concluded before that spotless is generally a good approach. On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani wrote: > Hi all, > > I want to propose to enable the spotless scalafmt integration and remove > the scalastyle plugin. > > From an initial analysis, scalafmt can do everything scalastyle can do, and > the integration with spotless looks easy to enable: > https://github.com/diffplug/spotless/tree/main/plugin-maven#scala. The > scalafmt conf file gets picked up automatically from every IDE, and it can > be heavily tuned. > > This way we can unify the formatting and integrate with our CI without any > additional configurations. And we won't need scalastyle anymore, as > scalafmt will take care of the checks: > > * mvn spotless:check will check both java and scala > * mvn spotless:apply will format both java and scala > > WDYT? > > FG > > > > -- > > Francesco Guardiani | Software Engineer > > france...@ververica.com > > > <https://www.ververica.com/> > > Follow us @VervericaData > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > > Ververica GmbH > > Registered at Amtsgericht Charlottenburg: HRB 158244 B > > Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, > Jinwei (Kevin) Zhang > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser
Congratulations, Martijn. Well deserved! On Fri, Mar 4, 2022 at 8:26 AM Yun Gao wrote: > Congratulations Martijn! > > Best, > Yun Gao > > > --Original Mail -- > Sender:Roman Khachatryan > Send Date:Fri Mar 4 15:22:08 2022 > Recipients:dev > Subject:Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser > Congratulations Martijn! > > Regards, > Roman > > On Fri, Mar 4, 2022 at 8:09 AM Sergey Nuyanzin > wrote: > > > > Congratulations Martijn! > > > > On Fri, Mar 4, 2022 at 8:07 AM David Morávek wrote: > > > > > Congratulations Martijn, well deserved! > > > > > > Best, > > > D. > > > > > > On Fri, Mar 4, 2022 at 7:25 AM Jiangang Liu > > > > wrote: > > > > > > > Congratulations Martijn! > > > > > > > > Best > > > > Liu Jiangang > > > > > > > > Lijie Wang 于2022年3月4日周五 14:00写道： > > > > > > > > > Congratulations Martijn! > > > > > > > > > > Best, > > > > > Lijie > > > > > > > > > > Jingsong Li 于2022年3月4日周五 13:42写道： > > > > > > > > > > > Congratulations Martijn! > > > > > > > > > > > > Best, > > > > > > Jingsong > > > > > > > > > > > > On Fri, Mar 4, 2022 at 1:09 PM Yang Wang > > > > wrote: > > > > > > > > > > > > > > Congratulations Martijn! > > > > > > > > > > > > > > Best, > > > > > > > Yang > > > > > > > > > > > > > > Yangze Guo 于2022年3月4日周五 11:33写道： > > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > Best, > > > > > > > > Yangze Guo > > > > > > > > > > > > > > > > On Fri, Mar 4, 2022 at 11:23 AM Lincoln Lee < > > > > lincoln.8...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > Congratulations Martijn! > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Lincoln Lee > > > > > > > > > > > > > > > > > > > > > > > > > > > Yu Li 于2022年3月4日周五 11:09写道： > > > > > > > > > > > > > > > > > > > Congratulations! > > > > > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > > > Yu > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Fri, 4 Mar 2022 at 10:31, Zhipeng Zhang < > > > > > > zhangzhipe...@gmail.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Congratulations Martijn! > > > > > > > > > > > > > > > > > > > > > > Qingsheng Ren 于2022年3月4日周五 > 10:14写道： > > > > > > > > > > > > > > > > > > > > > > > Congratulations Martijn! > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > > > > > > > > > > > Qingsheng Ren > > > > > > > > > > > > > > > > > > > > > > > > > On Mar 4, 2022, at 9:56 AM, Leonard Xu < > > > > xbjt...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Congratulations and well deserved Martjin ! > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > Leonard > > > > > > > > > > > > > > > > > > > > > > > > > >> 2022年3月4日 上午7:55，Austin Cawley-Edwards < > > > > > > austin.caw...@gmail.com > > > > > > > > > > > > > > > > > > > 写道： > > > > > > > > > > > > >> > > > > > > > > > > > > >> Congrats Martijn! > > > > > > > > > > > > >> > > > > > > > > > > > > >> On Thu, Mar 3, 2022 at 10:50 AM Robert Metzger < > > > > > > > > rmetz...@apache.org > > > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > >> > > > > > > > > > > > > >>> Hi everyone, > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> On behalf of the PMC, I'm very happy to announce > > > > Martijn > > > > > > > > Visser as > > > > > > > > > > a > > > > > > > > > > > > new > > > > > > > > > > > > >>> Flink committer. > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> Martijn is a very active Flink community member, > > > > driving > > > > > a > > > > > > lot > > > > > > > > of > > > > > > > > > > > > efforts > > > > > > > > > > > > >>> on the dev@flink mailing list. He also pushes > > > projects > > > > > > such as > > > > > > > > > > > > replacing > > > > > > > > > > > > >>> Google Analytics with Matomo, so that we can > generate > > > > our > > > > > > web > > > > > > > > > > > analytics > > > > > > > > > > > > >>> within the Apache Software Foundation. > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> Please join me in congratulating Martijn for > > > becoming a > > > > > > Flink > > > > > > > > > > > > committer! > > > > > > > > > > > > >>> > > > > > > > > > > > > >>> Cheers, > > > > > > > > > > > > >>> Robert > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > best, > > > > > > > > > > > Zhipeng > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > Best regards, > > Sergey -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] New Apache Flink Committer - David Morávek
Congratulations, David! Well deserved. On Fri, Mar 4, 2022 at 1:20 PM Gabor Somogyi wrote: > Congrats David! > > On Fri, Mar 4, 2022 at 12:35 PM Robert Metzger > wrote: > > > Hi everyone, > > > > On behalf of the PMC, I'm very happy to announce David Morávek as a new > > Flink committer. > > > > His first contributions to Flink date back to 2019. He has been > > increasingly active with reviews and driving major initiatives in the > > community. David brings valuable experience from being a committer in the > > Apache Beam project to Flink. > > > > > > Please join me in congratulating David for becoming a Flink committer! > > > > Cheers, > > Robert > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[VOTE] Release 1.14.4, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the version 1.14.4, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 8C3FB007FE60 DEFA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.14.4-rc1" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Konstantin [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351231 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.14.4-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1487/ [5] https://github.com/apache/flink/tree/release-1.14.4-rc1 [6] https://github.com/apache/flink-web/pull/510 -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Structure of the Flink Documentation (Languages & APIs)
Hi Jark, Thank you for joining the discussion. I am totally fine with the "Language Tabs". Would be good to hear what the PyFlink community thinks, because the Python Documentation is the biggest exception to the "Language Tabs" approach. @Jark Wu : Do you agree that SQL should be separate from Table API/DataStream API in either way? I think SQL does neither fit into the "Language Tabs" nor "Language First". Cheers, Konstantin On Wed, Mar 23, 2022 at 9:17 AM Jark Wu wrote: > Hi Konstantin, > > Thanks for starting this discussion. > > From my perspective, I prefer the "Language Tabs" approach. > But maybe we can improve the tabs to move to the sidebar or top menu, > which allows users to first decide on their language and then the API. > IMO, programming languages are just like spoken languages which can be > picked in the sidebar. > What I want to avoid is the duplicate docs and in-complete features in a > specific language. > "Language First" may confuse users about what is and where to find the > complete features provided by flink. > > For example, there are a lot of duplications in the "Window" pages[1] and > "Python Window" pages[2]. > And users can't have a complete overview of Flink's window mechanism from > the Python API part. > Users have to go through the Java/Scala DataStream API first to build the > overall knowledge, > and then to read the Python API part. > > > * Second, most of the Flink Documentation currently is using a "Language > Tabs" approach, but this might become obsolete in the long-term anyway as > we move more and more in a Scala-free direction. > > The Scala-free direction means users can pick arbitrary Scala versions, not > drop the Scala API. > So the "Language Tabs" is still necessary and helpful for switching > languages. > > Best, > Jark > > [1]: > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/datastream/operators/windows/ > [2]: > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/ > > > > > > > > On Tue, 22 Mar 2022 at 21:40, Konstantin Knauf wrote: > > > Hi everyone, > > > > I would like to discuss a particular aspect of our documentation: the > > top-level structure with respect to languages and APIs. The current > > structure is inconsistent and the direction is unclear to me, which makes > > it hard for me to contribute gradual improvements. > > > > Currently, the Python documentation has its own independent branch in the > > documentation [1]. In the rest of the documentation, Python is sometimes > > included like in this Table API page [2] and sometimes ignored like on > the > > project setup pages [3]. Scala and Java on the other hand are always > > documented in parallel next to each other in tabs. > > > > The way I see it, most parts (application development, connectors, > getting > > started, project setup) of our documentation have two primary dimensions: > > API (DataStream, Table API), Language (Python, Java, Scala) > > > > In addition, there is SQL, for which the language is only a minor factor > > (UDFs), but which generally requires a different structure (different > > audience, different tools). On the other hand, SQL and Table API have > some > > conceptual overlap, whereas I doubt these concepts are of big interest > > to SQL users. So, to me SQL should be treated separately in any case with > > links to the Table API documentation for some concepts. > > > > I think, in general, both approaches can work: > > > > > > *Option 1: "Language Tabs"* > > Application Development > > > DataStream API (Java, Scala, Python) > > > Table API (Java, Scala, Python) > > > SQL > > > > > > *Option 2: "Language First" * > > Java Development Guide > > > Getting Started > > > DataStream API > > > Table API > > Python Development Guide > > > Getting Started > > > Datastream API > > > Table API > > SQL Development Guide > > > > I don't have a strong opinion on this, but tend towards "Language First". > > > > * First, I assume, users actually first decide on their language/tools of > > choice and then move on to the API. > > > > * Second, most of the Flink Documentation currently is using a "Language > > Tabs" approach, but this might become obsolete in the long-term anyway as > > we move more and more in a Scala-free direction. > > > > For the conne
Re: [DISCUSS] Structure of the Flink Documentation (Languages & APIs)
Hi Dian, Thank you for sharing your thoughts. What do you propose going forward? I am not sure I got this from your email. Best, Konstantin On Wed, Mar 23, 2022 at 10:03 AM Dian Fu wrote: > Hi Konstantin, > > Thanks a lot for bringing up this discussion. > > Currently, the Python documentation is more like a mixture of Option 1 and > Option 2. It contains two parts: > 1) The first part is the independent page [1] which could be seen as the > main entrypoint for Python users. > 2) The second part is the Python tabs which are among the DataStream API / > Table API pages. > > The motivation to provide an independent page for Python documentation is > as follows: > 1) We are trying to create a Pythonic documentation for Python users (we > are still far away from that and I have received much feedback saying that > the Python documentation and API is too Java-like). However, to avoid > duplication, it will link to the DataStream API / Table API pages when > necessary instead of copying content. There are indeed exceptions, e.g. the > window example given by Jark, that's because it only provides a very > limited window support in Python DataStream API at present and to give > Python users a complete picture of what they can do in Python DataStream > API, we have added a dedicated page. We are trying to finalize the window > support in 1.16 [2] and remove the duplicate documentation. > 2) There are some kinds of documentations which are only applicable for > Python language, e.g. dependency management[2], conversion between Table > and Pandas DataFrame [3], etc. Providing an independent page helps to > provide a place to hold all these kinds of documentation together. > > Regarding Option 1: "Language Tabs", this makes it hard to create Pythonic > documentation for Python users. > Regarding Option 2: "Language First", it may mean a lot of duplications. > Currently, there are a lot of descriptions in the DataStream API / Table > API pages which are shared between Java/Scala/Python. > > > In the rest of the documentation, Python is sometimes > > included like in this Table API page [2] and sometimes ignored like on > the > > project setup pages [3]. > I agree that this is something that we need to improve. > > Regards, > Dian > > [1] > > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/overview/ > [2] https://issues.apache.org/jira/browse/FLINK-26477 > [2] > > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/dependency_management/ > [3] > > https://nightlies.apache.org/flink/flink-docs-stable/docs/dev/python/table/conversion_of_pandas/ > > On Wed, Mar 23, 2022 at 4:17 PM Jark Wu wrote: > > > Hi Konstantin, > > > > Thanks for starting this discussion. > > > > From my perspective, I prefer the "Language Tabs" approach. > > But maybe we can improve the tabs to move to the sidebar or top menu, > > which allows users to first decide on their language and then the API. > > IMO, programming languages are just like spoken languages which can be > > picked in the sidebar. > > What I want to avoid is the duplicate docs and in-complete features in a > > specific language. > > "Language First" may confuse users about what is and where to find the > > complete features provided by flink. > > > > For example, there are a lot of duplications in the "Window" pages[1] and > > "Python Window" pages[2]. > > And users can't have a complete overview of Flink's window mechanism from > > the Python API part. > > Users have to go through the Java/Scala DataStream API first to build the > > overall knowledge, > > and then to read the Python API part. > > > > > * Second, most of the Flink Documentation currently is using a > "Language > > Tabs" approach, but this might become obsolete in the long-term anyway as > > we move more and more in a Scala-free direction. > > > > The Scala-free direction means users can pick arbitrary Scala versions, > not > > drop the Scala API. > > So the "Language Tabs" is still necessary and helpful for switching > > languages. > > > > Best, > > Jark > > > > [1]: > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/datastream/operators/windows/ > > [2]: > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/ > > > > > > > > > > > > > > > > On Tue, 22 Mar 2022 at 21:40, Konstantin Knauf > wrote: > > > > > Hi everyone, > &g
Re: flink提交错误时报错信息不清晰
Hi there, Please use English on the dev@flink.apache.org mailing list. Images/attachements don't work on these mailing lists. You would have to host it somewhere else and link to it. If this was a question about using Apache Flink user@ (Englisch) or user-zh@ (Chinese) would be the correct mailing lists to address. Cheers, Konstantin On Thu, Mar 24, 2022 at 11:48 AM 丛鹏 wrote: > 嗨，我这边在使用flink1.12.2提交yarn-application的任务时发现一个问题， > 提交任务的flink官网的例子为 > ./bin/flink run-application -t yarn-application > ./examples/streaming/TopSpeedWindowing.jar > > 如果其中有部分写错的话 yarn-application 写成了 yarn-appliation的话 > 就会报错 > [image: image.png] > 这里我看了代码是CliFrontend 封装的 213行的配置集合 effectiveConfiguration > 有问题，导致DefaultClusterClientServiceLoader.java:83 判断进入报错 > [image: image.png] > 我觉得这里的报错信息的描述有问题 会造成误导，是使用者误认为是自己的HADOOP_CLASSPATH 环境有问题，希望可以得到大家的回复 > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Structure of the Flink Documentation (Languages & APIs)
Hi everyone, I would like to discuss a particular aspect of our documentation: the top-level structure with respect to languages and APIs. The current structure is inconsistent and the direction is unclear to me, which makes it hard for me to contribute gradual improvements. Currently, the Python documentation has its own independent branch in the documentation [1]. In the rest of the documentation, Python is sometimes included like in this Table API page [2] and sometimes ignored like on the project setup pages [3]. Scala and Java on the other hand are always documented in parallel next to each other in tabs. The way I see it, most parts (application development, connectors, getting started, project setup) of our documentation have two primary dimensions: API (DataStream, Table API), Language (Python, Java, Scala) In addition, there is SQL, for which the language is only a minor factor (UDFs), but which generally requires a different structure (different audience, different tools). On the other hand, SQL and Table API have some conceptual overlap, whereas I doubt these concepts are of big interest to SQL users. So, to me SQL should be treated separately in any case with links to the Table API documentation for some concepts. I think, in general, both approaches can work: *Option 1: "Language Tabs"* Application Development > DataStream API (Java, Scala, Python) > Table API (Java, Scala, Python) > SQL *Option 2: "Language First" * Java Development Guide > Getting Started > DataStream API > Table API Python Development Guide > Getting Started > Datastream API > Table API SQL Development Guide I don't have a strong opinion on this, but tend towards "Language First". * First, I assume, users actually first decide on their language/tools of choice and then move on to the API. * Second, most of the Flink Documentation currently is using a "Language Tabs" approach, but this might become obsolete in the long-term anyway as we move more and more in a Scala-free direction. For the connectors, I think, there is a good argument for "Language & API Embedded", because documenting every connector for each API and language separately would result in a lot of duplication. Here, I would go one step further then what we have right now and target Connectors -> Kafka (All APIs incl. SQL, All Languages) -> Kinesis (same) -> ... This also results in a quick overview for users about which connectors exist and plays well with our plan of externalizing connectors. For completeness & scope of the discussion: there are two outdated FLIPs on documentation (42, 60), which both have not been implemented, are partially contradicting each other and are generally out-of-date. I specifically don't intend to add another FLIP to this graveyard, but still reach a consensus on the high-level direction. What do you think? Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Deprecate NiFi connector
Hi everyone, +1. If or when someone starts an effort of migrating the new APIs in the future, the code will still be accessible even if it's already deleted in the latest versions. Thanks, Konstantin On Mon, Jan 31, 2022 at 11:46 AM Martijn Visser wrote: > Hi everyone, > > I would like to open up a vote to deprecate NiFi in Flink 1.15 and remove > it in the next version. I've previously mentioned that we were looking for > maintainers for NiFi, but so far none have come forward [1]. > > The vote will last for at least 72 hours, and will be accepted by a > consensus of active committers. > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > [1] https://lists.apache.org/thread/1vs3wmk66vsq6l4npjsfzltft4tz5tkq > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Releasing Flink 1.13.6
Hi everyone, I've created a release candidate, published all Maven artifacts and opened a PR for the website [1,2,3,4,5]. Python wheels are missing at this point as I am (like Thomas was) waiting for approval by Azure. I'll start the formal vote when the Python wheels are available. Of course, you can already start testing based on what is already there. Cheers, Konstantin [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.6-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1486/ [5] https://github.com/apache/flink/tree/release-1.13.6-rc1 [6] https://github.com/apache/flink-web/pull/505 On Tue, Feb 1, 2022 at 1:27 PM Martijn Visser wrote: > Hi Konstantin, > > Thanks for volunteering! There are no Blockers for 1.13.6 so you should be > good to go for creating a release candidate. > > Best regards, > > Martijn > > On Wed, 26 Jan 2022 at 08:32, Jing Zhang wrote: > > > +1 for releasing Flink 1.13.6 > > Thanks Martijn and Konstantin for driving this. > > > > Best, > > Jing Zhang > > > > David Morávek 于2022年1月25日周二 19:13写道： > > > > > Thanks for driving this Martijn, +1 for the release > > > > > > Also big thanks to Konstantin for volunteering > > > > > > Best, > > > D. > > > > > > On Mon, Jan 24, 2022 at 3:24 PM Till Rohrmann > > > wrote: > > > > > > > +1 for the 1.13.6 release and thanks for volunteering Konstantin. > > > > > > > > Cheers, > > > > Till > > > > > > > > On Mon, Jan 24, 2022 at 2:57 PM Konstantin Knauf > > > > wrote: > > > > > > > > > Thanks for starting the discussion and +1 to releasing. > > > > > > > > > > I am happy to manage the release aka learn how to do this. > > > > > > > > > > Cheers, > > > > > > > > > > Konstantin > > > > > > > > > > On Mon, Jan 24, 2022 at 2:52 PM Martijn Visser < > > mart...@ververica.com> > > > > > wrote: > > > > > > > > > > > I would like to start a discussion on releasing Flink 1.13.6. > Flink > > > > > 1.13.5 > > > > > > was the latest release on the 16th of December, which was the > > > emergency > > > > > > release for the Log4j CVE [1]. Flink 1.13.4 was cancelled, > leaving > > > > Flink > > > > > > 1.13.3 as the last real bugfix release. This one was released on > > the > > > > 19th > > > > > > of October last year. > > > > > > > > > > > > Since then, there have been 61 fixed tickets, excluding the test > > > > > > stabilities [3]. This includes a blocker and a couple of critical > > > > issues. > > > > > > > > > > > > Is there a PMC member who would like to manage the release? I'm > > more > > > > than > > > > > > happy to help with monitoring the status of the tickets. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn Visser > > > > > > https://twitter.com/MartijnVisser82 > > > > > > > > > > > > [1] > > > https://flink.apache.org/news/2021/12/16/log4j-patch-releases.html > > > > > > [2] https://flink.apache.org/news/2021/10/19/release-1.13.3.html > > > > > > [3] JQL filter: project = FLINK AND resolution = Fixed AND > > > fixVersion = > > > > > > 1.13.6 AND labels != test-stability ORDER BY priority DESC, > created > > > > DESC > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Konstantin Knauf > > > > > > > > > > https://twitter.com/snntrable > > > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi everyone, Thank you for sharing your perspectives. I was not aware of these limitations of per-job mode on YARN. It seems that there is a general agreement to deprecate per-job mode and to drop it once the limitations around YARN are resolved. I've started a corresponding vote in [1]. Thanks again, Konstantin [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky wrote: > Hi Yang, > > Thank you for the clarification. In general I think we will have time to > experiment with this until it will be removed totally and migrate our > solution to use application mode. > > Regards, > F > > On 2022/01/26 02:42:24 Yang Wang wrote: > > Hi all, > > > > I remember the application mode was initially named "cluster mode". As a > > contrast, the per-job mode is the "client mode". > > So I believe application mode should cover all the functionalities of > > per-job except where we are running the user main code. > > In the containerized or the Kubernetes world, the application mode is > more > > native and easy to use since all the Flink and user > > jars are bundled in the image. I am also in favor of deprecating and > > removing the per-job in the long run. > > > > > > > > @Ferenc > > IIRC, the YARN application mode could ship user jars and dependencies via > > "yarn.ship-files" config option. The only > > limitation is that we could not ship and load the user dependencies with > > user classloader, not the parent classloader. > > FLINK-24897 is trying to fix this via supporting "usrlib" directory > > automatically. > > > > > > Best, > > Yang > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > Hi Konstantin, > > > > > > First of all, sorry for the delay. We at Cloudera are currently > relying on > > > per-job mode deploying Flink applications over YARN. > > > > > > Specifically, we allow users to upload connector jars and other > artifacts. > > > There are also some default jars that we need to ship. These are all > stored > > > on the local file system of our service’s node. The Flink job is > submitted > > > on the users’ behalf by our service, which also specifies the jars to > ship. > > > The service runs on a single node, not on all nodes with Flink TM/JM. > It > > > would thus be difficult to manage the jars on every node. > > > > > > We are not familiar with the reasoning behind why application mode > > > currently doesn’t ship the user jars, besides the deployment being > faster > > > this way. Would it be possible for the application mode to (optionally, > > > enabled by some config) distribute these, or are there some technical > > > limitations? > > > > > > For us it would be crucial to achieve the functionality we have at the > > > moment over YARN. We started to track > > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > > mentioned as well. > > > > > > Considering the above, for us the more soonish removal does not sound > > > really well. We can live with this feature as deprecated of course, > but it > > > would be nice to have some time to figure out how we can utilize > > > Application Mode exactly and make necessary changes if required. > > > > > > Thank you, > > > F > > > > > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > > > Hi everyone, > > > > > > > > I would like to discuss and understand if the benefits of having > Per-Job > > > > Mode in Apache Flink outweigh its drawbacks. > > > > > > > > > > > > *# Background: Flink's Deployment Modes* > > > > Flink currently has three deployment modes. They differ in the > following > > > > dimensions: > > > > * main() method executed on Jobmanager or Client > > > > * dependencies shipped by client or bundled with all nodes > > > > * number of jobs per cluster & relationship between job and cluster > > > > lifecycle* (supported resource providers) > > > > > > > > ## Application Mode > > > > * main() method executed on Jobmanager > > > > * dependencies already need to be available on all nodes > > > > * dedicated cluster for all jobs executed from the same main()-method > > > > (Note: applications with more than one job, currently still > significant > >
[VOTE] Deprecate Per-Job Mode in Flink 1.15
Hi everyone, Based on the discussion in [1], I would like to start a vote on deprecating per-job mode in Flink 1.15. Consequently, we would target to drop it in Flink 1.16 or Flink 1.17 latest. The only limitation that would block dropping Per-Job mode mentioned in [1] is tracked in https://issues.apache.org/jira/browse/FLINK-24897. In general, the implementation of application mode in YARN should be on par with the standalone and Kubernetes before we drop per-job mode. The vote will last for at least 72 hours, and will be accepted by a consensus of active committers. Thanks, Konstantin [1] https://lists.apache.org/thread/b8g76cqgtr2c515rd1bs41vy285f317n -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Deprecate Per-Job Mode in Flink 1.15
Hi Xintong, Hi Yang, Hi everyone, Thank you for speaking up. The vote is formally only about the deprecation in Flink 1.15. We can and should continue to collect blockers for the deletion of per-job mode on YARN. Then there should be one release that allows users to switch. So, Flink 1.16 indeed is unrealistic for dropping, as we would need to address all Blockers still in Flink 1.15. I think a certain degree of urgency helps us to address these issues and encourages users to switch to application mode. So, I would continue to target Flink 1.17 for dropping per-job mode, but let's reevaluate after Flink 1.16. Hope this helps, Konstantin Since we recently decided that On Sun, Jan 30, 2022 at 4:13 AM Yang Wang wrote: > Hi all, > > > > I second Xintong’s comments to not drop the per-job mode too aggressively. > And I am afraid > > we need to get more inputs from users after deprecating the per-job mode in > release-1.15. > > > Most Flink on YARN users are using CLI command to integrate with the job > lifecycle management system. > > And they are still using the old compatibility mode "flink run -m > yarn-cluster", not the generic CLI mode "--target > yarn-per-job/yarn-application". > > Apart from the functionalities, they need some time to upgrade the external > systems. > > > BTW, the application mode does not support attached mode now. Some users > have asked for this in FLINK-25495[1]. > > > [1]. https://issues.apache.org/jira/browse/FLINK-25495 > > > > > Best, > > Yang > > Xintong Song 于2022年1月30日周日 08:35写道： > > > Hi Konstantin, > > > > Could we be more specific about what this vote is for? I'm asking > because I > > don't think we have consensus on all you have mentioned. > > > > To be specific, I'd be +1 for deprecating per-job mode in 1.15. However, > > I'm not sure about the following. > > - Targeting to drop it in 1.16 or 1.17. TBH, I'd expect to stay > compatible > > on the per-job mode a bit longer. > > - Targeting Yarn application mode on par with the standalone / K8s. I > think > > we need the Yarn application mode on par with the Yarn per-job mode, as > the > > latter is being dropped and users are migrating from. > > - FLINK-24897 being the only blocker for dropping the per-job mode. I > think > > a good time to drop the per-job mode is probably when we know most users > > have migrated to the application mode. Even if the Yarn application mode > > provides equivalent functionality as the Yarn per-job mode does, it's > > probably nicer to not force users to migrate if the per-job mode is still > > widely used. > > > > Discussing the above items is not my purpose here. Just trying to say > that > > IMHO in the previous discussion [1] we have not reached consensus on all > > the things mentioned in this voting thread. Consequently, if these are > all > > included in the scope of the vote, I'm afraid I cannot give my +1 on > this. > > Sorry if I'm nitpicking. > > > > Thank you~ > > > > Xintong Song > > > > > > [1] https://lists.apache.org/thread/b8g76cqgtr2c515rd1bs41vy285f317n > > > > On Sat, Jan 29, 2022 at 2:27 PM Jing Zhang wrote: > > > > > +1 (binding) > > > > > > Thanks Konstantin for driving this. > > > > > > Best, > > > Jing Zhang > > > > > > Chenya Zhang 于2022年1月29日周六 07:04写道： > > > > > > > +1 (non-binding) > > > > > > > > On Fri, Jan 28, 2022 at 12:46 PM Thomas Weise > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > On Fri, Jan 28, 2022 at 9:27 AM David Morávek > > wrote: > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > D. > > > > > > > > > > > > On Fri 28. 1. 2022 at 17:53, Till Rohrmann > > > > > wrote: > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > Cheers, > > > > > > > Till > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 4:57 PM Gabor Somogyi < > > > > > gabor.g.somo...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > We're intended to make tests when FLINK-24897 > > > > > > >
[VOTE] Release 1.13.6, release candidate #1
Hi everyone, Please review and vote on the release candidate #1 for the version 1.13.6, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to be deployed to dist.apache.org [2], which are signed with the key with fingerprint 8C3FB007FE60 DEFA [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.2.3-rc3" [5], * website pull request listing the new release and adding announcement blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Konstantin [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12351074 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.13.6-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1486/ [5] https://github.com/apache/flink/tree/release-1.13.6-rc1 [6] https://github.com/apache/flink-web/pull/505 -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[RESULT] [VOTE] Deprecate Per-Job Mode
Hi everyone, The vote on deprecating per-job mode in Flink 1.15 has been unanimously approved in [1]. I've created a ticket for deprecation [2] and dropping [3] and linked the current blockers for dropping it to the latter. Binding +1 Thomas Weise Xintong Song Yang Wang Jing Zhang Till Rohrmann Non-Binding +1 Chenya Zhang David Moravek Gabor Somogyi Cheers, Konstantin [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 [2] https://issues.apache.org/jira/browse/FLINK-25999 [3] https://issues.apache.org/jira/browse/FLINK-26000 -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] Deprecate Per-Job Mode in Flink 1.15
Thank you everyone, I've closed the vote and created a ticket for deprecation [1] and dropping [2] and linked the current blockers for dropping it to the latter. Please if or when you encounter new blockers link them to [2]. [1] https://issues.apache.org/jira/browse/FLINK-25999 [2] https://issues.apache.org/jira/browse/FLINK-26000 On Sun, Feb 6, 2022 at 6:44 AM Yang Wang wrote: > Thanks Konstantin for the explanation. > > +1 (binding) from me now. > > > Best, > Yang > > Xintong Song 于2022年2月3日周四 09:42写道： > > > Thanks for the clarification, Konstantin. > > > > +1 for deprecating per-job mode in Flink 1.15, and reevaluating when to > > drop it after Flink 1.16. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Tue, Feb 1, 2022 at 5:27 PM Konstantin Knauf > wrote: > > > > > Hi Xintong, Hi Yang, Hi everyone, > > > > > > Thank you for speaking up. The vote is formally only about the > > deprecation > > > in Flink 1.15. > > > > > > We can and should continue to collect blockers for the deletion of > > per-job > > > mode on YARN. Then there should be one release that allows users to > > switch. > > > So, Flink 1.16 indeed is unrealistic for dropping, as we would need to > > > address all Blockers still in Flink 1.15. > > > > > > I think a certain degree of urgency helps us to address these issues > and > > > encourages users to switch to application mode. So, I would continue to > > > target Flink 1.17 for dropping per-job mode, but let's reevaluate after > > > Flink 1.16. > > > > > > Hope this helps, > > > > > > Konstantin > > > > > > Since we recently decided that > > > On Sun, Jan 30, 2022 at 4:13 AM Yang Wang > wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > > > I second Xintong’s comments to not drop the per-job mode too > > > aggressively. > > > > And I am afraid > > > > > > > > we need to get more inputs from users after deprecating the per-job > > mode > > > in > > > > release-1.15. > > > > > > > > > > > > Most Flink on YARN users are using CLI command to integrate with the > > job > > > > lifecycle management system. > > > > > > > > And they are still using the old compatibility mode "flink run -m > > > > yarn-cluster", not the generic CLI mode "--target > > > > yarn-per-job/yarn-application". > > > > > > > > Apart from the functionalities, they need some time to upgrade the > > > external > > > > systems. > > > > > > > > > > > > BTW, the application mode does not support attached mode now. Some > > users > > > > have asked for this in FLINK-25495[1]. > > > > > > > > > > > > [1]. https://issues.apache.org/jira/browse/FLINK-25495 > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > Yang > > > > > > > > Xintong Song 于2022年1月30日周日 08:35写道： > > > > > > > > > Hi Konstantin, > > > > > > > > > > Could we be more specific about what this vote is for? I'm asking > > > > because I > > > > > don't think we have consensus on all you have mentioned. > > > > > > > > > > To be specific, I'd be +1 for deprecating per-job mode in 1.15. > > > However, > > > > > I'm not sure about the following. > > > > > - Targeting to drop it in 1.16 or 1.17. TBH, I'd expect to stay > > > > compatible > > > > > on the per-job mode a bit longer. > > > > > - Targeting Yarn application mode on par with the standalone / > K8s. I > > > > think > > > > > we need the Yarn application mode on par with the Yarn per-job > mode, > > as > > > > the > > > > > latter is being dropped and users are migrating from. > > > > > - FLINK-24897 being the only blocker for dropping the per-job > mode. I > > > > think > > > > > a good time to drop the per-job mode is probably when we know most > > > users > > > > > have migrated to the application mode. Even if the Yarn application > > > mode > > > > > provides equivalent functionality as the Yarn per-j
Re: [DISCUSS] Drop Jepsen tests
Thank you for raising this issue. What risks do you see if we drop it? Do you see any cheaper alternative to (partially) mitigate those risks? On Wed, Feb 9, 2022 at 12:40 PM Chesnay Schepler wrote: > For a few years by now we had a set of Jepsen tests that verify the > correctness of Flinks coordination layer in the case of process crashes. > In the past it has indeed found issues and thus provided value to the > project, and in general the core idea of it (and Jepsen for that matter) > is very sound. > > However, so far we neither made attempts to make further use of Jepsen > (and limited ourselves to very basic tests) nor to familiarize ourselves > with the tests/jepsen at all. > As a result these tests are difficult to maintain. They (and Jepsen) are > written in Clojure, which makes debugging, changes and upstreaming > contributions very difficult. > Additionally, the tests also make use of a very complicated > (Ververica-internal) terraform+ansible setup to spin up and tear down > AWS machines. While it works (and is actually pretty cool), it's > difficult to adjust because the people who wrote it have left the company. > > Why I'm raising this now (and not earlier) is because so far keeping the > tests running wasn't much of a problem; bump a few dependencies here and > there and we're good to go. > > However, this has changed with the recent upgrade to Zookeeper 3.5, > which isn't supported by Jepsen out-of-the-box, completely breaking the > tests. We'd now have to write a new Zookeeper 3.5+ integration for > Jepsen (again, in Clojure). While I started working on that and could > likely finish it, I started to wonder whether it even makes sense to do > so, and whether we couldn't invest this time elsewhere. > > Let me know what you think. > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Azure Pipelines are dealing with an incident, causing pipeline runs to fail
Thanks for following up on this, Martijn. On Fri, Feb 11, 2022 at 9:03 AM Martijn Visser wrote: > Hi all, > > The issue has been resolved now indeed. The cause was the following: > > - Azure had an outage [1] > - When that outage was resolved, all of Flink's free build agents were > gone. > - A temporary switch was made to paid agents, while we reached out once > more to Azure. However, those paid agents had different/lesser specs, > causing the e2e_ci runs to fail. > - It was confirmed that Azure had another outage, causing the free build > agents to be not available [2]. That outage has now been resolved too. > > Everything should work normally now. > > Best regards, > > Martijn > > [1] https://status.dev.azure.com/_event/287959626 > [2] https://status.dev.azure.com/_event/288297230 > > On Fri, 11 Feb 2022 at 06:56, Biao Geng wrote: > > > Just notice that there are some recent pipelines that have finished > > successfully[1]. It seems that this issue is mitigated. > > > > [1] > > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=31194=results > > > > Martijn Visser 于2022年2月10日周四 15:48写道： > > > > > It looks like the issues haven't been resolved yet unfortunately. > > > > > > On Wed, 9 Feb 2022 at 13:15, Robert Metzger > wrote: > > > > > > > I filed a support request with Microsoft: > > > > > > > > > > > > > > https://developercommunity.visualstudio.com/t/Number-of-Microsoft-hosted-agents-droppe/1658827?from=email=21=newest > > > > > > > > On Wed, Feb 9, 2022 at 1:04 PM Martijn Visser > > > > > wrote: > > > > > > > > > Unfortunately it looks like there are still failures. Will keep you > > > > posted > > > > > > > > > > On Wed, 9 Feb 2022 at 11:51, Martijn Visser > > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > The issue should now be resolved. > > > > > > > > > > > > Best regards, > > > > > > > > > > > > Martijn > > > > > > > > > > > > On Wed, 9 Feb 2022 at 10:55, Martijn Visser < > mart...@ververica.com > > > > > > > > wrote: > > > > > > > > > > > >> Hi everyone, > > > > > >> > > > > > >> Please keep in mind that Azure Pipelines currently is dealing > with > > > an > > > > > >> incident [1] which causes all CI pipeline runs on Azure to fail. > > > When > > > > > the > > > > > >> incident has been resolved, it will be required to retrigger > your > > > > > pipeline > > > > > >> to see if the pipeline then passes. > > > > > >> > > > > > >> Best regards, > > > > > >> > > > > > >> Martijn Visser > > > > > >> https://twitter.com/MartijnVisser82 > > > > > >> > > > > > >> [1] https://status.dev.azure.com/_event/287959626 > > > > > >> > > > > > > > > > > > > > > > > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Release Flink 1.14.4
Hi everyone, what do you think about a timely Flink 1.14.4 in order to release the fix for https://issues.apache.org/jira/browse/FLINK-25732. Currently, the landing page of Flink Web User Interface is not working when there are standby Jobmanagers. In addition, I propose to wait for https://issues.apache.org/jira/browse/FLINK-26018 to be resolved. I can volunteer as release manager. Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Future of Per-Job Mode
Thanks Thomas & Biao for your feedback. Any additional opinions on how we should proceed with per job-mode? As you might have guessed, I am leaning towards proposing to deprecate per-job mode. On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: > Regarding session mode: > > ## Session Mode > * main() method executed in client > > Session mode also supports execution of the main method on Jobmanager > with submission through REST API. That's how Flinkk k8s operators like > [1] work. It's actually an important capability because it allows for > allocation of the cluster resources prior to taking down the previous > job during upgrade when the goal is optimization for availability. > > Thanks, > Thomas > > [1] https://github.com/lyft/flinkk8soperator > > On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf > wrote: > > > > Hi everyone, > > > > I would like to discuss and understand if the benefits of having Per-Job > > Mode in Apache Flink outweigh its drawbacks. > > > > > > *# Background: Flink's Deployment Modes* > > Flink currently has three deployment modes. They differ in the following > > dimensions: > > * main() method executed on Jobmanager or Client > > * dependencies shipped by client or bundled with all nodes > > * number of jobs per cluster & relationship between job and cluster > > lifecycle* (supported resource providers) > > > > ## Application Mode > > * main() method executed on Jobmanager > > * dependencies already need to be available on all nodes > > * dedicated cluster for all jobs executed from the same main()-method > > (Note: applications with more than one job, currently still significant > > limitations like missing high-availability). Technically, a session > cluster > > dedicated to all jobs submitted from the same main() method. > > * supported by standalone, native kubernetes, YARN > > > > ## Session Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * cluster is shared by multiple jobs submitted from different clients, > > independent lifecycle > > * supported by standalone, Native Kubernetes, YARN > > > > ## Per-Job Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * dedicated cluster for a single job > > * supported by YARN only > > > > > > *# Reasons to Keep** There are use cases where you might need the > > combination of a single job per cluster, but main() method execution in > the > > client. This combination is only supported by per-job mode. > > * It currently exists. Existing users will need to migrate to either > > session or application mode. > > > > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > > modes that for most users probably do the same thing. Specifically, for > > those users that don't care where the main() method is executed and want > to > > submit a single job per cluster. Having two ways to do the same thing is > > confusing. > > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > > work towards support in Kubernetes and Standalone, too, to reduce special > > casing. > > * Dropping per-job mode would reduce complexity in the code and allow us > to > > dedicate more resources to the other two deployment modes. > > * I believe with session mode and application mode we have to easily > > distinguishable and understandable deployment modes that cover Flink's > use > > cases: > >* session mode: olap-style, interactive jobs/queries, short lived > batch > > jobs, very small jobs, traditional cluster-centric deployment mode (fits > > the "Hadoop world") > >* application mode: long-running streaming jobs, large scale & > > heterogenous jobs (resource isolation!), application-centric deployment > > mode (fits the "Kubernetes world") > > > > > > *# Call to Action* > > * Do you use per-job mode? If so, why & would you be able to migrate to > one > > of the other methods? > > * Am I missing any pros/cons? > > * Are you in favor of dropping per-job mode midterm? > > > > Cheers and thank you, > > > > Konstantin > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [VOTE] FLIP-203: Incremental savepoints
Thanks, Piotr. Proposal looks good. +1 (binding) On Mon, Jan 24, 2022 at 11:20 AM David Morávek wrote: > +1 (non-binding) > > Best, > D. > > On Mon, Jan 24, 2022 at 10:54 AM Dawid Wysakowicz > wrote: > > > +1 (binding) > > > > Best, > > > > Dawid > > > > On 24/01/2022 09:56, Piotr Nowojski wrote: > > > Hi, > > > > > > As there seems to be no further questions about the FLIP-203 [1] I > would > > > propose to start a voting thread for it. > > > > > > For me there are still two unanswered questions, whether we want to > > support > > > schema evolution and State Processor API with native format snapshots > or > > > not. But I would propose to tackle them as follow ups, since those are > > > pre-existing issues of the native format checkpoints, and could be done > > > completely independently of providing the native format support in > > > savepoints. > > > > > > Best, > > > Piotrek > > > > > > [1] > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints > > > > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Releasing Flink 1.13.6
Thanks for starting the discussion and +1 to releasing. I am happy to manage the release aka learn how to do this. Cheers, Konstantin On Mon, Jan 24, 2022 at 2:52 PM Martijn Visser wrote: > I would like to start a discussion on releasing Flink 1.13.6. Flink 1.13.5 > was the latest release on the 16th of December, which was the emergency > release for the Log4j CVE [1]. Flink 1.13.4 was cancelled, leaving Flink > 1.13.3 as the last real bugfix release. This one was released on the 19th > of October last year. > > Since then, there have been 61 fixed tickets, excluding the test > stabilities [3]. This includes a blocker and a couple of critical issues. > > Is there a PMC member who would like to manage the release? I'm more than > happy to help with monitoring the status of the tickets. > > Best regards, > > Martijn Visser > https://twitter.com/MartijnVisser82 > > [1] https://flink.apache.org/news/2021/12/16/log4j-patch-releases.html > [2] https://flink.apache.org/news/2021/10/19/release-1.13.3.html > [3] JQL filter: project = FLINK AND resolution = Fixed AND fixVersion = > 1.13.6 AND labels != test-stability ORDER BY priority DESC, created DESC > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-203: Incremental savepoints
> > > > Is `state.backend.incremental` the only configuration parameter that can > be > > used in this context? I would guess not? What about for example > > "state.storage.fs.memory-threshold" or all of the Advanced RocksDB State > > Backends Options [2]? > > > > David: > > > > > does this mean that we need to keep the checkpoints compatible across > > minor > > > versions? Or can we say, that the minor version upgrades are only > > > guaranteed with canonical savepoints? > > > > Good question. Frankly I was always assuming that this is implicitly > given. > > Otherwise users would not be able to recover jobs that are failing > because > > of bugs in Flink. But I'm pretty sure that was never explicitly stated. > > > > As Konstantin suggested, I've written down the pre-existing guarantees of > > checkpoints and savepoints followed by two proposals on how they should > be > > changed [1]. Could you take a look? > > > > I'm especially unsure about the following things: > > a) What about RocksDB upgrades? If we bump RocksDB version between Flink > > versions, do we support recovering from a native format snapshot > > (incremental checkpoint)? > > b) State Processor API - both pre-existing and what do we want to provide > > in the future > > c) Schema Evolution - both pre-existing and what do we want to provide in > > the future > > > > Best, > > Piotrek > > > > [1] > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-203%3A+Incremental+savepoints#FLIP203:Incrementalsavepoints-Checkpointvssavepointguarantees > > [2] > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#advanced-rocksdb-state-backends-options > > > > wt., 11 sty 2022 o 09:45 Konstantin Knauf > napisał(a): > > > > > Hi Piotr, > > > > > > would it be possible to provide a table that shows the > > > compatibility guarantees provided by the different snapshots going > > forward? > > > Like type of change (Topology. State Schema, Parallelism, ..) in one > > > dimension, and type of snapshot as the other dimension. Based on that, > it > > > would be easier to discuss those guarantees, I believe. > > > > > > Cheers, > > > > > > Konstantin > > > > > > On Mon, Jan 3, 2022 at 9:11 AM David Morávek wrote: > > > > > > > Hi Piotr, > > > > > > > > does this mean that we need to keep the checkpoints compatible across > > > minor > > > > versions? Or can we say, that the minor version upgrades are only > > > > guaranteed with canonical savepoints? > > > > > > > > My concern is especially if we'd want to change layout of the > > checkpoint. > > > > > > > > D. > > > > > > > > > > > > > > > > On Wed, Dec 29, 2021 at 5:19 AM Yu Li wrote: > > > > > > > > > Thanks for the proposal Piotr! Overall I'm +1 for the idea, and > below > > > are > > > > > my two cents: > > > > > > > > > > 1. How about adding a "Term Definition" section and clarify what > > > "native > > > > > format" (the "native" data persistence format of the current state > > > > backend) > > > > > and "canonical format" (the "uniform" format that supports > switching > > > > state > > > > > backends) means? > > > > > > > > > > 2. IIUC, currently the FLIP proposes to only support incremental > > > > savepoint > > > > > with native format, and there's no plan to add such support for > > > canonical > > > > > format, right? If so, how about writing this down explicitly in the > > > FLIP > > > > > doc, maybe in a "Limitations" section, plus the fact that > > > > > `HashMapStateBackend` cannot support incremental savepoint before > > > > FLIP-151 > > > > > is done? (side note: @Roman just a kindly reminder, that please > take > > > > > FLIP-203 into account when implementing FLIP-151) > > > > > > > > > > 3. How about changing the description of "the default configuration > > of > > > > the > > > > > checkpoints will be used to determine whether the savepoint should > be > > > > > incremental or not" to s
Re: Looking for Maintainers for Flink on YARN
Hi everyone, We are seeing an increasing number of test instabilities related to YARN [1]. Does someone in this group have the time to pick these up? The Flink Confluence contains a guide on how to triage test instability tickets. Thanks, Konstantin [1] https://issues.apache.org/jira/browse/FLINK-25514?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20%22Deployment%20%2F%20YARN%22%20AND%20labels%20%3D%20test-stability [2] https://cwiki.apache.org/confluence/display/FLINK/Triage+Test+Instability+Tickets On Mon, Sep 13, 2021 at 2:22 PM 柳尘 wrote: > Thanks to Konstantin for raising this question, and to Marton and Gabor > To strengthen! > > If i can help > In order to better participate in the work, please let me know. > > the best, > cheng xingyuan > > > > 2021年7月29日 下午4:15，Konstantin Knauf 写道： > > > > Dear community, > > > > We are looking for community members, who would like to maintain Flink's > > YARN support going forward. So far, this has been handled by teams at > > Ververica & Alibaba. The focus of these teams has shifted over the past > > months so that we only have little time left for this topic. Still, we > > think, it is important to maintain high quality support for Flink on > YARN. > > > > What does "Maintaining Flink on YARN" mean? There are no known bigger > > efforts outstanding. We are mainly talking about addressing > > "test-stability" issues, bugs, version upgrades, community contributions > & > > smaller feature requests. The prioritization of these would be up to the > > future maintainers, except "test-stability" issues which are important to > > address for overall productivity. > > > > If a group of community members forms itself, we are happy to give an > > introduction to relevant pieces of the code base, principles, > assumptions, > > ... and hand over open threads. > > > > If you would like to take on this responsibility or can join this effort > in > > a supporting role, please reach out! > > > > Cheers, > > > > Konstantin > > for the Deployment & Coordination Team at Ververica > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > > -- Konstantin Knauf | Head of Product +49 160 91394525 Follow us @VervericaData Ververica <https://www.ververica.com/> -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, Jinwei (Kevin) Zhang
Re: Looking for Maintainers for Flink on YARN
The link to the filter is not correct. Here is the correct link: https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20%22Deployment%20%2F%20YARN%22%20AND%20labels%20%3D%20test-stability On Wed, Jan 26, 2022 at 10:17 AM Konstantin Knauf wrote: > Hi everyone, > > We are seeing an increasing number of test instabilities related to YARN > [1]. Does someone in this group have the time to pick these up? The Flink > Confluence contains a guide on how to triage test instability tickets. > > Thanks, > > Konstantin > > [1] > https://issues.apache.org/jira/browse/FLINK-25514?jql=project%20%3D%20FLINK%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20component%20%3D%20%22Deployment%20%2F%20YARN%22%20AND%20labels%20%3D%20test-stability > [2] > https://cwiki.apache.org/confluence/display/FLINK/Triage+Test+Instability+Tickets > > On Mon, Sep 13, 2021 at 2:22 PM 柳尘 wrote: > >> Thanks to Konstantin for raising this question, and to Marton and Gabor >> To strengthen! >> >> If i can help >> In order to better participate in the work, please let me know. >> >> the best, >> cheng xingyuan >> >> >> > 2021年7月29日 下午4:15，Konstantin Knauf 写道： >> > >> > Dear community, >> > >> > We are looking for community members, who would like to maintain Flink's >> > YARN support going forward. So far, this has been handled by teams at >> > Ververica & Alibaba. The focus of these teams has shifted over the past >> > months so that we only have little time left for this topic. Still, we >> > think, it is important to maintain high quality support for Flink on >> YARN. >> > >> > What does "Maintaining Flink on YARN" mean? There are no known bigger >> > efforts outstanding. We are mainly talking about addressing >> > "test-stability" issues, bugs, version upgrades, community >> contributions & >> > smaller feature requests. The prioritization of these would be up to the >> > future maintainers, except "test-stability" issues which are important >> to >> > address for overall productivity. >> > >> > If a group of community members forms itself, we are happy to give an >> > introduction to relevant pieces of the code base, principles, >> assumptions, >> > ... and hand over open threads. >> > >> > If you would like to take on this responsibility or can join this >> effort in >> > a supporting role, please reach out! >> > >> > Cheers, >> > >> > Konstantin >> > for the Deployment & Coordination Team at Ververica >> > >> > -- >> > >> > Konstantin Knauf >> > >> > https://twitter.com/snntrable >> > >> > https://github.com/knaufk >> >> > > -- > > Konstantin Knauf | Head of Product > > +49 160 91394525 > > > Follow us @VervericaData Ververica <https://www.ververica.com/> > > > -- > > Join Flink Forward <https://flink-forward.org/> - The Apache Flink > Conference > > Stream Processing | Event Driven | Real Time > > -- > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany > > -- > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung Jason, > Jinwei (Kevin) Zhang > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Release Flink 1.16.3
+1. Thank you for picking it up. Yes, this will be the final bug fix for Flink 1.16. Am Mo., 6. Nov. 2023 um 03:47 Uhr schrieb Rui Fan <1996fan...@gmail.com>: > Hi all, > > I would like to discuss creating a new 1.16 patch release (1.16.3). The > last 1.16 release is over five months old, and since then, 50 tickets have > been closed [1], of which 10 are blocker/critical [2]. Some > of them are quite important, such as FLINK-32296 [3], FLINK-32548 [4] > and FLINK-33010[5]. > > In addition to this, FLINK-33149 [6] is important to bump snappy-java to > 1.1.10.4. > Although FLINK-33149 is unresolved, it was done in 1.16.3. > > I am not aware of any unresolved blockers and there are no in-progress > tickets [7]. Please let me know if there are any issues you'd like to be > included in this release but still not merged. > > Since 1.18.0 has been released, I'd suggest that we vote to make 1.16.3 > the final bugix release of 1.16, looking forward to any feedback from you. > Background info could be found at [8], and thanks Jing for the information. > > If the community agrees to create this new patch release, I could > volunteer as the release manager. > > Since there will be another flink-1.17.2 release request during the same > time, > I will work with Yun and Yu since many issues will be fixed in both > releases. > > [1] > > https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC > [2] > > https://issues.apache.org/jira/browse/FLINK-32231?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.16.3%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC > [3] https://issues.apache.org/jira/browse/FLINK-32296 > [4] https://issues.apache.org/jira/browse/FLINK-32548 > [5] https://issues.apache.org/jira/browse/FLINK-33010 > [6] https://issues.apache.org/jira/browse/FLINK-33149 > [7] https://issues.apache.org/jira/projects/FLINK/versions/12353259 > [8] https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv > > Best, > Rui > -- https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Release 1.17.2
Thank you for picking it up! +1 Cheers, Konstantin Am Mo., 6. Nov. 2023 um 03:48 Uhr schrieb Yun Tang : > Hi all, > > I would like to discuss creating a new 1.17 patch release (1.17.2). The > last 1.17 release is near half a year old, and since then, 79 tickets have > been closed [1], of which 15 are blocker/critical [2]. Some > of them are quite important, such as FLINK-32758 [3], FLINK-32296 [4], > FLINK-32548 [5] > and FLINK-33010[6]. > > In addition to this, FLINK-33149 [7] is important to bump snappy-java to > 1.1.10.4. > Although FLINK-33149 is unresolved, it was done in 1.17.2. > > I am not aware of any unresolved blockers and there are no in-progress > tickets [8]. Please let me know if there are any issues you'd like to be > included in this release but still not merged. > > If the community agrees to create this new patch release, I could > volunteer as the release manager with Yu Chen. > > Since there will be another flink-1.16.3 release request during the same > time, we will work with Rui Fan since many issues will be fixed in both > releases. > > [1] > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20%20and%20resolution%20%20!%3D%20%20Unresolved%20order%20by%20priority%20DESC > [2] > https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20fixVersion%20%3D%201.17.2%20and%20resolution%20%20!%3D%20Unresolved%20%20and%20priority%20in%20(Blocker%2C%20Critical)%20ORDER%20by%20priority%20%20DESC > [3] https://issues.apache.org/jira/browse/FLINK-32758 > [4] https://issues.apache.org/jira/browse/FLINK-32296 > [5] https://issues.apache.org/jira/browse/FLINK-32548 > [6] https://issues.apache.org/jira/browse/FLINK-33010 > [7] https://issues.apache.org/jira/browse/FLINK-33149 > [8] https://issues.apache.org/jira/projects/FLINK/versions/12353260 > > Best > Yun Tang > -- https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-364: Improve the restart-strategy
Hi Rui, Thank you for this proposal and working on this. I also agree that exponential back off makes sense as a new default in general. I think restarting indefinitely (no max attempts) makes sense by default, though, but of course allowing users to change is valuable. So, overall +1. Cheers, Konstantin Am Di., 17. Okt. 2023 um 07:11 Uhr schrieb Rui Fan <1996fan...@gmail.com>: > Hi all, > > I would like to start a discussion on FLIP-364: Improve the > restart-strategy[1] > > As we know, the restart-strategy is critical for flink jobs, it mainly > has two functions: > 1. When an exception occurs in the flink job, quickly restart the job > so that the job can return to the running state. > 2. When a job cannot be recovered after frequent restarts within > a certain period of time, Flink will not retry but will fail the job. > > The current restart-strategy support for function 2 has some issues: > 1. The exponential-delay doesn't have the max attempts mechanism, > it means that flink will restart indefinitely even if it fails frequently. > 2. For multi-region streaming jobs and all batch jobs, the failure of > each region will increase the total number of job failures by +1, > even if these failures occur at the same time. If the number of > failures increases too quickly, it will be difficult to set a reasonable > number of retries. > If the maximum number of failures is set too low, the job can easily > reach the retry limit, causing the job to fail. If set too high, some jobs > will never fail. > > In addition, when the above two problems are solved, we can also > discuss whether exponential-delay can replace fixed-delay as the > default restart-strategy. In theory, exponential-delay is smarter and > friendlier than fixed-delay. > > I also thank Zhu Zhu for his suggestions on the option name in > FLINK-32895[2] in advance. > > Looking forward to and welcome everyone's feedback and suggestions, thank > you. > > [1] https://cwiki.apache.org/confluence/x/uJqzDw > [2] https://issues.apache.org/jira/browse/FLINK-32895 > > Best, > Rui > -- https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] The Flink Speed Center and benchmark daily run are back online
Thanks a lot for working on this! Am Do., 19. Okt. 2023 um 10:24 Uhr schrieb Zakelly Lan < zakelly@gmail.com>: > Hi everyone, > > Flink benchmarks [1] generate daily performance reports in the Apache > Flink slack channel (#flink-dev-benchmarks) to detect performance > regression [2]. Those benchmarks previously were running on several > machines donated and maintained by Ververica. Unfortunately, those > machines were gone due to account issues [3] and the benchmarks daily > run stopped since August 24th delaying the release of Flink 1.18 a > bit. [4]. > > Ververica donated several new machines! After several weeks of work, I > have successfully re-established the codespeed panel and benchmark > daily run pipelines on them. At this time, we are pleased to announce > that the Flink Speed Center and benchmark pipelines are back online. > These new machines have a more formal management to ensure that > previous accidents will not occur in the future. > > What's more, I successfully recovered historical data backed up by > Yanfei Lei [5]. So with the old domain [6] redirected to the new > machines, the old links that existed in previous records will still be > valid. Besides the benchmarks with Java8 and Java11, I also added a > pipeline for Java17 running daily. > > How to use it: > We also registered a new domain name 'flink-speed.xyz' for the Flink > Speed Center [7]. It is recommended to use the new domain in the > future. Currently, the self-service method of triggering benchmarks is > unavailable considering the lack of resources and potential > vulnerabilities of Jenkins. Please contact one of Apache Flink PMCs to > submit a benchmark. More info is updated on the wiki[8]. > > Daily Monitoring: > The performance daily monitoring on the Apache Flink slack channel [2] > is still unavailable as the benchmark results need more time to > stabilize in the new environment. Once the baseline results become > available for regression detection, I will enable the daily > monitoring. > > Please feel free to reach out to me if you have any suggestions or > questions. Thanks Ververica again for denoting machines! > > > Best, > Zakelly > > [1] https://github.com/apache/flink-benchmarks > [2] https://lists.apache.org/thread/zok62sx4m50c79htfp18ymq5vmtgbgxj > [3] https://issues.apache.org/jira/browse/FLINK-33052 > [4] https://lists.apache.org//thread/5x28rp3zct4p603hm4zdwx6kfr101w38 > [5] https://issues.apache.org/jira/browse/FLINK-30890 > [6] http://codespeed.dak8s.net:8000 > [7] http://flink-speed.xyz > [8] > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115511847 > -- https://twitter.com/snntrable https://github.com/knaufk
Re: [ANNOUNCE] Release 1.18.0, release candidate #1
Hi everyone, I've just opened a PR for the release announcement [1] and I am looking forward to reviews and feedback. Cheers, Konstantin [1] https://github.com/apache/flink-web/pull/680 Am Fr., 6. Okt. 2023 um 11:03 Uhr schrieb Sergey Nuyanzin < snuyan...@gmail.com>: > sorry for not mentioning it in previous mail > > based on the reason above I'm > -1 (non-binding) > > also there is one more issue [1] > which blocks all the externalised connectors testing against the most > recent commits in > to corresponding branches > [1] https://issues.apache.org/jira/browse/FLINK-33175 > > > On Thu, Oct 5, 2023 at 11:19 PM Sergey Nuyanzin > wrote: > > > Thanks for creating RC1 > > > > * Downloaded artifacts > > * Built from sources > > * Verified checksums and gpg signatures > > * Verified versions in pom files > > * Checked NOTICE, LICENSE files > > > > The strange thing I faced is > > CheckpointAfterAllTasksFinishedITCase.testRestoreAfterSomeTasksFinished > > fails on AZP [1] > > > > which looks like it is related to [2], [3] fixed in 1.18.0 (not 100% > > sure). > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-33186 > > [2] https://issues.apache.org/jira/browse/FLINK-32996 > > [3] https://issues.apache.org/jira/browse/FLINK-32907 > > > > On Tue, Oct 3, 2023 at 2:53 PM Ferenc Csaky > > wrote: > > > >> Thanks everyone for the efforts! > >> > >> Checked the following: > >> > >> - Downloaded artifacts > >> - Built Flink from source > >> - Verified checksums/signatures > >> - Verified NOTICE, LICENSE files > >> - Deployed dummy SELECT job via SQL gateway on standalone cluster, > things > >> seemed fine according to the log files > >> > >> +1 (non-binding) > >> > >> Best, > >> Ferenc > >> > >> > >> --- Original Message --- > >> On Friday, September 29th, 2023 at 22:12, Gabor Somogyi < > >> gabor.g.somo...@gmail.com> wrote: > >> > >> > >> > > >> > > >> > Thanks for the efforts! > >> > > >> > +1 (non-binding) > >> > > >> > * Verified versions in the poms > >> > * Built from source > >> > * Verified checksums and signatures > >> > * Started basic workloads with kubernetes operator > >> > * Verified NOTICE and LICENSE files > >> > > >> > G > >> > > >> > On Fri, Sep 29, 2023, 18:16 Matthias Pohl > matthias.p...@aiven.io.invalid > >> > > >> > wrote: > >> > > >> > > Thanks for creating RC1. I did the following checks: > >> > > > >> > > * Downloaded artifacts > >> > > * Built Flink from sources > >> > > * Verified SHA512 checksums GPG signatures > >> > > * Compared checkout with provided sources > >> > > * Verified pom file versions > >> > > * Went over NOTICE file/pom files changes without finding anything > >> > > suspicious > >> > > * Deployed standalone session cluster and ran WordCount example in > >> batch > >> > > and streaming: Nothing suspicious in log files found > >> > > > >> > > +1 (binding) > >> > > > >> > > On Fri, Sep 29, 2023 at 10:34 AM Etienne Chauchot > >> echauc...@apache.org > >> > > wrote: > >> > > > >> > > > Hi all, > >> > > > > >> > > > Thanks to the team for this RC. > >> > > > > >> > > > I did a quick check of this RC against user pipelines (1) coded > with > >> > > > DataSet (even if deprecated and soon removed), DataStream and SQL > >> APIs > >> > > > > >> > > > based on the small scope of this test, LGTM > >> > > > > >> > > > +1 (non-binding) > >> > > > > >> > > > [1] https://github.com/echauchot/tpcds-benchmark-flink > >> > > > > >> > > > Best > >> > > > Etienne > >> > > > > >> > > > Le 28/09/2023 à 19:35, Jing Ge a écrit : > >> > > > > >> > > > > Hi everyone, > >> > > > > > >> > > > > The RC1 for Apache Flink 1.18.0 has been created. The related > >> voting > >> > > > > process will be triggered once the announcement is ready. The > RC1 > >> has > >> > > > > all > >> > > > > the artifacts that we would typically have for a release, except > >> for > >> > > > > the > >> > > > > release note and the website pull request for the release > >> announcement. > >> > > > > > >> > > > > The following contents are available for your review: > >> > > > > > >> > > > > - Confirmation of no benchmarks regression at the thread[1]. > >> > > > > - The preview source release and binary convenience releases > [2], > >> which > >> > > > > are signed with the key with fingerprint 96AE0E32CBE6E0753CE6 > [3]. > >> > > > > - all artifacts that would normally be deployed to the Maven > >> > > > > Central Repository [4]. > >> > > > > - source code tag "release-1.18.0-rc1" [5] > >> > > > > > >> > > > > Your help testing the release will be greatly appreciated! And > >> we'll > >> > > > > create the rc1 release and the voting thread as soon as all the > >> efforts > >> > > > > are > >> > > > > finished. > >> > > > > > >> > > > > [1] > >> https://lists.apache.org/thread/yxyphglwwvq57wcqlfrnk3qo9t3sr2ro > >> > > > > [2] > https://dist.apache.org/repos/dist/dev/flink/flink-1.18.0-rc1/ > >> > > > > [3]https://dist.apache.org/repos/dist/release/flink/KEYS > >> > > > > [4] > >> > > > > > >>
Re: [DISCUSS] [FLINK-32873] Add a config to allow disabling Query hints
Hi Bonnie, this makes sense to me, in particular, given that we already have this toggle for a different type of hints. Best, Konstantin Am Mi., 16. Aug. 2023 um 19:38 Uhr schrieb Bonnie Arogyam Varghese : > Hi Liu, > Options hints could be a security concern since users can override > settings. However, query hints specifically could affect performance. > Since we have a config to disable Options hint, I'm suggesting we also have > a config to disable Query hints. > > On Wed, Aug 16, 2023 at 9:41 AM liu ron wrote: > > > Hi, > > > > Thanks for driving this proposal. > > > > Can you explain why you would need to disable query hints because of > > security issues? I don't really understand why query hints affects > > security. > > > > Best, > > Ron > > > > Bonnie Arogyam Varghese 于2023年8月16日周三 > > 23:59写道： > > > > > Platform providers may want to disable hints completely for security > > > reasons. > > > > > > Currently, there is a configuration to disable OPTIONS hint - > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/table/config/#table-dynamic-table-options-enabled > > > > > > However, there is no configuration available to disable QUERY hints - > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/queries/hints/#query-hints > > > > > > The proposal is to add a new configuration: > > > > > > Name: table.query-options.enabled > > > Description: Enable or disable the QUERY hint, if disabled, an > > > exception would be thrown if any QUERY hints are specified > > > Note: The default value will be set to true. > > > > > > -- https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Maintain a Calcite repository for Flink to accelerate the development for Flink SQL features
I share Chesay's concerns. I fear that we will generally upgrade Calcite even fewer and - in consequence - when we upgrade it will be even more painful. Can you elaborate on why Calcite upgrades have been hard in the past? Is there anything we can do to make this better? On Fri, Apr 22, 2022 at 10:07 AM Chesnay Schepler wrote: > I'm overall against the idea of creating a fork. > It implies quite some maintenance overhead, like dealing with unstable > tests, CI, licensing etc. and the overall release overhead. > > Is there no alternative where we can collaborate more with the calcite > guys, like verifying new features so bugs are caught sooner? > > On 22/04/2022 09:31, godfrey he wrote: > > Dear devs, > > > > I would like to open a discussion on the fact that currently many > > Flink SQL function > > development relies on Calcite releases, which seriously blocks some > > Flink SQL's features release. > > Therefore, I would like to discuss whether it is possible to solve this > problem > > by creating Flink's own Calcite repository. > > > > Currently, Flink depends on Caclite-1.26, FLIP-204[1] relies on > Calcite-1.30, > > and we recently want to support fully join-hints functionatity in > Flink-1.16, > > which relies on Calcite-1.31 (maybe two or three months later will be > released). > > > > In order to support some new features or fix some bugs, we need to > upgrade > > the Calcite version, but every time we upgrade Calcite version > > (especially upgrades > > across multiple versions), the processing is very tough: I remember > clearly that > > the Calcite upgrade from 1.22 to 1.26 took two weeks of full-time to > complete. > > > > Currently, in order to fix some bugs while not upgrading the Calcite > version, > > we copy the corresponding Calcite class directly into the Flink project > > and then modify it accordingly.[2] This approach is rather hacky and > > hard for code maintenance and upgrades. > > > > So, I had an idea whether we could solve this problem by maintaining a > > Calcite repository > > in the Flink community. This approach has been practiced within my > > company for many years. > > There are similar practices in the industry. For example, Apache Dill > > also maintains > > a separate Calcite repository[3]. > > > > The following is a brief analysis of the approach and the pros and > > cons of maintaining a separate repository. > > > > Approach: > > 1. Where to put the code? https://github.com/flink-extended is a good > place. > > 2. What extra code can be added to this repository? Only bug fixes and > features > > that are already merged into Calcite can be cherry-picked to this > repository. > > We also should try to push bug fixes to the Calcite community. > > Btw, the copied Calcite class in the Flink project can be removed. > > 3. How to upgrade the Calcite version? Check out the target Calcite > > release branch > > and rebase our bug fix code. (As we upgrade, we will maintain fewer > > and fewer older bug > > fixes code.) And then, verify all Calcte's tests and Flink's tests in > > the developer's local > > environment. If all tests are OK, release the Calcite branch, or fix > > it in the branch and re-test. > > After the branch is released, then the version of Calcite in Flink > > can be upgraded. For example: > > checkout calcite-1.26.0-flink-v1-SNAPSHOT branch from calcite-1.26.0, > > move all the copied > > Calcite code in Flink to the branch, and pick all the hint related > > changes from Calcite-1.31 to > > the branch. Then we can change the Calcite version in Flink to > > calcite-1.26.0-flink-v1-SNAPSHOT, > > and verify all tests in the locale. Release calcite-1.26.0-flink-v1 > > after all tests are successful. > > At last upgrade the calcite version to > > calcite-1.26.0-flink-v10-flink-v1, and open a PR. > > 4. Who will maintain it? The maintenance workload is minimal, but the > > upgrade work is > > laborious (actually, it's similar to before). I can maintain it in > > the early stage and standardise the processing. > > > > Pros. > > 1. The release of Flink is decoupled from the release of Calcite, > > making feature development and bug fix quicker > > 2. Reduce the hassle of unnecessary calcite upgrades > > 3. No hacking in Flink to maintain the Calcite copied code > > > > cons. > > 1. Need to maintain an additional Calcite repository > > 2. The Upgrades are a little more complicated than before > > > > Any feedback is very welcome! > > > > > > [1] > https://cwiki.apache.org/confluence/display/FLINK/FLIP-204%3A+Introduce+Hash+Lookup+Join > > [2] > https://github.com/apache/flink/tree/master/flink-table/flink-table-planner/src/main/java/org/apache/calcite > > [3] https://github.com/apache/drill/blob/master/pom.xml#L64 > > > > Best, > > Godfrey > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: Alertmanager Sink Connector
cc user@, bcc dev@ Hi Dhruv, Yes, this should be totally possible. For 1, I would use a ProcessFunction to buffer the alerts and register timers per alert for the repeated firing (or to clear it from state if it is resolved). From this ProcessFunction you send records to an AlertManagerSink. For 2. the AsyncSink, Fabian Paul (cc) might be the best person to judge if it is a good fit. There is PR [1] to add a blog post about the Async Sink that might be of help for you in the meantime. Cheers, Konstantin [1] https://github.com/apache/flink-web/pull/517 On Thu, Apr 14, 2022 at 9:43 AM Dhruv Patel wrote: > Hi, >We have a use case where we want to send alerts to Prometheus > Alertmanager (https://prometheus.io/docs/alerting/latest/alertmanager/) > from Flink. Each of these alerts have a startsAt, endsAt field along with > alertMetadata. Alertmanager expects clients to send all the FIRING alerts > every X minutes (X is configurable) with an updated endsAt time (greater > than X since alertmanager would drop the alert once endsAt time is reached) > and once the alert is in RESOLVED state stop sending it. The state updates > of the alerts would come from Kafka. So the pipeline is something like this > State Updates to Alerts (Kafka) -> Flink (Some Enrichment) -> Alertmanager > Sink > They provide a rest endpoint where we can post these alerts. I have some > questions to see if its achievable to develop a sink connector for > alertmanager in flink? > > 1. Is it possible to send data to a sink every X minutes from a custom sink > connector since I have to somehow simulate a behavior of continuously > sending the same alerts even because state updates are only received from > Kafka for FIRING -> RESOLVED state and not for FIRING -> FIRING state? I > was thinking of having a state of active alerts and somehow the sink > connector would get the state every X minutes, batch it and then send it to > alertmanager. However, I am not able to find resources to write some > implementation around it. > > 2. In Flink 1.15, there are Async Sinks ( > https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink) > but not much documentation around. Also don't know if it would be > achievable to write the continuous firing logic in alertmanager > > Any other ideas are welcomed. > > -- > *Thanks,* > *Dhruv* > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits
Hi Sebastian, Hi Dawid, As part of this FLIP, the `AlignedSplitReader` interface (aka the stop & resume behavior) will be implemented for Kafka and Pulsar only, correct? +1 in general. I believe it is valuable to complete the watermark aligned story with this FLIP. Cheers, Konstantin On Thu, Apr 21, 2022 at 12:36 PM Dawid Wysakowicz wrote: > To be explicit, having worked on it, I support it ;) I think we can > start a vote thread soonish, as there are no concerns so far. > > Best, > > Dawid > > On 13/04/2022 11:27, Sebastian Mattheis wrote: > > Dear Flink developers, > > > > I would like to open a discussion on FLIP 217 [1] for an extension of > > Watermark Alignment to perform alignment also in SplitReaders. To do so, > > SplitReaders must be able to suspend and resume reading from split > sources > > where the SourceOperator coordinates and controlls suspend and resume. To > > gather information about current watermarks of the SplitReaders, we > extend > > the internal WatermarkOutputMulitplexer and report watermarks to the > > SourceOperator. > > > > There is a PoC for this FLIP [2], prototyped by Arvid Heise and revised > and > > reworked by Dawid Wysakowicz (He did most of the work.) and me. The > changes > > are backwards compatible in a way that if affected components do not > > support split alignment the behavior is as before. > > > > Best, > > Sebastian > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-217+Support+watermark+alignment+of+source+splits > > [2] https://github.com/dawidwys/flink/tree/aligned-splits > > > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[DISCUSS] Planning Flink 1.16
Hi everyone, With Flink 1.15 about to be released, the community has started planning & developing features for the next release, Flink 1.16. As such, I would like to start a discussion around managing this release. Specifically, Chesnay & myself would like to volunteer as release managers. Our focus as release managers would be * to propose a release timeline * to provide an overview of all ongoing development threads and ideally their current status to the community * to keep an eye on build stability * facilitate release testing * to do the actual release incl. communication (blog post, etc.) Is anyone else interested in acting as a release manager for Flink 1.16? If so, we are happy to make this a joint effort. Besides the question of who will act as a release manager, I think, we can already use this thread to align on a timeline. For collecting features and everything else, we would start a dedicated threads shortly. Given Flink 1.15 will be released in the next days, and aiming for a 4 months release cycle including stabilization, this would mean *feature freeze at the end of July*. The exact date could be determined later. Any thoughts on the timeline.? Looking forward to your thoughts! Thanks, Chesnay & Konstantin
Re: [DISCUSS] FLIP-224: Blacklist Mechanism
Hi Lijie,
hm, maybe the following is more appropriate in that case
POST: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}:merge
Best,
Konstantin
Am Mo., 16. Mai 2022 um 07:05 Uhr schrieb Lijie Wang <
wangdachui9...@gmail.com>:
> Hi Konstantin,
> thanks for your feedback.
>
> From what I understand, PUT should be idempotent. However, we have a
> *timeout* field in the request. This means that initiating the same request
> at two different times will lead to different resource status (timestamps
> of the items to be removed will be different).
>
> Should we use PUT in this case? WDYT?
>
> Best,
> Lijie
>
> Konstantin Knauf 于2022年5月13日周五 17:20写道：
>
> > Hi Lijie,
> >
> > wouldn't the REST API-idiomatic way for an update/replace be a PUT on the
> > resource?
> >
> > PUT: http://{jm_rest_address:port}/blocklist/taskmanagers/{id}
> >
> > Best,
> >
> > Konstantin
> >
> >
> >
> > Am Fr., 13. Mai 2022 um 11:01 Uhr schrieb Lijie Wang <
> > wangdachui9...@gmail.com>:
> >
> > > Hi everyone,
> > >
> > > I've had an offline discussion with Becket Qin and Zhu Zhu, and made
> the
> > > following changes on REST API:
> > > 1. To avoid ambiguity, *timeout* and *endTimestamp* can only choose
> one.
> > If
> > > both are specified, will return error.
> > > 2. If the specified item is already there, the *ADD* operation has two
> > > behaviors: *return error*(default value) or *merge/update*, and we
> add a
> > > flag to the request body to control it. You can find more details
> "Public
> > > Interface" section.
> > >
> > > If there is no more feedback, we will start the vote thread next week.
> > >
> > > Best,
> > > Lijie
> > >
> > > Lijie Wang 于2022年5月10日周二 17:14写道：
> > >
> > > > Hi Becket Qin,
> > > >
> > > > Thanks for your suggestions. I have moved the description of
> > > > configurations, metrics and REST API into "Public Interface" section,
> > and
> > > > made a few updates according to your suggestion. And in this FLIP,
> > there
> > > > no public java Interfaces or pluggables that users need to implement
> by
> > > > themselves.
> > > >
> > > > Answers for you questions:
> > > > 1. Yes, there 2 block actions: MARK_BLOCKED and.
> > > > MARK_BLOCKED_AND_EVACUATE_TASKS (has renamed). Currently, block items
> > can
> > > > only be added through the REST API, so these 2 action are mentioned
> in
> > > the
> > > > REST API part (The REST API part has beed moved to public interface
> > now).
> > > > 2. I agree with you. I have changed the "Cause" field to String, and
> > > allow
> > > > users to specify it via REST API.
> > > > 3. Yes, it is useful to allow different timeouts. As mentioned above,
> > we
> > > > will introduce 2 fields : *timeout* and *endTimestamp* into the ADD
> > REST
> > > > API to specify when to remove the blocked item. These 2 fields are
> > > > optional, if neither is specified, it means that the blocked item is
> > > > permanent and will not be removed. If both are specified, the minimum
> > of
> > > > *currentTimestamp+tiemout *and* endTimestamp* will be used as the
> time
> > to
> > > > remove the blocked item. To keep the configurations more minimal, we
> > have
> > > > removed the *cluster.resource-blocklist.item.timeout* configuration
> > > > option.
> > > > 4. Yes, the block item will be overridden if the specified item
> already
> > > > exists. The ADD operation is *ADD or UPDATE*.
> > > > 5. Yes. On JM/RM side, all the blocklist information is maintained in
> > > > JMBlocklistHandler/RMBlocklistHandler. The blocklist handler(or
> > > abstracted
> > > > to other interfaces) will be propagated to different components.
> > > >
> > > > Best,
> > > > Lijie
> > > >
> > > > Becket Qin 于2022年5月10日周二 11:26写道：
> > > >
> > > >> Hi Lijie,
> > > >>
> > > >> Thanks for updating the FLIP. It looks like the public interface
> > section
> > > >> did not fully reflect all the user sensible behavior and API. Can
> you
> > > put
> > > >> everything that users may be aware of there? That would include the
> > REST
> > >
