Re: [ANNOUNCE] Apache Flink 1.14.3 released

Congrats! Thanks Thomas & Martijn for driving the release and to everyone that contributed to it. Best, D. On Thu, Jan 20, 2022 at 11:34 AM Etienne Chauchot wrote: > Congrats to everyone involved ! > > Etienne > > Le 20/01/2022 à 05:29, Thomas Weise a écrit : > > The Apache Flink community is

Re: [VOTE] FLIP-201: Persist local state in working directory

+1 (non-binding) D. On Thu, Jan 20, 2022 at 10:14 AM Chesnay Schepler wrote: > +1 (binding) > > On 19/01/2022 17:23, Matthias Pohl wrote: > > +1 (binding) > > > > Best, > > Matthias > > > > On Mon, Jan 10, 2022 at 2:53 PM Till Rohrmann > wrote: > > > >> +1 (binding) > >> > >> Cheers, > >>

Re: [VOTE] FLIP-205: Support cache in DataStream for Batch Processing

+1 (non-binding) D. On Thu 20. 1. 2022 at 4:09, Xuannan Su wrote: > Hi devs, > > I would like to start the vote for FLIP-205 [1], which was discussed and > reached a consensus in the discussion thread [2]. > > The vote will be open for at least 72h, unless there is an objection or not > enough

Re: [DISCUSS] FLIP-203: Incremental savepoints

th as smooth as possible for end users as this may allow for faster adoption of new Flink versions, but there are new problems this might introduce and we should be aware of them. D. On Wed, Jan 19, 2022 at 9:04 AM Piotr Nowojski wrote: > Hi David, > > I didn't mean "best effort

[jira] [Created] (FLINK-25694) GSON/Alluxio Vulnerability

David Perkins created FLINK-25694: - Summary: GSON/Alluxio Vulnerability Key: FLINK-25694 URL: https://issues.apache.org/jira/browse/FLINK-25694 Project: Flink Issue Type: Bug

Re: [DISCUSS] FLIP-203: Incremental savepoints

t users operating a long-running, stateful Apache Flink > application > > have been in the situation, where a graceful "stop" was not possible > > anymore, because the Job was unable to take a Savepoint. This could be, > > because the Job is frequently restarting (e.g. poi

Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch Processing

> From:Xuannan Su > Send Time:2022 Jan. 17 (Mon.) 13:00 > To:dev > Subject:Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch > Processing > > Hi David, > > Thanks for pointing out the FLIP-187. After reading the FLIP, I think it > can

Re: [DISCUSS] Move Flink website to privacy friendly Analytics solution

+1, thanks for driving this Martijn On Fri 14. 1. 2022 at 15:01, Chesnay Schepler wrote: > +1 > > On 14/01/2022 14:47, Till Rohrmann wrote: > > Hi Martijn, > > > > big +1 for this effort. Thanks a lot for pushing this initiative forward! > > > > Cheers, > > Till > > > > On Fri, Jan 14, 2022 at

Re: [DISCUSS] FLIP-203: Incremental savepoints

> I have a very similar question to State Processor API. Is it the same scenario in this case? > Should it also be working with checkpoints but might be just untested? I have used the State Processor API with aligned, full checkpoints. There it has worked just fine. David On Thu, Jan 13

Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch Processing

e some intermediate results, so users can use cache to avoid > > > > re-computation. The intermediate result is not meaningful outside of > > > > the application. And the cache will be discarded after the > application > > > > is finished.

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

> Best Regards, > Yu > > [1] https://github.com/apache/flink-statefun > [2] https://github.com/apache/flink-ml > [3] https://github.com/apache/flink-connectors > > > On Mon, 10 Jan 2022 at 10:52, Jingsong Li wrote: > > > Hi David, thanks for your suggestion. > >

Re: [VOTE] Create a separate sub project for FLIP-188: flink-store

+1 for the separate repository under the Flink umbrella as we've already started creating more repositories with connectors, would it be possible to re-use the same build infrastructure for this one? (eg. shared set of Gradle plugins that unify the build experience)? Best, D. On Fri, Jan 7,

Re: [ANNOUNCE] Apache Flink ML 2.0.0 released

Great job! <3 Thanks Dong and Yun for managing the release and big thanks to everyone who has contributed! Best, D. On Fri, Jan 7, 2022 at 2:27 PM Yun Gao wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink ML 2.0.0. > > > > Apache Flink ML provides API

Re: [DISCUSS] FLIP-201: Persist local state in working directory

+1 the general direction here seems pretty solid D. On Wed, Jan 5, 2022 at 11:57 AM Till Rohrmann wrote: > If there is no other larger feedback, I would start the vote soonish. > > Cheers, > Till > > On Thu, Dec 30, 2021 at 4:28 PM Till Rohrmann > wrote: > >

Re: [DISCUSS] Drop Gelly

Most of the inquiries I've had about Gelly in recent memory have been from folks looking for a streaming solution, and it's only been a handful. +1 for dropping Gelly David On Mon, Jan 3, 2022 at 2:41 PM Till Rohrmann wrote: > I haven't seen any changes or requests to/for Gelly in ages. He

Re: [DISCUSS] Changing the minimal supported version of Hadoop

Rohrmann wrote: > If there are no users strongly objecting to dropping Hadoop support for < > 2.8, then I am +1 for this since otherwise we won't gain a lot as Xintong > said. > > Cheers, > Till > > On Wed, Dec 22, 2021 at 10:33 AM David Morávek wrote: > > > Agreed,

Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch Processing

[1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-188%3A+Introduce+Built-in+Dynamic+Table+Storage > > > On Wed, Dec 29, 2021 at 7:53 PM Xuannan Su wrote: > > > Hi David, > > > > Thanks for sharing your thoughts. > > > > You are right

Re: [DISCUSS] FLIP-203: Incremental savepoints

Hi Piotr, does this mean that we need to keep the checkpoints compatible across minor versions? Or can we say, that the minor version upgrades are only guaranteed with canonical savepoints? My concern is especially if we'd want to change layout of the checkpoint. D. On Wed, Dec 29, 2021 at

Re: [NOTICE] Table API is now Scala-free by introducing a flink-table-planner-loader

Great job! This brings the scala-free effort close to the finish line! D. On Thu, Dec 30, 2021 at 3:08 PM Timo Walther wrote: > Hi everyone, > > The new module flink-table-planner-loader replaces > flink-table-planner_2.12 and avoids the need for a specific Scala > version in downstream

Re: [DISCUSS] FLIP-201: Persist local state in working directory

Hi Till, thanks for drafting the FLIP, it looks really good. I did a quick pass over the PR and it seems to be heading in a right direction. It might be required to introduce a graceful shutdown of the TaskExecutor > in order to support proper cleanup of resources. > This is actively being

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

e JobManager itself. I think the user code > (like a webserver) should run outside of Flink (like via a sidecar) and use > only the provided interfaces to communicate. > > I would like to get @David Morávek opinion on the > technical part. > > Best regards, > > Martijn > >

Re: [DISCUSS] FLIP-205: Support cache in DataStream for Batch Processing

Hi Xuannan, thanks for drafting this FLIP. One immediate thought, from what I've seen for interactive data exploration with Spark, most people tend to use the higher level APIs, that allow for faster prototyping (Table API in Flink's case). Should the Table API also be covered by this FLIP?

[DISCUSS] Slimmed down docker images.

Hi, I did some quick prototyping on the slimmed down docker images, and I was able to cut the docker image size by ~40% with a minimum effort [1] (using a multi-stage build + trimming examples / opt + using slimmed down JRE image). I think this might be a low hanging fruit for reducing MTTR in

Re: [DISCUSS] Releasing Flink 1.14.3

> equality of the same (boxed) numeric values returns false -> @Caizhi Weng > any update or thoughts on this? > > > > Best regards, > > > > Martijn > > > > [1] https://lists.apache.org/thread/r0xhs9x01k8hnm0hyq2kk4ptrhkzgdw9 > > [2] https://fl

Re: [DISCUSS] Changing the minimal supported version of Hadoop

Agreed, if we drop the CI for lower versions, there is actually no point of having safeguards as we can't really test for them. Maybe one more thought (it's more of a feeling), I feel that users running really old Hadoop versions are usually slower to adopt (they most likely use what the current

Re: [DISCUSS] Changing the minimal supported version of Hadoop

CC user@f.a.o Is anyone aware of something that blocks us from doing the upgrade? D. On Tue, Dec 21, 2021 at 5:50 PM David Morávek wrote: > Hi Martijn, > > from person experience, most Hadoop users are lagging behind the release > lines by a lot, because upgrading a Ha

Re: [DISCUSS] Changing the minimal supported version of Hadoop

quot; APIs in the code. As for Till's concern, we can still wrap the reflection based logic, to be skipped in case of "NoClassDefFound" instead of "ClassNotFound" as we do now. D. On Tue, Dec 14, 2021 at 5:23 PM Martijn Visser wrote: > Hi David, > > Thanks for

Re: [DISCUSS] Releasing Flink 1.14.3

/thread/r0xhs9x01k8hnm0hyq2kk4ptrhkzgdw9 > [2] https://flink.apache.org/news/2021/12/16/log4j-patch-releases.html > > On Thu, 9 Dec 2021 at 17:21, David Morávek wrote: > > > Hi Martijn, I've just opened a backport PR [1] for FLINK-23946 [2]. > > > > [1] https://github.com

Re: [DISCUSS] FLIP-200: Support Multiple Rule and Dynamic Rule Changing (Flink CEP)

not sure about one of the rejected alternatives: > > > > > > > Have each subtask of an operator make the update on their own > > > > > >- > > > > > >It is hard to achieve consistency. > > >- > > > > >

Re: [VOTE] FLIP-198: Working directory for Flink processes

+1 (non-binding) Best, D. On Thu, Dec 16, 2021 at 4:30 PM Chesnay Schepler wrote: > +1 > > On 16/12/2021 14:42, Till Rohrmann wrote: > > Hi everyone, > > > > I'd like to start a vote on FLIP-198: Working directory for Flink > processes > > [1] which has been discussed in this thread [2]. > > >

Re: [DISCUSS] FLIP-198: Working directory for Flink processes

Hi Till, thanks for drafting this FLIP, I think it's really a valuable improvement. Agreed with Yang, that YARN / k8s implementation should be out of scope of this FLIP. Just few notes on the possible integrations: For k8s, I think we can also benefit from this FLIP without StatefulSet. If the

Re: [DISCUSS] Strong read-after-write consistency of Flink FileSystems

Any other thoughts on the topic? If there are no concerns, I'd continue with creating a FLIP for changing the "written" contract of the Flink FileSystems to reflect this. Best, D. On Wed, Dec 8, 2021 at 5:53 PM David Morávek wrote: > Hi Martijn, > > I simply wasn'

[DISCUSS] Changing the minimal supported version of Hadoop

Hi, I'd like to start a discussion about upgrading a minimal Hadoop version that Flink supports. Even though the default value for `hadoop.version` property is set to 2.8.3, we're still ensuring both runtime and compile compatibility with Hadoop 2.4.x with the scheduled pipeline[1]. Here is

Re: [DISCUSS] Deprecate MapR FS

+1, agreed with Seth's reasoning. There has been no real activity in MapR FS module for years [1], so the eventual users should be good with using the jars from the older Flink versions for quite some time [1] https://github.com/apache/flink/commits/master/flink-filesystems/flink-mapr-fs Best,

Re: [DISCUSS] Releasing Flink 1.14.1

ish it as soon as possible. > > > >> >> > > > >> >> Best, > > > >> >> Jingsong > > > >> >> > > > >> >> On Fri, Dec 3, 2021 at 10:25 PM Fabian Paul > wrote: > > > >> >>

Re: [VOTE] FLIP-194: Introduce the JobResultStore

Xintong Song > > > > > > > > > > > > > > > > On Mon, Dec 6, 2021 at 5:02 PM Till Rohrmann > > > wrote: > > > > > > > > > +1 (binding) > > > > > > > > > > Cheers, > > > > > Till >

Re: [DISCUSS] Strong read-after-write consistency of Flink FileSystems

microsoft.com/en-us/blog/a-closer-look-at-azure-data-lake-storage-gen2/ D. On Wed, Dec 8, 2021 at 4:34 PM Martijn Visser wrote: > Hi David, > > Just to be sure, since you've already included Azure Blob Storage, but did > you deliberately skip Azure Data Lake Store Gen2? That's currentl

Re: [VOTE] Deprecate Java 8 support

+1 (non-binding) On Mon, Dec 6, 2021 at 4:55 PM Ingo Bürk wrote: > +1 (non-binding) > > > Ingo > > On Mon, Dec 6, 2021 at 4:44 PM Chesnay Schepler > wrote: > > > Hello, > > > > after recent discussions on the dev > > and > >

[DISCUSS] Strong read-after-write consistency of Flink FileSystems

Hi Everyone, as outlined in FLIP-194 discussion [1], for the future directions of Flink HA services, I'd like to verify my thoughts around guarantees of the distributed filesystems used with Flink. Currently some of the services (*JobGraphStore*, *CompletedCheckpointStore*) are implemented using

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

for 1.16 ;) On Mon, Dec 6, 2021 at 11:03 AM Yang Wang wrote: > Thanks for the fruitful discussion. I also hope that we could remove all > the pointers in the HA store(ZK, ConfigMap) in the future. > After then, we only rely on the ZK/ConfigMap for leader election/retrieval. > >

[VOTE] FLIP-194: Introduce the JobResultStore

Hi everyone, I'd like to open a vote on FLIP-194: Introduce the JobResultStore [1] which has been discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. [1]

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

> I have no more concerns and +1 for the FLIP. > > Thanks, > Zhu > > Xintong Song 于2021年12月1日周三 下午12:56写道： > > > @David, > > > > Thanks for the clarification. > > > > No more concerns from my side. +1 for this FLIP. > > > > Thank you~ &g

Re: [ANNOUNCE] New Apache Flink Committer - Matthias Pohl

Congrats Matthias, well deserved ;) Best, D. On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz wrote: > Congratulations Matthias! Really well deserved! > > Best, > > Dawid > > On 02/12/2021 16:53, Nicolaus Weidner wrote: > > Congrats Matthias, well deserved! > > > > Best, > > Nico > > > > On

Re: [ANNOUNCE] New Apache Flink Committer - Ingo Bürk

Congrats Ingo, well deserved ;) Best, D. On Thu, Dec 2, 2021 at 5:17 PM Dawid Wysakowicz wrote: > Congratulations Ingo! Happy to have you onboard as a committer! > > Best, > > Dawid > > On 02/12/2021 17:14, Francesco Guardiani wrote: > > Congrats Ingo! > > > > On Thu, Dec 2, 2021 at 4:58 PM

Re: FLink Accessing two hdfs cluster

uot;chenqizhu" 写道： > > Hi David, > >I'm glad you can reply. > >--this exception doesn't seem to come from Flink, but rather from a > YARN container bootstrap. >--In this case the exception happens before any Flink code is executed > by the NodeManager. &

Re: [DISCUSS] FLIP-194: Introduce the JobResultStore

LeaderProcessFactory and referenced > > in DispatcherLeaderProcess. The Dispatcher will get the information about > > recovered JobGraphs (as it's currently done on master) and the JobResult > of > > globally-terminated jobs that are still marked as "dirty" by the > > JobRes

Re: [DISCUSS] Releasing Flink 1.14.1

-24038 and I don't see much > happening there, so I also expect that this would move to Flink 1.15. > David, could you confirm? > Till has prepared a prototype for this, but the change is too invasive to be introduced in a patch versions. We're moving this to 1.15. Best, D. On Thu, Nov 25, 202

Re: [DISCUSS] Deprecate Java 8 support

Thank you Chesnay for starting the discussion! This will generate bit of a work for some users, but it's a good thing to keep moving the project forward. Big +1 for this. Jingsong: Receiving this signal, the user may be unhappy because his application > may be all on Java 8. Upgrading is a big

[DISCUSS] FLIP-194: Introduce the JobResultStore

a FLIP-194 [2], which outlines the design and reasoning behind this new component. [1] https://issues.apache.org/jira/browse/FLINK-11813 [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=195726435 We're looking forward for your feedback ;) Best, Matthias, Mika and David

Re: [DISCUSS] Conventions on assertions to use in tests

med > at > >> > switching over to Junit5 [1, 2]. @Arvid Heise > knows > >> > more about the current status. > >> > > >> > Personally, I don't have a strong preference for which testing tools > to > >> > use. The important bit is that we agree as

Re: Flink support for OrderedListState

Hi Reuven, this would be a great addition to the Flink Runner and could help with broader adoption ;) to make an effective implementation that works well across different state backends, this will most likely require adding a new primitive state type to the Flink's state backend ecosystem. I'll

Re: [ANNOUNCE] New Apache Flink Committer - Fabian Paul

Congratulations Fabian!! Best, D. On Mon, Nov 15, 2021 at 5:10 PM Leonard Xu wrote: > Congratulations Fabian! > > > 在 2021年11月15日，22:50，Roman Khachatryan 写道： > > > > Congratulations Fabian! > > > > Regards, > > Roman > > > > On Mon, Nov 15, 2021 at 3:26 PM Dawid Wysakowicz > wrote: > >> > >>

Re: [DISCUSS] Conventions on assertions to use in tests

For what it's worth, I recently rewrote all of the tests in flink-training to use assertj, removing a mixture of junit4 assertions and hamcrest in the process. I chose assertj because I found it to be more expressive and made the tests more readable. +1 from me David On Fri, Nov 12, 2021 at 10

Re: [ANNOUNCE] Documentation now available at nightlies.apache.org

Also big thanks to Gavin for setting the redirects up! ;) Best, D. On Thu, Nov 11, 2021 at 2:34 AM Leonard Xu wrote: > Nice ! > Thanks chesnay for the continuous effort. > > > 在 2021年11月11日，06:58，Chesnay Schepler 写道： > > > > A redirect from ci.apache.org to nightlies.apache.org has been set

Re: [VOTE] FLIP-187: Adaptive Batch Job Scheduler

Thanks for the FLIP, this is going to be a great improvement to the batch execution. +1 (non-binding) Best, D. On Tue, Nov 9, 2021 at 1:05 AM Guowei Ma wrote: > Thanks for your excellent FLIP! > +1 binding > > Lijie Wang 于2021年11月8日 周一下午2:53写道： > > > Hi all, > > > > I would like to start the

Re: [NOTICE] Please keep flink-examples up to date

Hi Seth, thanks for bringing this up. What do you think about placing a safeguard for this? We should be able to setup java compiler for examples to fail on any usage of deprecated APIs. Something along the lines of: maven-compiler-plugin ... compile process-sources

Re: Elasticsearch6 connector in flink stand alone

Hi Ravi, I'm moving this thread to the user@flink mailing list, which is designed for these type of questions. For your issue, I don't think it's related to the elasticsearch integration. It seems like there is something wrong with your log4j setup. Either you have a conflicting log4j jars on

Re: [ANNOUNCE] Documentation now available at nightlies.apache.org

Hi Yun, I'm reaching out to the ASF infra to check the status of redirection efforts. The old documentation still uses 1.13.2 as the stable release, which could be really confusing :( Best, D. On Fri, Nov 5, 2021 at 4:44 AM Yun Tang wrote: > Hi Chesnay, > > It seems that the redirection has

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

Hi Fabian, thanks for drafting the FLIP! This is a really nice and useful topic to target ;) Few thoughts on the option 2) The file compaction is by definition quite costly IO bound operation. If I understand the proposal correctly, the aggregation itself would run during operator (aggregator)

Re: [DISCUSS] FLIP-187: Adaptive Batch Job Scheduler

Hi, thanks for drafting the FLIP, Lijie and Zhu Zhu. It already looks pretty solid and it will be a really great improvement to the batch scheduling. I'd second to the Till's feedback, especially when it comes to the consistent behavior between different deployment types / schedulers. What I'm

Re: [Discuss] Planning Flink 1.15

The contributor availability argument makes perfect sense, +1 for moving the feature freeze to 6/2. Best, D. On Thu, Oct 28, 2021 at 9:39 AM Till Rohrmann wrote: > I think it is important that most of the people who contribute new features > are available during the testing/stabilization

Re: [DISCUSS] Creating an external connector repository

We are mostly talking about the freedom this would bring to the connector authors, but we still don't have answers for the important topics: - How exactly are we going to maintain the high quality standard of the connectors? - How would the connector release cycle to look like? Is this going to

Re: [NOTICE] CiBot improvements

Nice! Thanks for the effort Chesnay, this is really a huge step forward! Best, D. On Mon, Oct 11, 2021 at 6:02 AM Xintong Song wrote: > Thanks for the effort, @Chesnay. This is super helpful. > > @Jing, > Every push to the PR branch should automatically trigger an entire new > build.

[jira] [Created] (FLINK-24478) gradle quickstart is out-of-date

David Anderson created FLINK-24478: -- Summary: gradle quickstart is out-of-date Key: FLINK-24478 URL: https://issues.apache.org/jira/browse/FLINK-24478 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-176: Unified Iteration to Support Algorithms (Flink ML)

Hi Yun, I did a quick pass over the design doc and it addresses all of the problems with the current iterations I'm aware of. It's great to see that you've been able to workaround the need of vectorized watermarks by giving up nested iterations (which IMO is more of an academic concept than

Re: [ANNOUNCE] Apache Flink 1.14.0 released

Thanks Dawid, Xintong and Joe for being really great release managers and everyone else who helped making this release possible! <3 Best, D. On Thu, Sep 30, 2021 at 3:31 PM Till Rohrmann wrote: > Thanks a lot for being our release managers Dawid, Xintong and Joe. Also a > big thanks to

Re: The Apache Flink should pay more attention to ensuring API compatibility.

ter/pom.xml#L2014:L2084 > > wt., 28 wrz 2021 o 15:59 David Morávek napisał(a): > > > This is a super interesting topic and there is already a great > discussion. > > Here are few thoughts: > > > > - There is a delicate balance between fast delivery of

Re: The Apache Flink should pay more attention to ensuring API compatibility.

This is a super interesting topic and there is already a great discussion. Here are few thoughts: - There is a delicate balance between fast delivery of the new features and API stability. Even though we should be careful with breaking evolving interfaces, it shouldn't stop us from making fast

Re: Beam with Flink runner - Issues when writing to S3 in Parquet Format

Hi Sandeep, Jan has already provided pretty good guidelines for getting more context on the issue ;) Because this is not for the first time, I would like to raise awareness, that it's not OK to send a user related question to four Apache mailing list (that I know of). Namely: -

Re: [DISCUSS] Automated architectural tests

Hi Ingo, +1 for this effort. This could automate lot of "written rules" that are easy to forget about / not to be aware of (such as that each test should extend the TestLogger as Till has already mentioned). I went trough your examples and ArchUnit looks really powerful and expressive while

[jira] [Created] (FLINK-24118) enable TaxiFareGenerator to produce a bounded stream

David Anderson created FLINK-24118: -- Summary: enable TaxiFareGenerator to produce a bounded stream Key: FLINK-24118 URL: https://issues.apache.org/jira/browse/FLINK-24118 Project: Flink

[jira] [Created] (FLINK-23926) change TaxiRide data model to have a single timestamp

David Anderson created FLINK-23926: -- Summary: change TaxiRide data model to have a single timestamp Key: FLINK-23926 URL: https://issues.apache.org/jira/browse/FLINK-23926 Project: Flink

Were Bundles meant to be internal?

, and I'm not convinced this is a good idea. E.g., see https://stackoverflow.com/questions/68811184/pre-shuffle-aggregation-in-flink . David

[jira] [Created] (FLINK-23840) Confusing message from MemCheckpointStreamFactory#checkSize

David Anderson created FLINK-23840: -- Summary: Confusing message from MemCheckpointStreamFactory#checkSize Key: FLINK-23840 URL: https://issues.apache.org/jira/browse/FLINK-23840 Project: Flink

[DISCUSS] Merging FLINK-21867 after feature freeze

Hi, We have a small UI change [1][2] that we'd like to get merged into 1.14 [3]. The change allows displaying concurrent exceptions alongside the main exception in job's exception history (these were already present in the Rest API). This could bring a nice improvement for debugging of failed

Re: Unable to read state Witten by Beam application with Flink runner using Flink's State Processor API

David Morávek wrote: > Hi Sandeep, thanks for the example, I'll take a look into it and will get > back to you ;) > > On Tue, Aug 3, 2021 at 9:44 PM Kathula, Sandeep < > sandeep_kath...@intuit.com> wrote: > >> Hi David, >> Thanks for the rep

[jira] [Created] (FLINK-23653) improve training exercises and tests so they are better examples

David Anderson created FLINK-23653: -- Summary: improve training exercises and tests so they are better examples Key: FLINK-23653 URL: https://issues.apache.org/jira/browse/FLINK-23653 Project: Flink

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

, David On Wed, Aug 4, 2021 at 8:08 AM Stephan Ewen wrote: > Hi all! > > *!!! If you are a big user of the Embedded RocksDB State Backend and have > performance sensitive workloads, please read this !!!* > > I want to quickly raise some awareness for a RocksDB version upgra

Re: Unable to read state Witten by Beam application with Flink runner using Flink's State Processor API

Hi Sandeep, thanks for the example, I'll take a look into it and will get back to you ;) On Tue, Aug 3, 2021 at 9:44 PM Kathula, Sandeep wrote: > Hi David, > Thanks for the reply. I tried with Beam 2.29 and Flink > 1.12 and still getting NullPointerException like

Re: Unable to read state Witten by Beam application with Flink runner using Flink's State Processor API

Hi Sandeep, In general I'd say it will be tricky to read Beam state this way as it doesn't use Flink primitives, but it's writing state in custom binary format (it can be de-serialized, but it's not easy to put all of the pieces together). Can you please share an example code of how you're

Re: [DISCUSS] Address deprecation warnings when upgrading dependencies

I know that sonar can report back by adding a comment to the issue (very similar way the FlinkBot does) and can block the merge (probably using check runs api [1]), if some quality gate fails. I was never setting it up, so I'd need to take a closer look. This is a feature set we were using on GH

Re: [DISCUSS] Address deprecation warnings when upgrading dependencies

> > For implementing this in practice, we could also extend our CI pipeline a > bit, and count the number of deprecation warnings while compiling Flink. > We would hard-code the current number of deprecations and fail the build if > that number increases. Maybe we could leverage sonar cloud

[jira] [Created] (FLINK-23128) Translate update to operations playground docs to Chinese

David Anderson created FLINK-23128: -- Summary: Translate update to operations playground docs to Chinese Key: FLINK-23128 URL: https://issues.apache.org/jira/browse/FLINK-23128 Project: Flink

[jira] [Created] (FLINK-23100) Update pyflink walkthrough playground for 1.13

David Anderson created FLINK-23100: -- Summary: Update pyflink walkthrough playground for 1.13 Key: FLINK-23100 URL: https://issues.apache.org/jira/browse/FLINK-23100 Project: Flink Issue

[jira] [Created] (FLINK-23099) Update table walkthrough playground for 1.13

David Anderson created FLINK-23099: -- Summary: Update table walkthrough playground for 1.13 Key: FLINK-23099 URL: https://issues.apache.org/jira/browse/FLINK-23099 Project: Flink Issue Type

[jira] [Created] (FLINK-23098) Update operations playground for 1.13

David Anderson created FLINK-23098: -- Summary: Update operations playground for 1.13 Key: FLINK-23098 URL: https://issues.apache.org/jira/browse/FLINK-23098 Project: Flink Issue Type: Sub

trying (and failing) to update pyflink-walkthrough for Flink 1.13

trying to write to kafka. Not sure what's wrong? Any suggestions? See [1] to review what I tried. Best, David [1] https://github.com/alpinegizmo/flink-playgrounds/commit/777274355ba04de6d8c8f1308b24be99ec86a0d6 21:40 $ docker-compose logs -f generator Attaching to pyflink-walkthrough_generator_1

[jira] [Created] (FLINK-23059) Update playgrounds for Flink 1.13

David Anderson created FLINK-23059: -- Summary: Update playgrounds for Flink 1.13 Key: FLINK-23059 URL: https://issues.apache.org/jira/browse/FLINK-23059 Project: Flink Issue Type

[jira] [Created] (FLINK-22948) Scala example for toDataStream does not compile

David Anderson created FLINK-22948: -- Summary: Scala example for toDataStream does not compile Key: FLINK-22948 URL: https://issues.apache.org/jira/browse/FLINK-22948 Project: Flink Issue

[jira] [Created] (FLINK-22894) Window Top-N should allow n=1

David Anderson created FLINK-22894: -- Summary: Window Top-N should allow n=1 Key: FLINK-22894 URL: https://issues.apache.org/jira/browse/FLINK-22894 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-22868) Update training exercises for 1.13

David Anderson created FLINK-22868: -- Summary: Update training exercises for 1.13 Key: FLINK-22868 URL: https://issues.apache.org/jira/browse/FLINK-22868 Project: Flink Issue Type

[jira] [Created] (FLINK-22737) Add support for CURRENT_WATERMARK to SQL

David Anderson created FLINK-22737: -- Summary: Add support for CURRENT_WATERMARK to SQL Key: FLINK-22737 URL: https://issues.apache.org/jira/browse/FLINK-22737 Project: Flink Issue Type: Sub

Re: [DISCUSS] Releasing Flink 1.13.1

Hi Konstantin, Would it be possible to add FLINK-22646 [1] into the release? This is a regression, that we need to workaround in order to support 1.13.x in Apache Beam [2]. Best, D. [1] https://issues.apache.org/jira/browse/FLINK-22646 [2] https://github.com/apache/beam/pull/14719 On Tue, May

Re: [DISCUSS] Watermark propagation with Sink API

Hi Eron, Thanks for starting this discussion. I've been thinking about this recently as we've run into "watermark related" issues, when chaining multiple pipelines together. My to cents to the discussion: How I like to think about the problem, is that there should an invariant that holds for any

[jira] [Created] (FLINK-22543) layout of exception history tab isn't very usable with Flink SQL

David Anderson created FLINK-22543: -- Summary: layout of exception history tab isn't very usable with Flink SQL Key: FLINK-22543 URL: https://issues.apache.org/jira/browse/FLINK-22543 Project: Flink

Re: [VOTE] Release 1.13.0, release candidate #2

+1 (non-binding) Checks: - I built from source, successfully. - I tested the new backpressure metrics and UI. I found one non-critical bug that's been around for years, and for which a fix has already been merged for 1.13.1 (https://issues.apache.org/jira/browse/FLINK-22489

[jira] [Created] (FLINK-22489) subtask backpressure indicator shows value for entire job

David Anderson created FLINK-22489: -- Summary: subtask backpressure indicator shows value for entire job Key: FLINK-22489 URL: https://issues.apache.org/jira/browse/FLINK-22489 Project: Flink

[jira] [Created] (FLINK-21639) docs still state that AsyncWaitOperator is not chainable

David Anderson created FLINK-21639: -- Summary: docs still state that AsyncWaitOperator is not chainable Key: FLINK-21639 URL: https://issues.apache.org/jira/browse/FLINK-21639 Project: Flink

Re: [VOTE] FLIP-151: Incremental snapshots for heap-based state backend

+1 (non-binding) On Mon, Mar 1, 2021 at 10:12 AM Roman Khachatryan wrote: > Hi everyone, > > since the discussion [1] about FLIP-151 [2] seems to have reached a > consensus, I'd like to start a formal vote for the FLIP. > > Please vote +1 to approve the FLIP, or -1 with a comment. The vote will

Re: [DISCUSS] FLIP-165: Operator's Flame Graphs

for Application Profiling & Debugging, which is more on point. I think it will be confusing if the flame graphs aren't together with the other profilers. David On Tue, Mar 2, 2021 at 11:36 PM Seth Wiesman wrote: > Cool feature +1 > > There is a subsection called monitoring in the opera

<    1   2   3   4   5   6   >