Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congrats to everyone involved! Thanks, Yu & Gary. Best, godfrey Yu Li 于2020年2月13日周四 下午12:57写道： > Hi Kristoff, > > Thanks for the question. > > About Java 11 support, please allow me to quote from our release note [1]: > > Lastly, note that the connectors for Cassandra, Hive, HBase, and Kafka >

Re: AW: How Flink Kafka Consumer works when it restarts

Thanks Theo ! From: "theo.diefent...@scoop-software.de" Date: Thursday, 13 February 2020 at 12:13 AM To: "Hegde, Mahendra" , "user@flink.apache.org" Subject: [External] AW: How Flink Kafka Consumer works when it restarts Hi Mahendra, Flink will regularly create checkpoints or manually trigger

Re: [ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Thanks for the great work, Jincheng. Regards, Dian > 在 2020年2月13日，下午1:32，jincheng sun 写道： > > Hi everyone, > > The Apache Flink community is very happy to announce the release of Apache > Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the > Apache Flink Python API 1.

[ANNOUNCE] Apache Flink Python API(PyFlink) 1.9.2 released

Hi everyone, The Apache Flink community is very happy to announce the release of Apache Flink Python API(PyFlink) 1.9.2, which is the first release to PyPI for the Apache Flink Python API 1.9 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, alw

Re: UI stability at high parallelism

Hi, Richard This is most likely that the Rest Api has timed out, you can try to find some evidence in the jobmanager log. You can provide the full log to help us find the root cause. Best Weihua Hu > 2020年2月13日 09:40，Richard Moorhead 写道： > > When I submit a job to flink session with paralle

Re: [ANNOUNCE] Apache Flink 1.10.0 released

The link for FLINK-14516 is fixed. Thanks for pointing it out Hailu! Best Regards, Yu On Thu, 13 Feb 2020 at 07:19, Hailu, Andreas wrote: > Congrats all! > > > > P.S. I noticed in the release notes that the bullet: > > > > *[FLINK-14516 ] The

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Hi Kristoff, Thanks for the question. About Java 11 support, please allow me to quote from our release note [1]: Lastly, note that the connectors for Cassandra, Hive, HBase, and Kafka 0.8–0.11 have not been tested with Java 11 because the respective projects did not provide Java 11 support at th

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Really great work and thanks everyone involved, especially for the release managers! Best, Zhijiang -- From:Kurt Young Send Time:2020 Feb. 13 (Thu.) 11:06 To:[None] Cc:user ; dev Subject:Re: [ANNOUNCE] Apache Flink 1.10.0 released

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations to everyone involved! Great thanks to Yu & Gary for being the release manager! Best, Kurt On Thu, Feb 13, 2020 at 10:06 AM Hequn Cheng wrote: > Great thanks to Yu & Gary for being the release manager! > Also thanks to everyone who made this release possible! > > Best, Hequn > >

Re: Flink Table API

Hi Flavio, There is 2 main entry points in Table API, one is `TableEnvironment`, another is `StreamTableEnvironment`. - `TableEnvironment` is used for pure Table API & SQL programs. - `StreamTableEnvironment` can be used to convert from/to DataStream. These two interface will be kept in the futur

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great thanks to Yu & Gary for being the release manager! Also thanks to everyone who made this release possible! Best, Hequn On Thu, Feb 13, 2020 at 9:54 AM Rong Rong wrote: > Congratulations, a big thanks to the release managers for all the hard > works!! > > -- > Rong > > On Wed, Feb 12, 2020

Re: UI stability at high parallelism

We also encountered a similar issue internally. cc +huweihua.ckl Richard Moorhead 于2020年2月13日周四 上午9:40写道： > When I submit a job to flink session with parallelism higher than 128, the > job is submitted and renders in the UI but when I view the job itself the > UI starts to rapidly emit errors in

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations, a big thanks to the release managers for all the hard works!! -- Rong On Wed, Feb 12, 2020 at 5:52 PM Yang Wang wrote: > Excellent work. Thanks Gary & Yu for being the release manager. > > > Best, > Yang > > Jeff Zhang 于2020年2月13日周四 上午9:36写道： > >> Congratulations! Really appre

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Excellent work. Thanks Gary & Yu for being the release manager. Best, Yang Jeff Zhang 于2020年2月13日周四 上午9:36写道： > Congratulations! Really appreciated your hard work. > > Yangze Guo 于2020年2月13日周四 上午9:29写道： > >> Thanks, Gary & Yu. Congrats to everyone involved! >> >> Best, >> Yangze Guo >> >> On

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

Hi folks， Thanks everyone for voting. I'm closing the vote now and will post the result as a separate email. Best, Jincheng Xingbo Huang 于2020年2月13日周四 上午9:28写道： > +1 (non-binding) > > - Install the PyFlink by `pip install` [SUCCESS] > - Run word_count.py [SUCCESS] > > Thanks, > Xingbo > > Bec

UI stability at high parallelism

When I submit a job to flink session with parallelism higher than 128, the job is submitted and renders in the UI but when I view the job itself the UI starts to rapidly emit errors in the upper right: Server Response: Unable to load requested file /bad-request. Is this a known issue? Is there a

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations! Really appreciated your hard work. Yangze Guo 于2020年2月13日周四 上午9:29写道： > Thanks, Gary & Yu. Congrats to everyone involved! > > Best, > Yangze Guo > > On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li > wrote: > > > > Congratulations! Great work. > > > > Best, > > Jingsong Lee > > > >

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Thanks, Gary & Yu. Congrats to everyone involved! Best, Yangze Guo On Thu, Feb 13, 2020 at 9:23 AM Jingsong Li wrote: > > Congratulations! Great work. > > Best, > Jingsong Lee > > On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu wrote: >> >> Great news！ >> Thanks everyone involved ! >> Thanks Gary a

Re: [VOTE] Release Flink Python API(PyFlink) 1.9.2 to PyPI, release candidate #1

+1 (non-binding) - Install the PyFlink by `pip install` [SUCCESS] - Run word_count.py [SUCCESS] Thanks, Xingbo Becket Qin 于2020年2月12日周三 下午2:28写道： > +1 (binding) > > - verified signature > - Ran word count example successfully. > > Thanks, > > Jiangjie (Becket) Qin > > On Wed, Feb 12, 2020 at 1

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations! Great work. Best, Jingsong Lee On Wed, Feb 12, 2020 at 11:05 PM Leonard Xu wrote: > Great news！ > Thanks everyone involved ! > Thanks Gary and Yu for being the release manager ! > > Best, > Leonard Xu > > 在 2020年2月12日，23:02，Stephan Ewen 写道： > > Congrats to us all. > > A big pi

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Thanks Gary & Yu and everyone involved, Great work! Regards, Dian > 在 2020年2月12日，下午11:57，Haibo Sun 写道： > > Thanks Gary & Yu. Great work! > > Best, > Haibo > > At 2020-02-12 21:31:00, "Yu Li" wrote: > >The Apache Flink community is very happy to announce the release of Apache > >Flink 1.10.0,

RE: [ANNOUNCE] Apache Flink 1.10.0 released

Congrats all! P.S. I noticed in the release notes that the bullet: [FLINK-14516] The non-credit-based network flow control code was removed, along with the configuration option taskmanager.network.credit.model. Moving forward, Flink will alway

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

Hi Arvid, I use a docker image. Here is the Dockerfile: FROM flink:1.9.1-scala_2.12 RUN mkdir /opt/flink/plugins/flink-s3-fs-hadoop RUN cp /opt/flink/opt/flink-s3-fs-hadoop-1.9.1.jar /opt/flink/plugins/flink-s3-fs-hadoop/ Please let me know if you need more information. On Wed, Feb 12, 2020 at

Size of state for any known production use case

Hi Team, I've done a POC using Flink and planning to give a presentation about my learnings and share the benefits of using Flink. I understand that companies are using Flink to handle Tera Bytes of state, but it would be great if you could point me to any reference of a company using Flink produ

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

Hi David, can you double-check the folder structure of your plugin? It should reside in its own subfolder. Here is an example. flink-dist ├── conf ├── lib ... └── plugins └── s3 └── flink-s3-fs-hadoop.jar I will investigate your error deeply in the next few days but I'd like to have

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

Hi Robert, I couldn't found any previous mention before the NoClassDefFoundError. Here is the full log [1] if you want to look for something more specific. [1] https://www.dropbox.com/s/l8tba6vft08flke/joda.out?dl=0 On Wed, Feb 12, 2020 at 12:45 PM Robert Metzger wrote: > According to this answ

AW: How Flink Kafka Consumer works when it restarts

Hi Mahendra, Flink will regularly create checkpoints or manually triggered savepoints. This is data managed and stored by Flink and that data also contains the kafka offsets. When restarting, you can configure to restart from the last checkpoint and or savepoint. You can additionally configure F

Re: Dedup all data in stream

Hi Kostas We are doing scans on 1000s of resources which we want to do it at some interval which is currently 2 mins. Scanning is the same operation we want to perform at every 2 minutes to check if everything is ok or not. Sometimes this scan operation takes lot of time which results in lag an

How Flink Kafka Consumer works when it restarts

Hi All, I am bit confused on Flink kafka consumer working. I read that Flink stores the kafka message offset in checkpoint and uses it in case if it restarts. Question is when exactly Flink is committing about successful consumption confirmation to kafka broker? And when Flink job restarts will

Flink Table API

Hi to all, I was trying to use the new Table API using the new Blink planner but I figured out that they do not use exactly the same APIs..for example I can't go back and forth from Tables to Dataset/Datastream anymore (using tableEnv.toDataset for example). Is this a temporary behaviour or this fu

Re:[ANNOUNCE] Apache Flink 1.10.0 released

Thanks Gary & Yu. Great work! Best, Haibo At 2020-02-12 21:31:00, "Yu Li" wrote: >The Apache Flink community is very happy to announce the release of Apache >Flink 1.10.0, which is the latest major release. > >Apache Flink® is an open-source stream processing framework for >distributed, high-p

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Hi all, I have a small question regarding 1.10 Correct me if I'm wrong, but 1.10 should support Java 11 right? If so, then I noticed that docker images [1] referenced in [2] are still based on openjdk8 not Java 11. Whats up with that? P.S. Congrats on releasing 1.10 ;) [1] https://github.com/a

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congrats team and a big thank you to the release managers! Am Mi., 12. Feb. 2020 um 16:33 Uhr schrieb Timo Walther : > Congratualations everyone! Great stuff :-) > > Regards, > Timo > > > On 12.02.20 16:05, Leonard Xu wrote: > > Great news！ > > Thanks everyone involved ! > > Thanks Gary and Yu fo

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratualations everyone! Great stuff :-) Regards, Timo On 12.02.20 16:05, Leonard Xu wrote: Great news！ Thanks everyone involved ! Thanks Gary and Yu for being the release manager ! Best, Leonard Xu 在 2020年2月12日，23:02，Stephan Ewen 写道： Congrats to us all. A big piece of work, nicely don

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great news！ Thanks everyone involved ! Thanks Gary and Yu for being the release manager ! Best, Leonard Xu > 在 2020年2月12日，23:02，Stephan Ewen 写道： > > Congrats to us all. > > A big piece of work, nicely done. > > Let's hope that this helps our users make their existing use cases easier and > a

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congrats to us all. A big piece of work, nicely done. Let's hope that this helps our users make their existing use cases easier and also opens up new use cases. On Wed, Feb 12, 2020 at 3:31 PM 张光辉 wrote: > Greet work. > > Congxian Qiu 于2020年2月12日周三 下午10:11写道： > >> Great work. >> Thanks everyo

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Greet work. Congxian Qiu 于2020年2月12日周三 下午10:11写道： > Great work. > Thanks everyone involved. > Thanks Gary and Yu for being the release manager > > > Best, > Congxian > > > Jark Wu 于2020年2月12日周三 下午9:46写道： > >> Congratulations to everyone involved! >> Great thanks to Yu & Gary for being the relea

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great work. Thanks everyone involved. Thanks Gary and Yu for being the release manager Best, Congxian Jark Wu 于2020年2月12日周三 下午9:46写道： > Congratulations to everyone involved! > Great thanks to Yu & Gary for being the release manager! > > Best, > Jark > > On Wed, 12 Feb 2020 at 21:42, Zhu Zhu

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Congratulations to everyone involved! Great thanks to Yu & Gary for being the release manager! Best, Jark On Wed, 12 Feb 2020 at 21:42, Zhu Zhu wrote: > Cheers! > Thanks Gary and Yu for the great job as release managers. > And thanks to everyone whose contribution makes the release possible! >

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Cheers! Thanks Gary and Yu for the great job as release managers. And thanks to everyone whose contribution makes the release possible! Thanks, Zhu Zhu Wyatt Chun 于2020年2月12日周三 下午9:36写道： > Sounds great. Congrats & Thanks! > > On Wed, Feb 12, 2020 at 9:31 PM Yu Li wrote: > >> The Apache Flink c

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great work, thanks Gary and Yu. Yun Tang 于2020年2月12日周三 下午9:38写道： > Great work, thanks Gary & Yu ! > > Best > Yun Tang > -- > *From:* Wyatt Chun > *Sent:* Wednesday, February 12, 2020 21:36 > *To:* Yu Li > *Cc:* user > *Subject:* Re: [ANNOUNCE] Apache Flink 1.10.0 r

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great work, thanks Gary & Yu ! Best Yun Tang From: Wyatt Chun Sent: Wednesday, February 12, 2020 21:36 To: Yu Li Cc: user Subject: Re: [ANNOUNCE] Apache Flink 1.10.0 released Sounds great. Congrats & Thanks! On Wed, Feb 12, 2020 at 9:31 PM Yu Li mailto:car...

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Sounds great. Congrats & Thanks! On Wed, Feb 12, 2020 at 9:31 PM Yu Li wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.10.0, which is the latest major release. > > Apache Flink® is an open-source stream processing framework for > distributed, high-pe

Re: [ANNOUNCE] Apache Flink 1.10.0 released

Great to hear that! Thanks for being the release managers, Gary & Yu. Great work! Thank you~ Xintong Song On Wed, Feb 12, 2020 at 9:31 PM Yu Li wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.10.0, which is the latest major release. > > Apache

[ANNOUNCE] Apache Flink 1.10.0 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.10.0, which is the latest major release. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is avai

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi Salva, Yes, the same applies to the Operator API as the output is not thread-safe and there is no way of "checkpointing" the "in-flight" data without explicit handling. If you want to dig deeper, I would recommend to have a look also at the source code of the AsyncWaitOperator to see how you co

Re: Backfilling From Kafka Topics With Varied Ingestion Rates

You can find more information here: https://cwiki.apache.org/confluence/display/FLINK/FLIP-27%3A+Refactor+Source+Interface On Wed, Feb 12, 2020 at 11:30 AM Akshay Aggarwal < akshay.aggar...@flipkart.com> wrote: > Thanks Aljoscha. Is there a JIRA where this is getting tracked? > > ~Akshay > > On W

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

According to this answer [1] the first exception "mentioning" org/joda/time/format/DateTimeParserBucket should be a different one. Can you go through the logs to make sure it is really a ClassNotFoundException, and not a ExceptionInInitializerError or something else? [1]https://stackoverflow.com/a

Re: Exactly once semantics for hdfs sink

Hi Vishwas, Please let me know if you have any specific questions about the StreamingFile sink. Regards, Roman On Wed, Feb 12, 2020 at 4:45 AM Zhijiang wrote: > Hi Vishwas, > > I guess this link [1] can help understand how it works and how to use in > practice for StreamingFileSink. > > [1] >

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Would your comment still apply if I was using AbstractStreamOperator (passing its output when registering the callbacks) instead of a UDF? Maybe the situation changes if I use the Operator API instead... -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

Hi Arvid, I'm using flink-s3-fs-hadoop-1.9.1.jar in plugins folder. Like I said previously, this works normally until an exception is throw inside the sink. It will try to recover again, but sometimes doesn't recover giving this error. To write to S3 I use *AvroParquetWriter* with the following c

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi Kostas, Thanks for your further comments. I will take a look at the AsyncIO pattern. Regards, Salva -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi Yun, Thanks for your prompt and clear answer! Salva -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Backfilling From Kafka Topics With Varied Ingestion Rates

Thanks Aljoscha. Is there a JIRA where this is getting tracked? ~Akshay On Wed, Feb 12, 2020 at 1:56 PM Aljoscha Krettek wrote: > Hi, > > I'm afraid your analysis is 100% correct. Currently there's no > out-of-box feature for dealing with this but our work on a new source > interface ([1]) will

Re: Dedup all data in stream

Hi Akshay, Could you be more specific on what you are trying to achieve with this scheme? I am asking because if your source is too fast and you want it to slow it down so that it produces data at the same rate as your process function can consume them, then Flink's backpressure will eventually d

Re: NoClassDefFoundError when an exception is throw inside invoke in a Custom Sink

Hi David, upon closer reviewing your stacktrace, it seems like you are trying to access S3 without our S3 plugin. That's in general not recommended at all. Best, Arvid On Tue, Feb 11, 2020 at 11:06 AM Arvid Heise wrote: > Hi David, > > this seems to be a bug in our s3 plugin. The joda depende

CEP with changing threshold

Hello Flink Users, I have a requirement to generate alerts for metrics like for example - if cpu utilization spike i.e *cpu_utilization > threshold* (>90%) n number of time in x minutes then generate alerts. For this I am using the CEP module. However, one of the requirements is for different devi

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi Salva and Yun, Yun is correct on that the collector is not thread-safe so writing should be guarded. In addition, such a pattern that issues a request to a 3rd party multi-threaded library and registers a callback for the future does not play well with checkpointing. In your case, if a failure

Re: rocksdb max open file descriptor issue crashed application

Hi Apoorv, I am not so familiar with the internal of RocksDB and how the number of open files correlates with the number of (keyed) states and the parallelism you have, but as a starting point you can have a look to [1] for recommendations on how to tune RocksDb for large state and I am also cc'in

Re: Aggregation for last n seconds for each event

Hi Oleg, Could you be more specific on what do you mean by "for events of last n seconds(time units in general) for every incoming event."? Do you mean that you have a stream of parallelism 1 and you want for each incoming element to have your function fire with input the event itself and all the

Re: Flink complaining when trying to write to s3 in Parquet format

Hi Fatima, I am not super familiar with Parquet but your issue seems to be related to [1], which seems to be expected behaviour on the Parquet side. The reason for this behaviour seems to be the format of the parquet files which store only the leaf fields but not the structure of the groups, so if

[DISCUSS] What parts of Flink SQL should we improve for usability?

Hi, all We have listened some user's compliant voice about Flink SQL usability from online and offline, especially about using Flink SQL to build ETL job, but it's not detailed. So, I'd like to start a discussion about Flink SQL usability, hope we can have more input from user side which will

Re: Flink job fails with org.apache.kafka.common.KafkaException: Failed to construct kafka consumer

Hi John, As you suggested, I would also lean towards increasing the number of allowed open handles, but for recommendation on best practices, I am cc'ing Piotr who may be more familiar with the Kafka consumer. Cheers, Kostas On Tue, Feb 11, 2020 at 9:43 PM John Smith wrote: > > Just wondering i

Re: Backfilling From Kafka Topics With Varied Ingestion Rates

Hi, I'm afraid your analysis is 100% correct. Currently there's no out-of-box feature for dealing with this but our work on a new source interface ([1]) will enable us to add a feature that we call "event-time alignment" where source readers would slow down reading from certain source partiti

Re: Using multithreaded library within ProcessFunction with callbacks relying on the out parameter

Hi Salva, As far as I know, 1. Out : Collector[T] could not support multi-thread accessing, namely there could be only one thread writing records at one time. If there are multiple threads using `out`, the access should need to be coordinated in some way (for e