Re: Apache Flink - How to destroy global window and release it's resources

2019-04-10 Thread Guowei Ma
Hi, I think you could return a proper TriggerResult, which defines how to deal with the window elements after computing a window in your trigger implementation. You could find the detail information from the doc[1]. 1. https://ci.apache.org/projects/flink/flink-docs-release-1.8/dev/stream/operator

Re: Does Flink apply API for setScale of BigDecimal type

2019-04-10 Thread 徐涛
Hi Victor, Yeah, I think it is a Java question, but I just wonder how could set scale of BigDecimal type in Flink. For example, I use the Types.DECIMAL type in table connector, and set the column name to A, then I divide the column A by 1.1 in SQL. Then it will report exception

Re: Is copying flink-hadoop-compatibility jar to FLINK_HOME/lib the only way to make it work?

2019-04-10 Thread Morven Huang
Hi Fabian, Packaging that dependency into a fat jar doesn't help, here is the pom.xml I use, could you please help to take a look if there're some problems? http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4

Re: Does HadoopOutputFormat create MapReduce job internally?

2019-04-10 Thread Morven Huang
Hi Fabian, Thank you for the clarification. Best, Morven Huang On Wed, Apr 10, 2019 at 9:57 PM Fabian Hueske wrote: > Hi, > > Flink's Hadoop compatibility functions just wrap functions that were > implemented against Hadoop's interfaces in wrapper functions that are > implemented against Flink

Re: Apache Flink - Question about broadcast state pattern usage

2019-04-10 Thread M Singh
Hi Guowei; Thanks for your answer. Do you have any example which illustrates using broadcast is used with multiple descriptors ? Thanks On Sunday, April 7, 2019, 10:10:15 PM EDT, Guowei Ma wrote: Hi1. I think you could use "Using Managed Operator State"[1] (context.getOperatorState

Apache Flink - How to destroy global window and release it's resources

2019-04-10 Thread M Singh
Hi: I have a use case where I need to create a global window where I need to wait for unknown time for certain events for a particular key.  I understand that I can create a global window and use a custom trigger to initiate the function computation.  But I am not sure how to destroy the window

Query on job restoration using relocated savepoint

2019-04-10 Thread Parth Sarathy
Hi All, We are trying to restore a job using relocated savepoint files. As pointed out in the FAQs of savepoint documentation, savepoints have absolute paths recorded in them and hence a simple relocation to restore the job would fail. As directed in the documentation we tried out th

Re: Timestamp Watermark Assigner bpund question

2019-04-10 Thread Vijay Balakrishnan
Hi Guowei, Thx for your reply. I am trying to understand the logic behind the Point 1 i.e current Watermark being currMaxTimestamp minus the bound. So, does this mean the Operator processing a task has a current Event time < current Watermark < currMaxTimestamp ??? Then the Operator progresses to t

Re: FlinkException: The assigned slot was removed

2019-04-10 Thread Konstantin Knauf
Hi Konstantinos, looks like your using Spring to build your Flink job. Do you maybe use Spring's dependency injection mechanism to inject objects into objects, which are serialization and shipped to the taskmanagers? I could imagine this being the problem. In general, when a slot is removed this u

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Rong Rong
Congrats! Thanks Aljoscha for being the release manager and all for making the release possible. -- Rong On Wed, Apr 10, 2019 at 4:23 AM Stefan Richter wrote: > Congrats and thanks to Aljoscha for managing the release! > > Best, > Stefan > > > On 10. Apr 2019, at 13:01, Biao Liu wrote: > > >

Re: Does HadoopOutputFormat create MapReduce job internally?

2019-04-10 Thread Fabian Hueske
Hi, Flink's Hadoop compatibility functions just wrap functions that were implemented against Hadoop's interfaces in wrapper functions that are implemented against Flink's interfaces. There is no Hadoop cluster started or MapReduce job being executed. Job is just a class of the Hadoop API. It does

Re: Is copying flink-hadoop-compatibility jar to FLINK_HOME/lib the only way to make it work?

2019-04-10 Thread Fabian Hueske
Hi, Packaging the flink-hadoop-compatibility dependency with your code into a "fat" job jar should work as well. Best, Fabian Am Mi., 10. Apr. 2019 um 15:08 Uhr schrieb Morven Huang < morven.hu...@gmail.com>: > Hi, > > > > I’m using Flink 1.5.6 and Hadoop 2.7.1. > > > > *My requirement is to re

Re: Flink forward talks

2019-04-10 Thread Konstantin Knauf
Hi Navneeth, they are usually published on https://www.ververica.com/resources, but they are not ready yet. Olivia (cc) should know more. Cheers, Konstantin On Wed, Apr 10, 2019 at 11:19 AM Navneeth Krishnan wrote: > Hi All, > > Where can I get the videos of latest flink forward talks? > > Th

Does HadoopOutputFormat create MapReduce job internally?

2019-04-10 Thread Morven Huang
Hi, I’d like to sink my data into hdfs using SequenceFileAsBinaryOutputFormat with compression, and I find a way from the link https://ci.apache.org/projects/flink/flink-docs-stable/dev/batch/hadoop_compatibility.html, the code works, but I’m curious to know, since it creates a mapreduce Job ins

Is copying flink-hadoop-compatibility jar to FLINK_HOME/lib the only way to make it work?

2019-04-10 Thread Morven Huang
Hi, I’m using Flink 1.5.6 and Hadoop 2.7.1. *My requirement is to read hdfs sequence file (SequenceFileInputFormat), then write it back to hdfs (SequenceFileAsBinaryOutputFormat with compression).* Below code won’t work until I copy the flink-hadoop-compatibility jar to FLINK_HOME/lib. I f

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Stefan Richter
Congrats and thanks to Aljoscha for managing the release! Best, Stefan > On 10. Apr 2019, at 13:01, Biao Liu wrote: > > Great news! Thanks Aljoscha and all the contributors. > > Till Rohrmann mailto:trohrm...@apache.org>> > 于2019年4月10日周三 下午6:11写道: > Thanks a lot to Aljoscha for being our rele

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Biao Liu
Great news! Thanks Aljoscha and all the contributors. Till Rohrmann 于2019年4月10日周三 下午6:11写道: > Thanks a lot to Aljoscha for being our release manager and to the > community making this release possible! > > Cheers, > Till > > On Wed, Apr 10, 2019 at 12:09 PM Hequn Cheng wrote: > >> Thanks a lot

Re: Flink 1.7.2 UI : Jobs removed from Completed Jobs section

2019-04-10 Thread Guowei Ma
I am not very sure about this problem. But you could try to increase jobstore.expiration-time in config. Best, Guowei Jins George 于2019年4月10日周三 下午1:01写道: > Any input on this UI behavior ? > > > > Thanks, > > Jins > > > > *From: *Timothy Victor > *Date: *Monday, April 8, 2019 at 10:47 AM > *To:

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Till Rohrmann
Thanks a lot to Aljoscha for being our release manager and to the community making this release possible! Cheers, Till On Wed, Apr 10, 2019 at 12:09 PM Hequn Cheng wrote: > Thanks a lot for the great release Aljoscha! > Also thanks for the work by the whole community. :-) > > Best, Hequn > > On

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Hequn Cheng
Thanks a lot for the great release Aljoscha! Also thanks for the work by the whole community. :-) Best, Hequn On Wed, Apr 10, 2019 at 6:03 PM Fabian Hueske wrote: > Congrats to everyone! > > Thanks Aljoscha and all contributors. > > Cheers, Fabian > > Am Mi., 10. Apr. 2019 um 11:54 Uhr schrieb

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Fabian Hueske
Congrats to everyone! Thanks Aljoscha and all contributors. Cheers, Fabian Am Mi., 10. Apr. 2019 um 11:54 Uhr schrieb Congxian Qiu < qcx978132...@gmail.com>: > Cool! > > Thanks Aljoscha a lot for being our release manager, and all the others > who make this release possible. > > Best, Congxian

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Congxian Qiu
Cool! Thanks Aljoscha a lot for being our release manager, and all the others who make this release possible. Best, Congxian On Apr 10, 2019, 17:47 +0800, Jark Wu , wrote: > Cheers! > > Thanks Aljoscha and all others who make 1.8.0 possible. > > On Wed, 10 Apr 2019 at 17:33, vino yang wrote: >

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Guowei Ma
Congratulations! Thanks Aljoscha and all contributors! Best, Guowei Jark Wu 于2019年4月10日周三 下午5:47写道: > Cheers! > > Thanks Aljoscha and all others who make 1.8.0 possible. > > On Wed, 10 Apr 2019 at 17:33, vino yang wrote: > > > Great news! > > > > Thanks Aljoscha for being the release manage

What is the best way to handle data skew processing in Data Stream applications?

2019-04-10 Thread Felipe Gutierrez
Hi, I am studying data skew processing in Flink and how I can change the low-level control of physical partition ( https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/#physical-partitioning) in order to have an even processing of tuples. I have created synthetic skewed data

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Jark Wu
Cheers! Thanks Aljoscha and all others who make 1.8.0 possible. On Wed, 10 Apr 2019 at 17:33, vino yang wrote: > Great news! > > Thanks Aljoscha for being the release manager and thanks to all the > contributors! > > Best, > Vino > > Driesprong, Fokko 于2019年4月10日周三 下午4:54写道: > >> Great news! G

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread zhijiang
Cool! Finally see the FLINK 1.8.0 release. Thanks Aljoscha for this excellent work and efforts for other contributors. We would continue working hard for FLINK 1.9.0 Best, Zhijiang -- From:vino yang Send Time:2019年4月10日(星期三) 17:33

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread vino yang
Great news! Thanks Aljoscha for being the release manager and thanks to all the contributors! Best, Vino Driesprong, Fokko 于2019年4月10日周三 下午4:54写道: > Great news! Great effort by the community to make this happen. Thanks all! > > Cheers, Fokko > > Op wo 10 apr. 2019 om 10:50 schreef Shaoxuan Wan

Flink forward talks

2019-04-10 Thread Navneeth Krishnan
Hi All, Where can I get the videos of latest flink forward talks? Thanks, Navneeth

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Shaoxuan Wang
Thanks Aljoscha and all others who made contributions to FLINK 1.8.0. Looking forward to FLINK 1.9.0. Regards, Shaoxuan On Wed, Apr 10, 2019 at 4:31 PM Aljoscha Krettek wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.8.0, which is the next major rel

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Zili Chen
Cheers! Glad to meet the release of 1.8.0. Thank Aljoscha for being the release manager and all contributors who made this release possible. Best, tison. Aljoscha Krettek 于2019年4月10日周三 下午4:31写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.8.0, which

Re: [ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread jincheng sun
Thanks a lot for being our release manager @Aljoscha Krettek Great job! And also a big thanks to the community for making this release possible. Cheers, Jincheng Aljoscha Krettek 于2019年4月10日周三 下午4:31写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.8.0,

[ANNOUNCE] Apache Flink 1.8.0 released

2019-04-10 Thread Aljoscha Krettek
The Apache Flink community is very happy to announce the release of Apache Flink 1.8.0, which is the next major release. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is ava