Re: [DISCUSS] Exact feature freeze date
+1 for extending the feature freeze until May 15th. Best, Zhijiang -- From:Danny Chan Send Time:2020 Apr. 24 (Fri.) 10:51 To:dev Subject:Re: [DISCUSS] Exact feature freeze date +1 for extending the feature freeze until May 15th. Best, Danny Chan 在 2020年4月24日 +0800 AM9:51，Yangze Guo ，写道： > +1 > > Best, > Yangze Guo > > On Fri, Apr 24, 2020 at 9:49 AM Dian Fu wrote: > > > > +1 > > > > Regards, > > Dian > > > > > 在 2020年4月24日，上午9:47，Leonard Xu 写道： > > > > > > + 1 for the feature freeze date > > > > > > Best, > > > Leonard Xu > > > > > > > > > > 在 2020年4月24日，09:32，Jingsong Li 写道： > > > > > > > > +1 > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > > > > > > > > > +1 for extending the code freeze date. > > > > > FLIP-119 could benefit from it. > > > > > May 15th sounds reasonable. > > > > > > > > > > Thanks, > > > > > Zhu Zhu > > > > > > > > > > Jark Wu 于2020年4月24日周五 上午12:01写道： > > > > > > > > > > > +1 > > > > > > > > > > > > Thanks, > > > > > > Jark > > > > > > > > > > > > On Thu, 23 Apr 2020 at 22:36, Xintong Song > > > > > wrote: > > > > > > > > > > > > > +1 > > > > > > > From our side we can also benefit from the extending of feature > > > > > > > freeze, > > > > > > for > > > > > > > pluggable slot allocation, GPU support and perjob mode on > > > > > > > Kubernetes > > > > > > > deployment. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther > > > > > > wrote: > > > > > > > > > > > > > > > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could > > > > > > > > benefit > > > > > > > > from extending the feature freeze. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Timo > > > > > > > > > > > > > > > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > On 23.04.20 15:23, Till Rohrmann wrote: > > > > > > > > > > +1 for extending the feature freeze until May 15th. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < > > > > > pi...@ververica.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > As release manager I’ve seen that quite a bit of features > > > > > > > > > > > could > > > > > use > > > > > > > > > > > of the > > > > > > > > > > > extra couple of weeks. This also includes some features > > > > > > > > > > > that I’m > > > > > > > > > > > involved > > > > > > > > > > > with, like FLIP-76, or limiting the in-flight buffers. > > > > > > > > > > > > > > > > > > > > > > +1 From my side for extending the feature freeze until > > > > > > > > > > > May 15th. > > > > > > > > > > > > > > > > > > > > > > Piotrek > > > > > > > > > > > > > > > > > > > > > > > On 23 Apr 2020, at 10:10, Stephan Ewen > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > I want to bring up a discussion about when we want to > > > > > > > > > > > > do the > > > > > > feature > > > > > > > > > > > freeze > > > > > > > > > > > > for 1.11. > > > > > > > > > > > > > > > > > > > > > > > > When kicking off the release cycle, we tentatively set > > > > > > > > > > > > the date > > > > > to > > > > > > > > > > > > end of > > > > > > > > > > > > April, which would be in one week. > > > > > > > > > > > > > > > > > > > > > > > > I can say from the features I am involved with (FLIP-27, > > > > > FLIP-115, > > > > > > > > > > > > reviewing some state backend improvements, etc.) that > > > > > > > > > > > > it would > > > > > be > > > > > > > > > > > > helpful > > > > > > > > > > > > to have two additional weeks. > > > > > > > > > > > > > > > > > > > > > > > > When looking at various other feature threads, my > > > > > > > > > > > > feeling is > > > > > that > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > more contributors and committers that could use a few > > > > > > > > > > > > more days. > > > > > > > > > > > > The last two months were quite exceptional in and we > > > > > > > > > > > > did lose a > > > > > > bit > > > > > > > of > > > > > > > > > > > > development speed here and there. > > > > > > > > > > > > > > > > > > > > > > > > How do you think about making *May 15th* the feature > > > > > > > > > > > > freeze? > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best, Jingsong Lee > > > > >
[jira] [Created] (FLINK-17364) Support StreamingFileSink in PrestoS3FileSystem
Lu Niu created FLINK-17364: -- Summary: Support StreamingFileSink in PrestoS3FileSystem Key: FLINK-17364 URL: https://issues.apache.org/jira/browse/FLINK-17364 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Reporter: Lu Niu For S3, currently the StreamingFileSink supports only the Hadoop-based FileSystem implementation, not the implementation based on Presto At the same time, presto is the recommended file system for checkpointing. implementing StreamingFileSink in PrestoS3FileSystem helps filling the gap, enables user to use PrestoS3FileSystem in all access to S3. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17363) Asynchronous association dimension table timeout
robert created FLINK-17363: -- Summary: Asynchronous association dimension table timeout Key: FLINK-17363 URL: https://issues.apache.org/jira/browse/FLINK-17363 Project: Flink Issue Type: Bug Components: API / DataStream Affects Versions: 1.9.0 Reporter: robert java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator. at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:137) at org.apache.flink.streaming.api.operators.async.Emitter.run(Emitter.java:85) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.flink.streaming.api.operators.async.queue.StreamRecordQueueEntry.get(StreamRecordQueueEntry.java:68) at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:129) ... 2 more Caused by: java.util.concurrent.TimeoutException: Async function call has timed out. at org.apache.flink.streaming.api.functions.async.AsyncFunction.timeout(AsyncFunction.java:97) at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$1.onProcessingTime(AsyncWaitOperator.java:215) at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:281) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 2020-04-24 11:30:22,933 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator - Discarding checkpoint 492 of job 075a0b260dd779f28bc87aaa59d7222a. java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator. at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:137) at org.apache.flink.streaming.api.operators.async.Emitter.run(Emitter.java:85) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at org.apache.flink.streaming.api.operators.async.queue.StreamRecordQueueEntry.get(StreamRecordQueueEntry.java:68) at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:129) ... 2 more Caused by: java.util.concurrent.TimeoutException: Async function call has timed out. at org.apache.flink.streaming.api.functions.async.AsyncFunction.timeout(AsyncFunction.java:97) at org.apache.flink.streaming.api.operators.async.AsyncWaitOperator$1.onProcessingTime(AsyncWaitOperator.java:215) at org.apache.flink.streaming.runtime.tasks.SystemProcessingTimeService$TriggerTask.run(SystemProcessingTimeService.java:281) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ... 1 more 2020-04-24 11:30:22,933 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job sql_details (075a0b260dd779f28bc87aaa59d7222a) switched from state RUNNING to FAILING. java.lang.Exception: An async function call terminated with an exception. Failing the AsyncWaitOperator. at org.apache.flink.streaming.api.operators.async.Emitter.output(Emitter.java:137) at org.apache.flink.streaming.api.operators.async.Emitter.run(Emitter.java:85) at java.lang.Thread.run(Thread.java:748) Caused by: java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Async function call has timed out. at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357) at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at
Re: [DISCUSS] Exact feature freeze date
+1 for extending the feature freeze until May 15th. Best, Danny Chan 在 2020年4月24日 +0800 AM9:51，Yangze Guo ，写道： > +1 > > Best, > Yangze Guo > > On Fri, Apr 24, 2020 at 9:49 AM Dian Fu wrote: > > > > +1 > > > > Regards, > > Dian > > > > > 在 2020年4月24日，上午9:47，Leonard Xu 写道： > > > > > > + 1 for the feature freeze date > > > > > > Best, > > > Leonard Xu > > > > > > > > > > 在 2020年4月24日，09:32，Jingsong Li 写道： > > > > > > > > +1 > > > > > > > > Best, > > > > Jingsong Lee > > > > > > > > On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > > > > > > > > > +1 for extending the code freeze date. > > > > > FLIP-119 could benefit from it. > > > > > May 15th sounds reasonable. > > > > > > > > > > Thanks, > > > > > Zhu Zhu > > > > > > > > > > Jark Wu 于2020年4月24日周五 上午12:01写道： > > > > > > > > > > > +1 > > > > > > > > > > > > Thanks, > > > > > > Jark > > > > > > > > > > > > On Thu, 23 Apr 2020 at 22:36, Xintong Song > > > > > wrote: > > > > > > > > > > > > > +1 > > > > > > > From our side we can also benefit from the extending of feature > > > > > > > freeze, > > > > > > for > > > > > > > pluggable slot allocation, GPU support and perjob mode on > > > > > > > Kubernetes > > > > > > > deployment. > > > > > > > > > > > > > > Thank you~ > > > > > > > > > > > > > > Xintong Song > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther > > > > > > wrote: > > > > > > > > > > > > > > > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could > > > > > > > > benefit > > > > > > > > from extending the feature freeze. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Timo > > > > > > > > > > > > > > > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > Aljoscha > > > > > > > > > > > > > > > > > > On 23.04.20 15:23, Till Rohrmann wrote: > > > > > > > > > > +1 for extending the feature freeze until May 15th. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Till > > > > > > > > > > > > > > > > > > > > On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < > > > > > pi...@ververica.com > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Stephan, > > > > > > > > > > > > > > > > > > > > > > As release manager I’ve seen that quite a bit of features > > > > > > > > > > > could > > > > > use > > > > > > > > > > > of the > > > > > > > > > > > extra couple of weeks. This also includes some features > > > > > > > > > > > that I’m > > > > > > > > > > > involved > > > > > > > > > > > with, like FLIP-76, or limiting the in-flight buffers. > > > > > > > > > > > > > > > > > > > > > > +1 From my side for extending the feature freeze until > > > > > > > > > > > May 15th. > > > > > > > > > > > > > > > > > > > > > > Piotrek > > > > > > > > > > > > > > > > > > > > > > > On 23 Apr 2020, at 10:10, Stephan Ewen > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > Hi all! > > > > > > > > > > > > > > > > > > > > > > > > I want to bring up a discussion about when we want to > > > > > > > > > > > > do the > > > > > > feature > > > > > > > > > > > freeze > > > > > > > > > > > > for 1.11. > > > > > > > > > > > > > > > > > > > > > > > > When kicking off the release cycle, we tentatively set > > > > > > > > > > > > the date > > > > > to > > > > > > > > > > > > end of > > > > > > > > > > > > April, which would be in one week. > > > > > > > > > > > > > > > > > > > > > > > > I can say from the features I am involved with (FLIP-27, > > > > > FLIP-115, > > > > > > > > > > > > reviewing some state backend improvements, etc.) that > > > > > > > > > > > > it would > > > > > be > > > > > > > > > > > > helpful > > > > > > > > > > > > to have two additional weeks. > > > > > > > > > > > > > > > > > > > > > > > > When looking at various other feature threads, my > > > > > > > > > > > > feeling is > > > > > that > > > > > > > > there > > > > > > > > > > > are > > > > > > > > > > > > more contributors and committers that could use a few > > > > > > > > > > > > more days. > > > > > > > > > > > > The last two months were quite exceptional in and we > > > > > > > > > > > > did lose a > > > > > > bit > > > > > > > of > > > > > > > > > > > > development speed here and there. > > > > > > > > > > > > > > > > > > > > > > > > How do you think about making *May 15th* the feature > > > > > > > > > > > > freeze? > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > Stephan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best, Jingsong Lee > > > > >
[RESULT][VOTE] Release 1.9.3, release candidate #1
Hi everyone, I'm happy to announce that we have unanimously approved this release. There are 9 approving votes, 4 of which are binding: * Robert (binding) * Ufuk (binding) * Dawid (binding) * Jincheng (binding) * Yu * ZhuZhu * Fabian * Leonard Xu * Dian There are no disapproving votes. Thanks everyone!
Re: [VOTE] Release 1.9.3, release candidate #1
Thanks Jincheng for the help. :) Thanks, Dian > 在 2020年4月24日，上午9:49，jincheng sun 写道： > > +1（binding） > > with the check of built form the source and run some test case. > > BTW： I added you(dianfu) as a PyPI maintainer of Apache Flink. > > Best, > Jincheng > > > Dawid Wysakowicz 于2020年4月23日周四 下午7:43写道： > >> +1 (binding) >> >> - verified checksums and signatures >> >> - verified no binary artifacts in source release >> >> - reviewed the website PR >> >> - built from source >> >> - checked diff of source release between 1.9.2 for changes in licenses >> and dependencies >> >> - started standalone cluster and run StreamSQLExample >> >> Best, >> >> Dawid >> >> On 23/04/2020 12:09, Ufuk Celebi wrote: >>> +1 (binding) >>> >>> - checked release notes ✅ >>> - verified sums and hashes ✅ >>> - verified no binary artifacts in source release ✅ >>> - reviewed website PR ✅ >>> - built from source ✅ >>> - built an internal Flink distribution based on the 1.9.3-rc1 commit ✅ >>> - built internal jobs against the staging repo ✅ >>> - deployed those jobs to a 1.9.3 job cluster on Kubernetes and tested >>> checkpointing, SSL ✅ >>> >>> >>> On Thu, Apr 23, 2020 at 10:58 AM Robert Metzger >> wrote: >>> +1 (binding) - for license documentation, I checked the diff between 1.9.2 and 1.9.3: >> https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc >> was updated in PR https://github.com/apache/flink/pull/11617/files - one test dependency change that is not relevant - checked the maven staging repo files - 1.9.3 set correctly, java quickstart looks fine - checked license files of some shaded jars On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: > +1 (non-binding) > > - checked/verified signatures and hashes > - checked the release note > - checked that there are no missing artifacts in staging area > - built from source sing scala 2.11 and using Scala 2.12 succeeded > - ran a couple of end-to-end tests locally and succeeded > - went through all commits checked in between 1.9.3 and 1.9.2, make >> sure > all issues set the "fixVersion" property > - started a cluster, WebUI was accessible, submitted a wordcount job >> and > ran succeeded, no suspicious log output > - the web PR looks good > > Best, > Leonard Xu > >> 在 2020年4月22日，17:58，Dian Fu 写道： >> >> +1 (non-binding) >> >> - verified the checksum and signature >> - checked the release note >> - checked that there are no new dependencies introduced since 1.9.2 which >> may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 >> which >> doesn't affect the license) >> - checked that there are no missing artifacts in staging area >> - checked that the pyflink package could be pip installed >> >> Regards, >> Dian >> >> On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < > fabianp...@data-artisans.com> >> wrote: >> >>> +1 (non-binding) >>> >>> - Verified signature >>> - Built from source (Java8) >>> - Run custom jobs on Kubernetes >>> >>> Regards, >>> Fabian >>> On 18. Apr 2020, at 04:37, Dian Fu wrote: Hi everyone, Please review and vote on the release candidate #1 for the version > 1.9.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific >> comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], * the official Apache source release and binary convenience releases to >>> be deployed to dist.apache.org [2], which are signed with the key with fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], * all artifacts to be deployed to the Maven Central Repository [4], * source code tag "release-1.9.3-rc1" [5], * website pull request listing the new release and adding announcement >>> blog post [6]. The vote will be open for at least 72 hours. It is adopted by majority >>> approval, with at least 3 PMC affirmative votes. Thanks, Dian [1] >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ [3] https://dist.apache.org/repos/dist/release/flink/KEYS [4] https://repository.apache.org/content/repositories/orgapacheflink-1353/ [5] >> https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 [6] https://github.com/apache/flink-web/pull/329 >>> > >> >>
Re: [DISCUSS] Exact feature freeze date
+1 Best, Yangze Guo On Fri, Apr 24, 2020 at 9:49 AM Dian Fu wrote: > > +1 > > Regards, > Dian > > > 在 2020年4月24日，上午9:47，Leonard Xu 写道： > > > > + 1 for the feature freeze date > > > > Best, > > Leonard Xu > > > > > >> 在 2020年4月24日，09:32，Jingsong Li 写道： > >> > >> +1 > >> > >> Best, > >> Jingsong Lee > >> > >> On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > >> > >>> +1 for extending the code freeze date. > >>> FLIP-119 could benefit from it. > >>> May 15th sounds reasonable. > >>> > >>> Thanks, > >>> Zhu Zhu > >>> > >>> Jark Wu 于2020年4月24日周五 上午12:01写道： > >>> > +1 > > Thanks, > Jark > > On Thu, 23 Apr 2020 at 22:36, Xintong Song > >>> wrote: > > > +1 > > From our side we can also benefit from the extending of feature freeze, > for > > pluggable slot allocation, GPU support and perjob mode on Kubernetes > > deployment. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther > wrote: > > > >> From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > >> from extending the feature freeze. > >> > >> Thanks, > >> Timo > >> > >> On 23.04.20 16:11, Aljoscha Krettek wrote: > >>> +1 > >>> > >>> Aljoscha > >>> > >>> On 23.04.20 15:23, Till Rohrmann wrote: > +1 for extending the feature freeze until May 15th. > > Cheers, > Till > > On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < > >>> pi...@ververica.com > > > wrote: > > > Hi Stephan, > > > > As release manager I’ve seen that quite a bit of features could > >>> use > > of the > > extra couple of weeks. This also includes some features that I’m > > involved > > with, like FLIP-76, or limiting the in-flight buffers. > > > > +1 From my side for extending the feature freeze until May 15th. > > > > Piotrek > > > >> On 23 Apr 2020, at 10:10, Stephan Ewen > >>> wrote: > >> > >> Hi all! > >> > >> I want to bring up a discussion about when we want to do the > feature > > freeze > >> for 1.11. > >> > >> When kicking off the release cycle, we tentatively set the date > >>> to > >> end of > >> April, which would be in one week. > >> > >> I can say from the features I am involved with (FLIP-27, > >>> FLIP-115, > >> reviewing some state backend improvements, etc.) that it would > >>> be > >> helpful > >> to have two additional weeks. > >> > >> When looking at various other feature threads, my feeling is > >>> that > >> there > > are > >> more contributors and committers that could use a few more days. > >> The last two months were quite exceptional in and we did lose a > bit > > of > >> development speed here and there. > >> > >> How do you think about making *May 15th* the feature freeze? > >> > >> Best, > >> Stephan > > > > > > >> > >> > > > > >>> > >> > >> > >> -- > >> Best, Jingsong Lee > > >
Re: [VOTE] Release 1.9.3, release candidate #1
Hi all, Thanks a lot for checking and voting for this release! I’ll summarize the result in another email. Thanks, Dian > 在 2020年4月23日，下午7:43，Dawid Wysakowicz 写道： > > +1 (binding) > > - verified checksums and signatures > > - verified no binary artifacts in source release > > - reviewed the website PR > > - built from source > > - checked diff of source release between 1.9.2 for changes in licenses > and dependencies > > - started standalone cluster and run StreamSQLExample > > Best, > > Dawid > > On 23/04/2020 12:09, Ufuk Celebi wrote: >> +1 (binding) >> >> - checked release notes ✅ >> - verified sums and hashes ✅ >> - verified no binary artifacts in source release ✅ >> - reviewed website PR ✅ >> - built from source ✅ >> - built an internal Flink distribution based on the 1.9.3-rc1 commit ✅ >> - built internal jobs against the staging repo ✅ >> - deployed those jobs to a 1.9.3 job cluster on Kubernetes and tested >> checkpointing, SSL ✅ >> >> >> On Thu, Apr 23, 2020 at 10:58 AM Robert Metzger wrote: >> >>> +1 (binding) >>> >>> - for license documentation, I checked the diff between 1.9.2 and 1.9.3: >>> https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 >>> - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc was >>> updated in PR https://github.com/apache/flink/pull/11617/files >>> - one test dependency change that is not relevant >>> - checked the maven staging repo files >>> - 1.9.3 set correctly, java quickstart looks fine >>> - checked license files of some shaded jars >>> >>> >>> >>> On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: >>> +1 (non-binding) - checked/verified signatures and hashes - checked the release note - checked that there are no missing artifacts in staging area - built from source sing scala 2.11 and using Scala 2.12 succeeded - ran a couple of end-to-end tests locally and succeeded - went through all commits checked in between 1.9.3 and 1.9.2, make sure all issues set the "fixVersion" property - started a cluster, WebUI was accessible, submitted a wordcount job and ran succeeded, no suspicious log output - the web PR looks good Best, Leonard Xu > 在 2020年4月22日，17:58，Dian Fu 写道： > > +1 (non-binding) > > - verified the checksum and signature > - checked the release note > - checked that there are no new dependencies introduced since 1.9.2 >>> which > may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 which > doesn't affect the license) > - checked that there are no missing artifacts in staging area > - checked that the pyflink package could be pip installed > > Regards, > Dian > > On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < fabianp...@data-artisans.com> > wrote: > >> +1 (non-binding) >> >> - Verified signature >> - Built from source (Java8) >> - Run custom jobs on Kubernetes >> >> Regards, >> Fabian >> >>> On 18. Apr 2020, at 04:37, Dian Fu wrote: >>> >>> Hi everyone, >>> >>> Please review and vote on the release candidate #1 for the version 1.9.3, >>> as follows: >>> [ ] +1, Approve the release >>> [ ] -1, Do not approve the release (please provide specific comments) >>> >>> >>> The complete staging area is available for your review, which >>> includes: >>> * JIRA release notes [1], >>> * the official Apache source release and binary convenience releases >>> to >> be >>> deployed to dist.apache.org [2], which are signed with the key with >>> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], >>> * all artifacts to be deployed to the Maven Central Repository [4], >>> * source code tag "release-1.9.3-rc1" [5], >>> * website pull request listing the new release and adding >>> announcement >> blog >>> post [6]. >>> >>> The vote will be open for at least 72 hours. It is adopted by >>> majority >> approval, with at least 3 PMC affirmative votes. >>> Thanks, >>> Dian >>> >>> [1] >>> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >>> [4] >>> https://repository.apache.org/content/repositories/orgapacheflink-1353/ >>> [5] >>> https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 >>> [6] https://github.com/apache/flink-web/pull/329 >> >
[jira] [Created] (FLINK-17362) Improve table examples to reflect latest status
Kurt Young created FLINK-17362: -- Summary: Improve table examples to reflect latest status Key: FLINK-17362 URL: https://issues.apache.org/jira/browse/FLINK-17362 Project: Flink Issue Type: Improvement Components: Examples Reporter: Kurt Young Fix For: 1.11.0 Currently the table examples seems outdated, especially after blink planner becomes the default choice. We might need to refactor the structure of all examples, and cover the following items: # streaming sql & table api examples # batch sql & table api examples # table/sql & datastream interoperation # table/sql & dataset interoperation # DDL & DML examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release 1.9.3, release candidate #1
+1（binding） with the check of built form the source and run some test case. BTW： I added you(dianfu) as a PyPI maintainer of Apache Flink. Best, Jincheng Dawid Wysakowicz 于2020年4月23日周四 下午7:43写道： > +1 (binding) > > - verified checksums and signatures > > - verified no binary artifacts in source release > > - reviewed the website PR > > - built from source > > - checked diff of source release between 1.9.2 for changes in licenses > and dependencies > > - started standalone cluster and run StreamSQLExample > > Best, > > Dawid > > On 23/04/2020 12:09, Ufuk Celebi wrote: > > +1 (binding) > > > > - checked release notes ✅ > > - verified sums and hashes ✅ > > - verified no binary artifacts in source release ✅ > > - reviewed website PR ✅ > > - built from source ✅ > > - built an internal Flink distribution based on the 1.9.3-rc1 commit ✅ > > - built internal jobs against the staging repo ✅ > > - deployed those jobs to a 1.9.3 job cluster on Kubernetes and tested > > checkpointing, SSL ✅ > > > > > > On Thu, Apr 23, 2020 at 10:58 AM Robert Metzger > wrote: > > > >> +1 (binding) > >> > >> - for license documentation, I checked the diff between 1.9.2 and 1.9.3: > >> > https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 > >> - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc > was > >> updated in PR https://github.com/apache/flink/pull/11617/files > >> - one test dependency change that is not relevant > >> - checked the maven staging repo files > >> - 1.9.3 set correctly, java quickstart looks fine > >> - checked license files of some shaded jars > >> > >> > >> > >> On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: > >> > >>> +1 (non-binding) > >>> > >>> - checked/verified signatures and hashes > >>> - checked the release note > >>> - checked that there are no missing artifacts in staging area > >>> - built from source sing scala 2.11 and using Scala 2.12 succeeded > >>> - ran a couple of end-to-end tests locally and succeeded > >>> - went through all commits checked in between 1.9.3 and 1.9.2, make > sure > >>> all issues set the "fixVersion" property > >>> - started a cluster, WebUI was accessible, submitted a wordcount job > and > >>> ran succeeded, no suspicious log output > >>> - the web PR looks good > >>> > >>> Best, > >>> Leonard Xu > >>> > 在 2020年4月22日，17:58，Dian Fu 写道： > > +1 (non-binding) > > - verified the checksum and signature > - checked the release note > - checked that there are no new dependencies introduced since 1.9.2 > >> which > may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 > which > doesn't affect the license) > - checked that there are no missing artifacts in staging area > - checked that the pyflink package could be pip installed > > Regards, > Dian > > On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < > >>> fabianp...@data-artisans.com> > wrote: > > > +1 (non-binding) > > > > - Verified signature > > - Built from source (Java8) > > - Run custom jobs on Kubernetes > > > > Regards, > > Fabian > > > >> On 18. Apr 2020, at 04:37, Dian Fu wrote: > >> > >> Hi everyone, > >> > >> Please review and vote on the release candidate #1 for the version > >>> 1.9.3, > >> as follows: > >> [ ] +1, Approve the release > >> [ ] -1, Do not approve the release (please provide specific > comments) > >> > >> > >> The complete staging area is available for your review, which > >> includes: > >> * JIRA release notes [1], > >> * the official Apache source release and binary convenience releases > >> to > > be > >> deployed to dist.apache.org [2], which are signed with the key with > >> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], > >> * all artifacts to be deployed to the Maven Central Repository [4], > >> * source code tag "release-1.9.3-rc1" [5], > >> * website pull request listing the new release and adding > >> announcement > > blog > >> post [6]. > >> > >> The vote will be open for at least 72 hours. It is adopted by > >> majority > > approval, with at least 3 PMC affirmative votes. > >> Thanks, > >> Dian > >> > >> [1] > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 > >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ > >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1353/ > >> [5] > >> > https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 > >> [6] https://github.com/apache/flink-web/pull/329 > > > >>> > >
Re: [DISCUSS] Exact feature freeze date
+1 Regards, Dian > 在 2020年4月24日，上午9:47，Leonard Xu 写道： > > + 1 for the feature freeze date > > Best, > Leonard Xu > > >> 在 2020年4月24日，09:32，Jingsong Li 写道： >> >> +1 >> >> Best, >> Jingsong Lee >> >> On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: >> >>> +1 for extending the code freeze date. >>> FLIP-119 could benefit from it. >>> May 15th sounds reasonable. >>> >>> Thanks, >>> Zhu Zhu >>> >>> Jark Wu 于2020年4月24日周五 上午12:01写道： >>> +1 Thanks, Jark On Thu, 23 Apr 2020 at 22:36, Xintong Song >>> wrote: > +1 > From our side we can also benefit from the extending of feature freeze, for > pluggable slot allocation, GPU support and perjob mode on Kubernetes > deployment. > > Thank you~ > > Xintong Song > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther wrote: > >> From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit >> from extending the feature freeze. >> >> Thanks, >> Timo >> >> On 23.04.20 16:11, Aljoscha Krettek wrote: >>> +1 >>> >>> Aljoscha >>> >>> On 23.04.20 15:23, Till Rohrmann wrote: +1 for extending the feature freeze until May 15th. Cheers, Till On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < >>> pi...@ververica.com > wrote: > Hi Stephan, > > As release manager I’ve seen that quite a bit of features could >>> use > of the > extra couple of weeks. This also includes some features that I’m > involved > with, like FLIP-76, or limiting the in-flight buffers. > > +1 From my side for extending the feature freeze until May 15th. > > Piotrek > >> On 23 Apr 2020, at 10:10, Stephan Ewen >>> wrote: >> >> Hi all! >> >> I want to bring up a discussion about when we want to do the feature > freeze >> for 1.11. >> >> When kicking off the release cycle, we tentatively set the date >>> to >> end of >> April, which would be in one week. >> >> I can say from the features I am involved with (FLIP-27, >>> FLIP-115, >> reviewing some state backend improvements, etc.) that it would >>> be >> helpful >> to have two additional weeks. >> >> When looking at various other feature threads, my feeling is >>> that >> there > are >> more contributors and committers that could use a few more days. >> The last two months were quite exceptional in and we did lose a bit > of >> development speed here and there. >> >> How do you think about making *May 15th* the feature freeze? >> >> Best, >> Stephan > > >> >> > >>> >> >> >> -- >> Best, Jingsong Lee >
Re: [DISCUSS] Exact feature freeze date
+ 1 for the feature freeze date Best, Leonard Xu > 在 2020年4月24日，09:32，Jingsong Li 写道： > > +1 > > Best, > Jingsong Lee > > On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > >> +1 for extending the code freeze date. >> FLIP-119 could benefit from it. >> May 15th sounds reasonable. >> >> Thanks, >> Zhu Zhu >> >> Jark Wu 于2020年4月24日周五 上午12:01写道： >> >>> +1 >>> >>> Thanks, >>> Jark >>> >>> On Thu, 23 Apr 2020 at 22:36, Xintong Song >> wrote: >>> +1 From our side we can also benefit from the extending of feature freeze, >>> for pluggable slot allocation, GPU support and perjob mode on Kubernetes deployment. Thank you~ Xintong Song On Thu, Apr 23, 2020 at 10:31 PM Timo Walther >>> wrote: > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > from extending the feature freeze. > > Thanks, > Timo > > On 23.04.20 16:11, Aljoscha Krettek wrote: >> +1 >> >> Aljoscha >> >> On 23.04.20 15:23, Till Rohrmann wrote: >>> +1 for extending the feature freeze until May 15th. >>> >>> Cheers, >>> Till >>> >>> On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < >> pi...@ververica.com >>> wrote: >>> Hi Stephan, As release manager I’ve seen that quite a bit of features could >> use of the extra couple of weeks. This also includes some features that I’m involved with, like FLIP-76, or limiting the in-flight buffers. +1 From my side for extending the feature freeze until May 15th. Piotrek > On 23 Apr 2020, at 10:10, Stephan Ewen >> wrote: > > Hi all! > > I want to bring up a discussion about when we want to do the >>> feature freeze > for 1.11. > > When kicking off the release cycle, we tentatively set the date >> to > end of > April, which would be in one week. > > I can say from the features I am involved with (FLIP-27, >> FLIP-115, > reviewing some state backend improvements, etc.) that it would >> be > helpful > to have two additional weeks. > > When looking at various other feature threads, my feeling is >> that > there are > more contributors and committers that could use a few more days. > The last two months were quite exceptional in and we did lose a >>> bit of > development speed here and there. > > How do you think about making *May 15th* the feature freeze? > > Best, > Stephan >>> > > >>> >> > > > -- > Best, Jingsong Lee
Re: [DISCUSS] Exact feature freeze date
+1 Best, Jingsong Lee On Fri, Apr 24, 2020 at 2:27 AM Zhu Zhu wrote: > +1 for extending the code freeze date. > FLIP-119 could benefit from it. > May 15th sounds reasonable. > > Thanks, > Zhu Zhu > > Jark Wu 于2020年4月24日周五 上午12:01写道： > > > +1 > > > > Thanks, > > Jark > > > > On Thu, 23 Apr 2020 at 22:36, Xintong Song > wrote: > > > > > +1 > > > From our side we can also benefit from the extending of feature freeze, > > for > > > pluggable slot allocation, GPU support and perjob mode on Kubernetes > > > deployment. > > > > > > Thank you~ > > > > > > Xintong Song > > > > > > > > > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther > > wrote: > > > > > > > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > > > > from extending the feature freeze. > > > > > > > > Thanks, > > > > Timo > > > > > > > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > > > > +1 > > > > > > > > > > Aljoscha > > > > > > > > > > On 23.04.20 15:23, Till Rohrmann wrote: > > > > >> +1 for extending the feature freeze until May 15th. > > > > >> > > > > >> Cheers, > > > > >> Till > > > > >> > > > > >> On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski < > pi...@ververica.com > > > > > > > >> wrote: > > > > >> > > > > >>> Hi Stephan, > > > > >>> > > > > >>> As release manager I’ve seen that quite a bit of features could > use > > > > >>> of the > > > > >>> extra couple of weeks. This also includes some features that I’m > > > > >>> involved > > > > >>> with, like FLIP-76, or limiting the in-flight buffers. > > > > >>> > > > > >>> +1 From my side for extending the feature freeze until May 15th. > > > > >>> > > > > >>> Piotrek > > > > >>> > > > > On 23 Apr 2020, at 10:10, Stephan Ewen > wrote: > > > > > > > > Hi all! > > > > > > > > I want to bring up a discussion about when we want to do the > > feature > > > > >>> freeze > > > > for 1.11. > > > > > > > > When kicking off the release cycle, we tentatively set the date > to > > > > end of > > > > April, which would be in one week. > > > > > > > > I can say from the features I am involved with (FLIP-27, > FLIP-115, > > > > reviewing some state backend improvements, etc.) that it would > be > > > > helpful > > > > to have two additional weeks. > > > > > > > > When looking at various other feature threads, my feeling is > that > > > > there > > > > >>> are > > > > more contributors and committers that could use a few more days. > > > > The last two months were quite exceptional in and we did lose a > > bit > > > of > > > > development speed here and there. > > > > > > > > How do you think about making *May 15th* the feature freeze? > > > > > > > > Best, > > > > Stephan > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > -- Best, Jingsong Lee
Re: [DISCUSS] FLIP-36 - Support Interactive Programming in Flink Table API
Hi Xuannan, Thanks for picking up the FLIP. It looks good to me overall. Some quick comments / questions below: 1. Do we also need changes in the Java API? 2. What are the cases that users may want to retry reading the intermediate result? It seems that once the intermediate result has gone, it will not be available later without being generated again, right? 3. In the "semantic of cache() method" section, the description "The semantic of the *cache() *method is a little different depending on whether auto caching is enabled or not." seems not explained. Thanks, Jiangjie (Becket) Qin On Wed, Apr 22, 2020 at 4:00 PM Xuannan Su wrote: > Hi folks, > > I'd like to start the discussion about FLIP-36 Support Interactive > Programming in Flink Table API > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-36%3A+Support+Interactive+Programming+in+Flink > > The FLIP proposes to add support for interactive programming in Flink Table > API. Specifically, it let users cache the intermediate results(tables) and > use them in the later jobs. > > Even though the FLIP has been discussed in the past[1], the FLIP hasn't > formally passed the vote yet. And some of the design and implementation > detail have to change to incorporates the cluster partition proposed in > FLIP-67[2]. > > Looking forward to your feedback. > > Thanks, > Xuannan > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-67%3A+Cluster+partitions+lifecycle > [2] > > https://lists.apache.org/thread.html/b372fd7b962b9f37e4dace3bc8828f6e2a2b855e56984e58bc4a413f@%3Cdev.flink.apache.org%3E >
[jira] [Created] (FLINK-17361) Support creating of a JDBC table using a custom query
Flavio Pompermaier created FLINK-17361: -- Summary: Support creating of a JDBC table using a custom query Key: FLINK-17361 URL: https://issues.apache.org/jira/browse/FLINK-17361 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Flavio Pompermaier In a long discussion on the mailing list it has emerged how it is not possible to create a JDBC table that extract data using a custom query. A temporary workaround could be to assign as 'connector.table' the target query. However this is undesirable. Moreover, in relation to https://issues.apache.org/jira/browse/FLINK-17360, a query could be actually a statement that requires parameters to be filled by the custom parameter values provider -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17360) Support custom partitioners in JDBCReadOptions
Flavio Pompermaier created FLINK-17360: -- Summary: Support custom partitioners in JDBCReadOptions Key: FLINK-17360 URL: https://issues.apache.org/jira/browse/FLINK-17360 Project: Flink Issue Type: Improvement Components: Table SQL / API Reporter: Flavio Pompermaier Suport custom ParameterValuesProvider. At the moment only NumericBetweenParametersProvider is handled if a partition column and min/max values are specified. In a discussion in the mailing list some discussion about this was made. Me ([~f.pompermaier]): Then we can add a *scan.parametervalues.provider.class* in order to customize the splitting of the query (we can also add a check that the query contains at least 1 question mark). If we introduce a custom parameters provider we need also to specify parameters, using something like: 'scan.parametervalues.0.name' = 'minDate', 'scan.parametervalues.0.value'= '12/10/2019' 'scan.parametervalues.1.name' = 'maxDate', 'scan.parametervalues.1.value'= '01/01/2020' [~lzljs3620320] Maybe we need add something like "scan.parametervalues.provider.type", it can be "bound, specify, custom": - when *bound*, using old p*artitionLowerBound* and *partitionUpperBound*, *numPartitions* - when *specify*, using specify parameters like your proposal - when *custom*, need *scan.parametervalues.provider.class* Actually I don't know if specify and custom can be separated..but this can be further discussed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17359) Entropy key is not resolved if flink-s3-fs-hadoop is added as a plugin
Prem Santosh created FLINK-17359:
Summary: Entropy key is not resolved if flink-s3-fs-hadoop is
added as a plugin
Key: FLINK-17359
URL: https://issues.apache.org/jira/browse/FLINK-17359
Project: Flink
Issue Type: Bug
Reporter: Prem Santosh
Using flink 1.10
I added the flink-s3-fs-hadoop jar in plugins dir but I am seeing the
checkpoints paths like
{{s3://my_app/__ENTROPY__/app_name-staging/flink/checkpoints/e10f47968ae74934bd833108d2272419/chk-3071}}
which means the entropy injection key is not being resolved. After some
debugging I found that in the
[EntropyInjector|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/EntropyInjector.java#L97]
we check if the given fileSystem is of type {{SafetyNetWrapperFileSystem}} and
if so we check if the delegate is of type {{EntropyInjectingFileSystem}} but
don't check for {{[ClassLoaderFixingFileSystem
|https://github.com/apache/flink/blob/release-1.10.0/flink-core/src/main/java/org/apache/flink/core/fs/PluginFileSystemFactory.java#L65]}}
which would be the type if S3 file system dependencies are added as a plugin.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17358) JDBCTableSource support FiltertableTableSource
Flavio Pompermaier created FLINK-17358: -- Summary: JDBCTableSource support FiltertableTableSource Key: FLINK-17358 URL: https://issues.apache.org/jira/browse/FLINK-17358 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: Flavio Pompermaier Let JDBCTableSource implement FiltertableTableSource -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17357) add "DROP catalog" DDL to blink planner
Flavio Pompermaier created FLINK-17357: -- Summary: add "DROP catalog" DDL to blink planner Key: FLINK-17357 URL: https://issues.apache.org/jira/browse/FLINK-17357 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Flavio Pompermaier https://cwiki.apache.org/confluence/display/FLINK/FLIP+69+-+Flink+SQL+DDL+Enhancement some customers who have internal streaming platform requested this feature, as it's not possible on a platform to load catalogs dynamically at runtime now via sql client yaml. Catalog DDL will come into play (Similarly to https://issues.apache.org/jira/browse/FLINK-15349) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17356) Properly set constraints (PK and UNIQUE)
Flavio Pompermaier created FLINK-17356: -- Summary: Properly set constraints (PK and UNIQUE) Key: FLINK-17356 URL: https://issues.apache.org/jira/browse/FLINK-17356 Project: Flink Issue Type: Sub-task Components: Connectors / JDBC Reporter: Flavio Pompermaier At the moment the PostgresCatalog does not create field constraints (at the moment there's only UNIQUE and PRIMARY_KEY in the TableSchema..could it worth to add also NOT_NULL?) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17355) Exactly once kafka checkpointing sensitive to single node failures
Teng Fei Liao created FLINK-17355: - Summary: Exactly once kafka checkpointing sensitive to single node failures Key: FLINK-17355 URL: https://issues.apache.org/jira/browse/FLINK-17355 Project: Flink Issue Type: Bug Components: Connectors / Kafka Affects Versions: 1.10.0 Reporter: Teng Fei Liao With exactly one semantics, when checkpointing, FlinkKafkaProducer creates a new KafkaProducer for each checkpoint. KafkaProducer#initTransactions can timeout if a kafka node becomes unavailable, even in the case of multiple brokers and in-sync replicas (see [https://stackoverflow.com/questions/55955379/enabling-exactly-once-causes-streams-shutdown-due-to-timeout-while-initializing]). In non-flink cases, this might be fine since I imagine a KafkaProducer is not created very often. With flink however, this is called per checkpoint which means practically an HA kafka cluster isn't actually HA. This makes rolling a kafka node particularly painful even in intentional cases such as config changes or upgrades. In our specific setup, these are our settings: 5 kafka nodes Per topic, we have a replication factor = 3 and in-sync replicas = 2 and partitions = 3 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Exact feature freeze date
+1 for extending the code freeze date. FLIP-119 could benefit from it. May 15th sounds reasonable. Thanks, Zhu Zhu Jark Wu 于2020年4月24日周五 上午12:01写道： > +1 > > Thanks, > Jark > > On Thu, 23 Apr 2020 at 22:36, Xintong Song wrote: > > > +1 > > From our side we can also benefit from the extending of feature freeze, > for > > pluggable slot allocation, GPU support and perjob mode on Kubernetes > > deployment. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther > wrote: > > > > > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > > > from extending the feature freeze. > > > > > > Thanks, > > > Timo > > > > > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > > > +1 > > > > > > > > Aljoscha > > > > > > > > On 23.04.20 15:23, Till Rohrmann wrote: > > > >> +1 for extending the feature freeze until May 15th. > > > >> > > > >> Cheers, > > > >> Till > > > >> > > > >> On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski > > > > >> wrote: > > > >> > > > >>> Hi Stephan, > > > >>> > > > >>> As release manager I’ve seen that quite a bit of features could use > > > >>> of the > > > >>> extra couple of weeks. This also includes some features that I’m > > > >>> involved > > > >>> with, like FLIP-76, or limiting the in-flight buffers. > > > >>> > > > >>> +1 From my side for extending the feature freeze until May 15th. > > > >>> > > > >>> Piotrek > > > >>> > > > On 23 Apr 2020, at 10:10, Stephan Ewen wrote: > > > > > > Hi all! > > > > > > I want to bring up a discussion about when we want to do the > feature > > > >>> freeze > > > for 1.11. > > > > > > When kicking off the release cycle, we tentatively set the date to > > > end of > > > April, which would be in one week. > > > > > > I can say from the features I am involved with (FLIP-27, FLIP-115, > > > reviewing some state backend improvements, etc.) that it would be > > > helpful > > > to have two additional weeks. > > > > > > When looking at various other feature threads, my feeling is that > > > there > > > >>> are > > > more contributors and committers that could use a few more days. > > > The last two months were quite exceptional in and we did lose a > bit > > of > > > development speed here and there. > > > > > > How do you think about making *May 15th* the feature freeze? > > > > > > Best, > > > Stephan > > > >>> > > > >>> > > > >> > > > > > > > > >
Re: [DISCUSS] Exact feature freeze date
+1 Thanks, Jark On Thu, 23 Apr 2020 at 22:36, Xintong Song wrote: > +1 > From our side we can also benefit from the extending of feature freeze, for > pluggable slot allocation, GPU support and perjob mode on Kubernetes > deployment. > > Thank you~ > > Xintong Song > > > > On Thu, Apr 23, 2020 at 10:31 PM Timo Walther wrote: > > > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > > from extending the feature freeze. > > > > Thanks, > > Timo > > > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > > +1 > > > > > > Aljoscha > > > > > > On 23.04.20 15:23, Till Rohrmann wrote: > > >> +1 for extending the feature freeze until May 15th. > > >> > > >> Cheers, > > >> Till > > >> > > >> On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski > > >> wrote: > > >> > > >>> Hi Stephan, > > >>> > > >>> As release manager I’ve seen that quite a bit of features could use > > >>> of the > > >>> extra couple of weeks. This also includes some features that I’m > > >>> involved > > >>> with, like FLIP-76, or limiting the in-flight buffers. > > >>> > > >>> +1 From my side for extending the feature freeze until May 15th. > > >>> > > >>> Piotrek > > >>> > > On 23 Apr 2020, at 10:10, Stephan Ewen wrote: > > > > Hi all! > > > > I want to bring up a discussion about when we want to do the feature > > >>> freeze > > for 1.11. > > > > When kicking off the release cycle, we tentatively set the date to > > end of > > April, which would be in one week. > > > > I can say from the features I am involved with (FLIP-27, FLIP-115, > > reviewing some state backend improvements, etc.) that it would be > > helpful > > to have two additional weeks. > > > > When looking at various other feature threads, my feeling is that > > there > > >>> are > > more contributors and committers that could use a few more days. > > The last two months were quite exceptional in and we did lose a bit > of > > development speed here and there. > > > > How do you think about making *May 15th* the feature freeze? > > > > Best, > > Stephan > > >>> > > >>> > > >> > > > > >
[jira] [Created] (FLINK-17354) Introduce range sort for batch sort operator in blink
Benchao Li created FLINK-17354: -- Summary: Introduce range sort for batch sort operator in blink Key: FLINK-17354 URL: https://issues.apache.org/jira/browse/FLINK-17354 Project: Flink Issue Type: Improvement Components: Table SQL / Planner, Table SQL / Runtime Reporter: Benchao Li The original question in user ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/batch-range-sort-support-td34600.html Currently blink planner only support global sort operator which maybe a bottleneck in large data scale. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17353) Broken links in Flink docs master
Seth Wiesman created FLINK-17353: Summary: Broken links in Flink docs master Key: FLINK-17353 URL: https://issues.apache.org/jira/browse/FLINK-17353 Project: Flink Issue Type: Bug Components: chinese-translation, Documentation Reporter: Seth Wiesman http://localhost:4000/concepts/programming-model.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/concepts/runtime.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/internals/stream_checkpointing.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/concepts/flink-architecture.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/ops/memory/config.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/concepts/programming-model.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/concepts/runtime.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/dev/dev/table/python/installation.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/internals/stream_checkpointing.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/dev/table/sql.html: Remote file does not exist -- broken link!!! -- http://localhost:4000/zh/ops/memory/config.html: Remote file does not exist -- broken link!!! -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Exact feature freeze date
+1 >From our side we can also benefit from the extending of feature freeze, for pluggable slot allocation, GPU support and perjob mode on Kubernetes deployment. Thank you~ Xintong Song On Thu, Apr 23, 2020 at 10:31 PM Timo Walther wrote: > From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit > from extending the feature freeze. > > Thanks, > Timo > > On 23.04.20 16:11, Aljoscha Krettek wrote: > > +1 > > > > Aljoscha > > > > On 23.04.20 15:23, Till Rohrmann wrote: > >> +1 for extending the feature freeze until May 15th. > >> > >> Cheers, > >> Till > >> > >> On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski > >> wrote: > >> > >>> Hi Stephan, > >>> > >>> As release manager I’ve seen that quite a bit of features could use > >>> of the > >>> extra couple of weeks. This also includes some features that I’m > >>> involved > >>> with, like FLIP-76, or limiting the in-flight buffers. > >>> > >>> +1 From my side for extending the feature freeze until May 15th. > >>> > >>> Piotrek > >>> > On 23 Apr 2020, at 10:10, Stephan Ewen wrote: > > Hi all! > > I want to bring up a discussion about when we want to do the feature > >>> freeze > for 1.11. > > When kicking off the release cycle, we tentatively set the date to > end of > April, which would be in one week. > > I can say from the features I am involved with (FLIP-27, FLIP-115, > reviewing some state backend improvements, etc.) that it would be > helpful > to have two additional weeks. > > When looking at various other feature threads, my feeling is that > there > >>> are > more contributors and committers that could use a few more days. > The last two months were quite exceptional in and we did lose a bit of > development speed here and there. > > How do you think about making *May 15th* the feature freeze? > > Best, > Stephan > >>> > >>> > >> > >
Re: [DISCUSS] Exact feature freeze date
From the SQL side, I'm sure that FLIP-95 and FLIP-105 could benefit from extending the feature freeze. Thanks, Timo On 23.04.20 16:11, Aljoscha Krettek wrote: +1 Aljoscha On 23.04.20 15:23, Till Rohrmann wrote: +1 for extending the feature freeze until May 15th. Cheers, Till On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski wrote: Hi Stephan, As release manager I’ve seen that quite a bit of features could use of the extra couple of weeks. This also includes some features that I’m involved with, like FLIP-76, or limiting the in-flight buffers. +1 From my side for extending the feature freeze until May 15th. Piotrek On 23 Apr 2020, at 10:10, Stephan Ewen wrote: Hi all! I want to bring up a discussion about when we want to do the feature freeze for 1.11. When kicking off the release cycle, we tentatively set the date to end of April, which would be in one week. I can say from the features I am involved with (FLIP-27, FLIP-115, reviewing some state backend improvements, etc.) that it would be helpful to have two additional weeks. When looking at various other feature threads, my feeling is that there are more contributors and committers that could use a few more days. The last two months were quite exceptional in and we did lose a bit of development speed here and there. How do you think about making *May 15th* the feature freeze? Best, Stephan
Re: [DISCUSS] Removing deprecated state methods in 1.11
Definitely +1! I'm always game for decreasing the API surface if it doesn't decrease functionality. Aljoscha On 23.04.20 14:18, DONG, Weike wrote: Hi Stephan, +1 for the removal, as there are so many deprecated methods scattered around, making APIs a little bit messy and confusing. Best, Weike Stephan Ewen 于2020年4月23日 周四下午8:07写道： Hi all! There are a bunch of deprecated methods for state access: - RuntimeContext.getFoldingState(...) - OperatorStateStore.getSerializableListState(...) - OperatorStateStore.getOperatorState(...) All of them have been deprecated for around three years now. All have good alternatives. Time to remove them in 1.11? Best, Stephan
Re: [DISCUSS] Exact feature freeze date
+1 Aljoscha On 23.04.20 15:23, Till Rohrmann wrote: +1 for extending the feature freeze until May 15th. Cheers, Till On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski wrote: Hi Stephan, As release manager I’ve seen that quite a bit of features could use of the extra couple of weeks. This also includes some features that I’m involved with, like FLIP-76, or limiting the in-flight buffers. +1 From my side for extending the feature freeze until May 15th. Piotrek On 23 Apr 2020, at 10:10, Stephan Ewen wrote: Hi all! I want to bring up a discussion about when we want to do the feature freeze for 1.11. When kicking off the release cycle, we tentatively set the date to end of April, which would be in one week. I can say from the features I am involved with (FLIP-27, FLIP-115, reviewing some state backend improvements, etc.) that it would be helpful to have two additional weeks. When looking at various other feature threads, my feeling is that there are more contributors and committers that could use a few more days. The last two months were quite exceptional in and we did lose a bit of development speed here and there. How do you think about making *May 15th* the feature freeze? Best, Stephan
Re: [PROPOSAL] Google Season of Docs 2020.
Hi, Manish. If you would like to contribute to Flink, please read the "How to Contribute" section on the website [1]. This thread has a different scope. Feel free to reach out (in a separate thread) if you have any questions. Marta [1] https://flink.apache.org/contributing/how-to-contribute.html On Thu, Apr 23, 2020 at 3:44 PM Manish G wrote: > Hi, > > I would also like to contribute to documentation and unit testing. > Please let me know. > > Manish > > On Thu, Apr 23, 2020 at 7:08 PM Marta Paes Moreira > wrote: > > > > Thanks for the feedback! > > > > So far, the projects on the table are: > > > >1. Improving the Table API/SQL documentation. > >2. Improving the documentation about Deployments. > >3. Restructuring and standardizing the documentation about Connectors. > >4. Finishing the Chinese translation. > > > > I think 2. would require a lot of technical knowledge about Flink, which > > might not be a good fit for GSoD (as discussed last year). > > > > As for mentors, we have: > > > >- Aljoscha (Table API/SQL) > >- Till (Deployments) > >- Stephan also said he'd be happy to participate as a mentor if > needed. > > > > For the translation project, I'm pulling in the people involved in last > > year's thread (Jark and Jincheng), as we would need two chinese-speaking > > mentors. > > > > I'll follow up with a draft proposal early next week, once we reach a > > consensus and have enough mentors (2 per project). Thanks again! > > > > Marta > > > > > > On Fri, Apr 17, 2020 at 2:53 PM Till Rohrmann > wrote: > > > > > Thanks for driving this effort Marta. > > > > > > I'd be up for mentoring improvements for the deployment section as > > > described in FLIP-42. > > > > > > Cheers, > > > Till > > > > > > On Fri, Apr 17, 2020 at 11:20 AM Aljoscha Krettek > > > > wrote: > > > > > > > Hi, > > > > > > > > first, excellent that you're driving this, Marta! > > > > > > > > By now I have made quite some progress on the FLIP-42 restructuring > so > > > > that is not a good effort for someone to join now. Plus there is also > > > > [1], which is about incorporating the existing Flink Training > material > > > > into the concepts section of the Flink doc. > > > > > > > > What I think would be very good is working on the Table API/SQL > > > > documentation [2]. We don't necessarily have to take the FLIP as a > basis > > > > but we can, or we can start from a blank slate. I think the current > > > > structure as well as the content is sub-optimal (not good, really). > It > > > > would be ideal to have someone get to now the system and then write > > > > documentation for that part of Flink that has both good structure and > > > > content and nicely guides new users. > > > > > > > > I would be very happy to mentor that effort. > > > > > > > > Best, > > > > Aljoscha > > > > > > > > [1] > > > > > > > > > > > > https://lists.apache.org/thread.html/rea9cbfffd9a1c0ca6f78f7e8c8497d81f32ed4a7b9e7a05d59f3b2e9%40%3Cdev.flink.apache.org%3E > > > > > > > > [2] > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685 > > > > > > > > On 17.04.20 09:21, Robert Metzger wrote: > > > > > Thanks a lot for volunteering to drive an application for the Flink > > > > project! > > > > > > > > > > Last year, we discussed finishing the chinese translation as a > > > potential > > > > > project. I believe there's still a need for this. > > > > > Since the work on the project starts pretty far in the future > > > > (September), > > > > > the translation project is a good fit as well (there's currently no > > > major > > > > > effort on the translation, rather a constant flow of PRs, but I > don't > > > > think > > > > > that is enough to finish the translation). > > > > > > > > > > > > > > > On Fri, Apr 17, 2020 at 9:15 AM Konstantin Knauf < > kna...@apache.org> > > > > wrote: > > > > > > > > > >> Hi Marta, > > > > >> > > > > >> Thanks for kicking off the discussion. Aljoscha has recently > revived > > > the > > > > >> implementation of the FLIP-42 and has already moved things around > > > quite > > > > a > > > > >> bit. [1] > > > > >> > > > > >> There are a lot of areas that can be improved of course, but a > lot of > > > > them > > > > >> require very deep knowledge about the system (e.g. the > "Deployment" or > > > > >> "Concepts" section). One area that I could imagine working well in > > > such > > > > a > > > > >> format is to work on the "Connectors" section. Aljoscha has > already > > > > moved > > > > >> this to the top-level, but it besides that it has not been > touched yet > > > > in > > > > >> the course of FLIP-42. The documentation project could be around > > > > >> restructuring, standardization and generally improving the > > > > documentation of > > > > >> our connectors for both Datastream as well as Table API/SQL. > > > > >> > > > > >> Cheers, > > > > >> > > > > >> Konstantin > > > > >> > > > > >> [1] https://ci.apache.org/projects/flink/flink-docs-master/ > >
Re: [PROPOSAL] Google Season of Docs 2020.
Hi, I would also like to contribute to documentation and unit testing. Please let me know. Manish On Thu, Apr 23, 2020 at 7:08 PM Marta Paes Moreira wrote: > > Thanks for the feedback! > > So far, the projects on the table are: > >1. Improving the Table API/SQL documentation. >2. Improving the documentation about Deployments. >3. Restructuring and standardizing the documentation about Connectors. >4. Finishing the Chinese translation. > > I think 2. would require a lot of technical knowledge about Flink, which > might not be a good fit for GSoD (as discussed last year). > > As for mentors, we have: > >- Aljoscha (Table API/SQL) >- Till (Deployments) >- Stephan also said he'd be happy to participate as a mentor if needed. > > For the translation project, I'm pulling in the people involved in last > year's thread (Jark and Jincheng), as we would need two chinese-speaking > mentors. > > I'll follow up with a draft proposal early next week, once we reach a > consensus and have enough mentors (2 per project). Thanks again! > > Marta > > > On Fri, Apr 17, 2020 at 2:53 PM Till Rohrmann wrote: > > > Thanks for driving this effort Marta. > > > > I'd be up for mentoring improvements for the deployment section as > > described in FLIP-42. > > > > Cheers, > > Till > > > > On Fri, Apr 17, 2020 at 11:20 AM Aljoscha Krettek > > wrote: > > > > > Hi, > > > > > > first, excellent that you're driving this, Marta! > > > > > > By now I have made quite some progress on the FLIP-42 restructuring so > > > that is not a good effort for someone to join now. Plus there is also > > > [1], which is about incorporating the existing Flink Training material > > > into the concepts section of the Flink doc. > > > > > > What I think would be very good is working on the Table API/SQL > > > documentation [2]. We don't necessarily have to take the FLIP as a basis > > > but we can, or we can start from a blank slate. I think the current > > > structure as well as the content is sub-optimal (not good, really). It > > > would be ideal to have someone get to now the system and then write > > > documentation for that part of Flink that has both good structure and > > > content and nicely guides new users. > > > > > > I would be very happy to mentor that effort. > > > > > > Best, > > > Aljoscha > > > > > > [1] > > > > > > > > https://lists.apache.org/thread.html/rea9cbfffd9a1c0ca6f78f7e8c8497d81f32ed4a7b9e7a05d59f3b2e9%40%3Cdev.flink.apache.org%3E > > > > > > [2] > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685 > > > > > > On 17.04.20 09:21, Robert Metzger wrote: > > > > Thanks a lot for volunteering to drive an application for the Flink > > > project! > > > > > > > > Last year, we discussed finishing the chinese translation as a > > potential > > > > project. I believe there's still a need for this. > > > > Since the work on the project starts pretty far in the future > > > (September), > > > > the translation project is a good fit as well (there's currently no > > major > > > > effort on the translation, rather a constant flow of PRs, but I don't > > > think > > > > that is enough to finish the translation). > > > > > > > > > > > > On Fri, Apr 17, 2020 at 9:15 AM Konstantin Knauf > > > wrote: > > > > > > > >> Hi Marta, > > > >> > > > >> Thanks for kicking off the discussion. Aljoscha has recently revived > > the > > > >> implementation of the FLIP-42 and has already moved things around > > quite > > > a > > > >> bit. [1] > > > >> > > > >> There are a lot of areas that can be improved of course, but a lot of > > > them > > > >> require very deep knowledge about the system (e.g. the "Deployment" or > > > >> "Concepts" section). One area that I could imagine working well in > > such > > > a > > > >> format is to work on the "Connectors" section. Aljoscha has already > > > moved > > > >> this to the top-level, but it besides that it has not been touched yet > > > in > > > >> the course of FLIP-42. The documentation project could be around > > > >> restructuring, standardization and generally improving the > > > documentation of > > > >> our connectors for both Datastream as well as Table API/SQL. > > > >> > > > >> Cheers, > > > >> > > > >> Konstantin > > > >> > > > >> [1] https://ci.apache.org/projects/flink/flink-docs-master/ > > > >> > > > >> On Wed, Apr 15, 2020 at 12:11 PM Marta Paes Moreira < > > > ma...@ververica.com> > > > >> wrote: > > > >> > > > >>> Hi, Everyone. > > > >>> > > > >>> Google is running its Season of Docs [1] program again this year. The > > > >> goal > > > >>> of the program is to pair open source organizations/projects with > > > >>> professional technical writers to improve their documentation. > > > >>> > > > >>> The Flink community submitted an application in 2019 (led by > > > Konstantin) > > > >>> [2,3], but was unfortunately not accepted into the program. This > > year, > > > >> I'm > > > >>> volunteering to write and submit the proposal in the
Re: [PROPOSAL] Google Season of Docs 2020.
Thanks for the feedback! So far, the projects on the table are: 1. Improving the Table API/SQL documentation. 2. Improving the documentation about Deployments. 3. Restructuring and standardizing the documentation about Connectors. 4. Finishing the Chinese translation. I think 2. would require a lot of technical knowledge about Flink, which might not be a good fit for GSoD (as discussed last year). As for mentors, we have: - Aljoscha (Table API/SQL) - Till (Deployments) - Stephan also said he'd be happy to participate as a mentor if needed. For the translation project, I'm pulling in the people involved in last year's thread (Jark and Jincheng), as we would need two chinese-speaking mentors. I'll follow up with a draft proposal early next week, once we reach a consensus and have enough mentors (2 per project). Thanks again! Marta On Fri, Apr 17, 2020 at 2:53 PM Till Rohrmann wrote: > Thanks for driving this effort Marta. > > I'd be up for mentoring improvements for the deployment section as > described in FLIP-42. > > Cheers, > Till > > On Fri, Apr 17, 2020 at 11:20 AM Aljoscha Krettek > wrote: > > > Hi, > > > > first, excellent that you're driving this, Marta! > > > > By now I have made quite some progress on the FLIP-42 restructuring so > > that is not a good effort for someone to join now. Plus there is also > > [1], which is about incorporating the existing Flink Training material > > into the concepts section of the Flink doc. > > > > What I think would be very good is working on the Table API/SQL > > documentation [2]. We don't necessarily have to take the FLIP as a basis > > but we can, or we can start from a blank slate. I think the current > > structure as well as the content is sub-optimal (not good, really). It > > would be ideal to have someone get to now the system and then write > > documentation for that part of Flink that has both good structure and > > content and nicely guides new users. > > > > I would be very happy to mentor that effort. > > > > Best, > > Aljoscha > > > > [1] > > > > > https://lists.apache.org/thread.html/rea9cbfffd9a1c0ca6f78f7e8c8497d81f32ed4a7b9e7a05d59f3b2e9%40%3Cdev.flink.apache.org%3E > > > > [2] > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=127405685 > > > > On 17.04.20 09:21, Robert Metzger wrote: > > > Thanks a lot for volunteering to drive an application for the Flink > > project! > > > > > > Last year, we discussed finishing the chinese translation as a > potential > > > project. I believe there's still a need for this. > > > Since the work on the project starts pretty far in the future > > (September), > > > the translation project is a good fit as well (there's currently no > major > > > effort on the translation, rather a constant flow of PRs, but I don't > > think > > > that is enough to finish the translation). > > > > > > > > > On Fri, Apr 17, 2020 at 9:15 AM Konstantin Knauf > > wrote: > > > > > >> Hi Marta, > > >> > > >> Thanks for kicking off the discussion. Aljoscha has recently revived > the > > >> implementation of the FLIP-42 and has already moved things around > quite > > a > > >> bit. [1] > > >> > > >> There are a lot of areas that can be improved of course, but a lot of > > them > > >> require very deep knowledge about the system (e.g. the "Deployment" or > > >> "Concepts" section). One area that I could imagine working well in > such > > a > > >> format is to work on the "Connectors" section. Aljoscha has already > > moved > > >> this to the top-level, but it besides that it has not been touched yet > > in > > >> the course of FLIP-42. The documentation project could be around > > >> restructuring, standardization and generally improving the > > documentation of > > >> our connectors for both Datastream as well as Table API/SQL. > > >> > > >> Cheers, > > >> > > >> Konstantin > > >> > > >> [1] https://ci.apache.org/projects/flink/flink-docs-master/ > > >> > > >> On Wed, Apr 15, 2020 at 12:11 PM Marta Paes Moreira < > > ma...@ververica.com> > > >> wrote: > > >> > > >>> Hi, Everyone. > > >>> > > >>> Google is running its Season of Docs [1] program again this year. The > > >> goal > > >>> of the program is to pair open source organizations/projects with > > >>> professional technical writers to improve their documentation. > > >>> > > >>> The Flink community submitted an application in 2019 (led by > > Konstantin) > > >>> [2,3], but was unfortunately not accepted into the program. This > year, > > >> I'm > > >>> volunteering to write and submit the proposal in the upcoming weeks. > To > > >>> achieve this, there are a few things that need to be sorted out in > > >> advance: > > >>> > > >>> - > > >>> *Mentors *Each proposed project idea requires at least two volunteers > > to > > >>> mentor technical writers through the process. *Who would like to > > >>> participate as a mentor*? You can read about the responsibilities > > here > > >>> [4]. > > >>> > > >>> > > >>> - >
Re: [DISCUSS] Exact feature freeze date
+1 for extending the feature freeze until May 15th. Cheers, Till On Thu, Apr 23, 2020 at 1:00 PM Piotr Nowojski wrote: > Hi Stephan, > > As release manager I’ve seen that quite a bit of features could use of the > extra couple of weeks. This also includes some features that I’m involved > with, like FLIP-76, or limiting the in-flight buffers. > > +1 From my side for extending the feature freeze until May 15th. > > Piotrek > > > On 23 Apr 2020, at 10:10, Stephan Ewen wrote: > > > > Hi all! > > > > I want to bring up a discussion about when we want to do the feature > freeze > > for 1.11. > > > > When kicking off the release cycle, we tentatively set the date to end of > > April, which would be in one week. > > > > I can say from the features I am involved with (FLIP-27, FLIP-115, > > reviewing some state backend improvements, etc.) that it would be helpful > > to have two additional weeks. > > > > When looking at various other feature threads, my feeling is that there > are > > more contributors and committers that could use a few more days. > > The last two months were quite exceptional in and we did lose a bit of > > development speed here and there. > > > > How do you think about making *May 15th* the feature freeze? > > > > Best, > > Stephan > >
[jira] [Created] (FLINK-17352) All doc links w/ site.baseurl & link tag are broken
David Anderson created FLINK-17352:
--
Summary: All doc links w/ site.baseurl & link tag are broken
Key: FLINK-17352
URL: https://issues.apache.org/jira/browse/FLINK-17352
Project: Flink
Issue Type: Improvement
Components: Documentation
Reporter: David Anderson
Assignee: David Anderson
Using {{ site.baseurl }}{% link foo.md %} creates a link containing something
like
https://ci.apache.org/projects/flink/flink-docs-master//ci.apache.org/projects/flink/flink-docs-master/concepts/stateful-stream-processing.html
The link tag includes site.baseurl, so no need to include it again.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] FLIP-124: Add open/close and Collector to (De)SerializationSchema
Hi! Sorry for being a bit late to the party. One very important thing to consider for "serialization under checkpoint lock or not" is: - If you do it under checkpoint lock, things are automatically correct. Checkpoint barriers go between original records that correspond to offsets in the source. - If you deserialize outside the checkpoint lock, then you read a record from the source but only partially emit it. In that case you need to store the difference (not emitted part) in the checkpoint. ==> I would advise against trying to emit partial records, i.e. doing things outside the checkpoint lock. FLIP-27 will by default also not do partial emission of unnested events. Also, it is questionable whether optimizing this in the source makes sense when no other operator supports that (flatMap, etc.). Regarding Seth's comment about performance: - For that it does probably makes not so much difference whether this is under lock or not, but more whether this can be pushed to another thread (source's I/O thread), so that it does not add load to the main task processing thread. ==> This means that the I/O thread deserialized that "batch" that it hands over. ==> Still, it is important that all records coming from one original source record are emitted atomically, otherwise we have the same issue as above. Best, Stephan On Tue, Apr 14, 2020 at 10:35 AM Dawid Wysakowicz wrote: > Hi Xiaogang, > > I very much agree with Jark's and Aljoscha's responses. > > > On 10/04/2020 17:35, Jark Wu wrote: > > Hi Xiaogang, > > > > I think this proposal doesn't conflict with your use case, you can still > > chain a ProcessFunction after a source which emits raw data. > > But I'm not in favor of chaining ProcessFunction after source, and we > > should avoid that, because: > > > > 1) For correctness, it is necessary to perform the watermark generation > as > > early as possible in order to be close to the actual data > > generation within a source's data partition. This is also the purpose of > > per-partition watermark and event-time alignment. > > Many on going FLIPs (e.g. FLIP-27, FLIP-95) works a lot on this effort. > > Deseriazing records and generating watermark in chained > > ProcessFunction makes it difficult to do per-partition watermark in the > > future. > > 2) In Flink SQL, a source should emit the deserialized row instead of raw > > data. Otherwise, users have to define raw byte[] as the > > single column of the defined table, and parse them in queries, which is > > very inconvenient. > > > > Best, > > Jark > > > > On Fri, 10 Apr 2020 at 09:18, SHI Xiaogang > wrote: > > > >> Hi, > >> > >> I don't think the proposal is a good solution to the problems. I am in > >> favour of using a ProcessFunction chained to the source/sink function to > >> serialize/deserialize the records, instead of embedding > (de)serialization > >> schema in source/sink function. > >> > >> Message packing is heavily used in our production environment to allow > >> compression and improve throughput. As buffered messages have to be > >> delivered when the time exceeds the limit, timers are also required in > our > >> cases. I think it's also a common need for other users. > >> > >> In the this proposal, with more components added into the context, in > the > >> end we will find the serialization/deserialization schema is just > another > >> wrapper of ProcessFunction. > >> > >> Regards, > >> Xiaogang > >> > >> Aljoscha Krettek 于2020年4月7日周二 下午6:34写道： > >> > >>> On 07.04.20 08:45, Dawid Wysakowicz wrote: > >>> > @Jark I was aware of the implementation of SinkFunction, but it was a > conscious choice to not do it that way. > > Personally I am against giving a default implementation to both the > new > and old methods. This results in an interface that by default does > nothing or notifies the user only in the runtime, that he/she has not > implemented a method of the interface, which does not sound like a > good > practice to me. Moreover I believe the method without a Collector will > still be the preferred method by many users. Plus it communicates > explicitly what is the minimal functionality required by the > interface. > Nevertheless I am happy to hear other opinions. > >>> Dawid and I discussed this before. I did the extension of the > >>> SinkFunction but by now I think it's better to do it this way, because > >>> otherwise you can implement the interface without implementing any > >> methods. > @all I also prefer the buffering approach. Let's wait a day or two > more > to see if others think differently. > >>> I'm also in favour of buffering outside the lock. > >>> > >>> Also, +1 to this FLIP. > >>> > >>> Best, > >>> Aljoscha > >>> > >
[jira] [Created] (FLINK-17351) CheckpointCoordinator and CheckpointFailureManager ignores checkpoint timeouts
Piotr Nowojski created FLINK-17351:
--
Summary: CheckpointCoordinator and CheckpointFailureManager
ignores checkpoint timeouts
Key: FLINK-17351
URL: https://issues.apache.org/jira/browse/FLINK-17351
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.10.0, 1.9.2
Reporter: Piotr Nowojski
As described in point 2:
https://issues.apache.org/jira/browse/FLINK-17327?focusedCommentId=17090576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17090576
(copy of description from above linked comment):
The logic in how {{CheckpointCoordinator}} handles checkpoint timeouts is
broken. In your [~qinjunjerry] examples, your job should have failed after
first checkpoint failure, but checkpoints were time outing on
CheckpointCoordinator after 5 seconds, before {{FlinkKafkaProducer}} was
detecting Kafka failure after 2 minutes. Those timeouts were not checked
against {{setTolerableCheckpointFailureNumber(...)}} limit, so the job was keep
going with many timed out checkpoints. Now funny thing happens:
FlinkKafkaProducer detects Kafka failure. Funny thing is that it depends where
the failure was detected:
a) on processing record? no problem, job will failover immediately once failure
is detected (in this example after 2 minutes)
b) on checkpoint? heh, the failure is reported to {{CheckpointCoordinator}}
*and gets ignored, as PendingCheckpoint has already been discarded 2 minutes
ago* :) So theoretically the checkpoints can keep failing forever and the job
will not restart automatically, unless something else fails.
Even more funny things can happen if we mix FLINK-17350 . or b) with
intermittent external system failure. Sink reports an exception, transaction
was lost/aborted, Sink is in failed state, but if there will be a happy
coincidence that it manages to accept further records, this exception can be
lost and all of the records in those failed checkpoints will be lost forever as
well. In all of the examples that [~qinjunjerry] posted it hasn't happened.
{{FlinkKafkaProducer}} was not able to recover after the initial failure and it
was keep throwing exceptions until the job finally failed (but much later then
it should have). And that's not guaranteed anywhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17350) StreamTask should always fail immediately on failures in synchronous part of a checkpoint
Piotr Nowojski created FLINK-17350:
--
Summary: StreamTask should always fail immediately on failures in
synchronous part of a checkpoint
Key: FLINK-17350
URL: https://issues.apache.org/jira/browse/FLINK-17350
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Runtime / Task
Affects Versions: 1.10.0, 1.9.2, 1.8.3, 1.7.2, 1.6.4
Reporter: Piotr Nowojski
This bugs also Affects 1.5.x branch.
As described
https://issues.apache.org/jira/browse/FLINK-17327?focusedCommentId=17090576=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17090576
{{setTolerableCheckpointFailureNumber(...)}} and its deprecated
{{setFailTaskOnCheckpointError(...)}} predecessor are implemented incorrectly.
Since Flink 1.5 (https://issues.apache.org/jira/browse/FLINK-4809) they can
lead to operators (and especially sinks with an external state) end up in an
inconsistent state. That's also true even if they are not used, because of
another issue: PLACEHOLDER
For details please check FLINK-17327.
The problem boils down to a fact, that if operator/user functions throws an
exception, job should always fail. There is no recovery from this. In case of
{{FlinkKafkaProducer}} ignoring such failures might mean that whole transaction
with all of it's records will be lost forever.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17349) Reduce runtime of LocalExecutorITCase
Aljoscha Krettek created FLINK-17349:
Summary: Reduce runtime of LocalExecutorITCase
Key: FLINK-17349
URL: https://issues.apache.org/jira/browse/FLINK-17349
Project: Flink
Issue Type: Improvement
Components: Table SQL / Client, Table SQL / Legacy Planner, Table SQL
/ Planner, Table SQL / Runtime
Reporter: Aljoscha Krettek
Running the while ITCase takes ~3 minutes on my machine, which is not
acceptable for developer productivity and is also quite the burden on our CI
systems and PR iteration time.
The issue is mostly that this does many costly operations, such as compiling
SQL queries. Some tests are inefficient in that they do a lot more repetitions
or test things that are not needed here. Also {{LocalExecutor}} itself is a bit
wasteful because every time a session property is changed, when opening a
session, and for other things we trigger reloading/re-parsing the environment,
which means all the defined catalogs, sources/sinks, and views.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Removing deprecated state methods in 1.11
Hi Stephan, +1 for the removal, as there are so many deprecated methods scattered around, making APIs a little bit messy and confusing. Best, Weike Stephan Ewen 于2020年4月23日 周四下午8:07写道： > Hi all! > > There are a bunch of deprecated methods for state access: > > - RuntimeContext.getFoldingState(...) > - OperatorStateStore.getSerializableListState(...) > - OperatorStateStore.getOperatorState(...) > > All of them have been deprecated for around three years now. All have good > alternatives. > > Time to remove them in 1.11? > > Best, > Stephan >
[DISCUSS] Removing deprecated state methods in 1.11
Hi all! There are a bunch of deprecated methods for state access: - RuntimeContext.getFoldingState(...) - OperatorStateStore.getSerializableListState(...) - OperatorStateStore.getOperatorState(...) All of them have been deprecated for around three years now. All have good alternatives. Time to remove them in 1.11? Best, Stephan
[jira] [Created] (FLINK-17348) Expose metric group to ascendingTimestampExtractor
Theo Diefenthal created FLINK-17348: --- Summary: Expose metric group to ascendingTimestampExtractor Key: FLINK-17348 URL: https://issues.apache.org/jira/browse/FLINK-17348 Project: Flink Issue Type: Improvement Reporter: Theo Diefenthal A common use case in Flink + kafka is that one has lots of kafka Partitions with each having ascending timestamps. In my scenario, due to various operational reasons, we put log files from Filesystem to kafka, one server per partition, and then consume those in Flink. Sometimes, it can happen that we collect the files in wrong order into kafka which leads to ascending timestamp problems. If that happens and we have the default logging violation handler enabled, we produce several gb of logs in a very short amount of time, which we would like to circumvent. What we really want : track the number of violations in a metric and define an alarm on that in our monitoring dashboard. Currently, there is sadly no way to reference the metric group from the ascending timestamp extractor. I wish, there could be something similar like the open method on other rich functions. My current workaround is to add a custom map task post to the source. For that task I need to pass on the kafka partition from the source, which I usually don't care about and I need to keep track of each partitions current timestamp manually, exactly the same way as the extractor does. - > workaround with "polluting" my pipeline quite a bit just for a single metric. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Contribution
Hi Manish! Glad to hear that you're willing to contribute! All of our current tasks are on the JIRA issue list ( https://issues.apache.org/jira/projects/FLINK/issues). Feel free to pick what you like and start discussing and coding. Manish G 于2020年4月23日周四 下午7:35写道： > Hi All, > > I have just joined the dev list. > > I would like to contribute in my capacity as a programmer. > > Please let me know if I can be helpful to any of the current task. > > Manish >
Re: [VOTE] Release 1.9.3, release candidate #1
+1 (binding) - verified checksums and signatures - verified no binary artifacts in source release - reviewed the website PR - built from source - checked diff of source release between 1.9.2 for changes in licenses and dependencies - started standalone cluster and run StreamSQLExample Best, Dawid On 23/04/2020 12:09, Ufuk Celebi wrote: > +1 (binding) > > - checked release notes ✅ > - verified sums and hashes ✅ > - verified no binary artifacts in source release ✅ > - reviewed website PR ✅ > - built from source ✅ > - built an internal Flink distribution based on the 1.9.3-rc1 commit ✅ > - built internal jobs against the staging repo ✅ > - deployed those jobs to a 1.9.3 job cluster on Kubernetes and tested > checkpointing, SSL ✅ > > > On Thu, Apr 23, 2020 at 10:58 AM Robert Metzger wrote: > >> +1 (binding) >> >> - for license documentation, I checked the diff between 1.9.2 and 1.9.3: >> https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 >> - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc was >> updated in PR https://github.com/apache/flink/pull/11617/files >> - one test dependency change that is not relevant >> - checked the maven staging repo files >> - 1.9.3 set correctly, java quickstart looks fine >> - checked license files of some shaded jars >> >> >> >> On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: >> >>> +1 (non-binding) >>> >>> - checked/verified signatures and hashes >>> - checked the release note >>> - checked that there are no missing artifacts in staging area >>> - built from source sing scala 2.11 and using Scala 2.12 succeeded >>> - ran a couple of end-to-end tests locally and succeeded >>> - went through all commits checked in between 1.9.3 and 1.9.2, make sure >>> all issues set the "fixVersion" property >>> - started a cluster, WebUI was accessible, submitted a wordcount job and >>> ran succeeded, no suspicious log output >>> - the web PR looks good >>> >>> Best, >>> Leonard Xu >>> 在 2020年4月22日，17:58，Dian Fu 写道： +1 (non-binding) - verified the checksum and signature - checked the release note - checked that there are no new dependencies introduced since 1.9.2 >> which may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 which doesn't affect the license) - checked that there are no missing artifacts in staging area - checked that the pyflink package could be pip installed Regards, Dian On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < >>> fabianp...@data-artisans.com> wrote: > +1 (non-binding) > > - Verified signature > - Built from source (Java8) > - Run custom jobs on Kubernetes > > Regards, > Fabian > >> On 18. Apr 2020, at 04:37, Dian Fu wrote: >> >> Hi everyone, >> >> Please review and vote on the release candidate #1 for the version >>> 1.9.3, >> as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not approve the release (please provide specific comments) >> >> >> The complete staging area is available for your review, which >> includes: >> * JIRA release notes [1], >> * the official Apache source release and binary convenience releases >> to > be >> deployed to dist.apache.org [2], which are signed with the key with >> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], >> * all artifacts to be deployed to the Maven Central Repository [4], >> * source code tag "release-1.9.3-rc1" [5], >> * website pull request listing the new release and adding >> announcement > blog >> post [6]. >> >> The vote will be open for at least 72 hours. It is adopted by >> majority > approval, with at least 3 PMC affirmative votes. >> Thanks, >> Dian >> >> [1] >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 >> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ >> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >> [4] >> https://repository.apache.org/content/repositories/orgapacheflink-1353/ >> [5] >> https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 >> [6] https://github.com/apache/flink-web/pull/329 > >>> signature.asc Description: OpenPGP digital signature
Contribution
Hi All, I have just joined the dev list. I would like to contribute in my capacity as a programmer. Please let me know if I can be helpful to any of the current task. Manish
[DISCUSS] Move docker development into versioned branches
Hello everyone, Currently, all development in docker-flink occurs on the master branch, for all releases at once. The master branch also serves as a publishing area for our docker files. This means that all versions share the same generators, templates (i.e., production code that we plugin in some variables like the Flink version) and test setup. So far this worked fine because all versions worked the same, but we are now coming to a point where we are adding version-specific features, like support for Java 11 (FLINK-16260) and per-job-clusters (FLINK-17164). By virtue of all files being shared this means that we would have to introduce version checks in both the generators, templates and test code, so things continue to work for all versions. This is rather painful to do, and presents maintenance problems as these conditions must of course be updated at some point, and diverging behaviors require all versions to be tested on each PR (in contrast to the current state where we only test 1 version). It is also simply different to the process we're using for flink/flink-shaded, creating an additional entry barrier. I propose moving the development of the docker files into dedicated branches (dev-master, dev-1.10, dev-1.9). PullRequests will usually be opened against dev-master, possibly ported to other versions, and when it is time for a release we will update the master branch with the latest dockerfiles. Quite similar to how things work in the Flink repo. The master branch would continue to be our release publishing area to stick to docker conventions. Regards, Chesnay
Re: Reading a single input file in parallel?
Hi, Yes, you are correct. Flink does use a minSplitSize to determine the splits. I missed that part. Also this is the part I do not intend to change. In this I would focus on essentially: - "is splittable" is decided per file --> Similar to Hadoop: Both a codec and fileformat get a "isSplittable" method/marker that let's this to be decided per file at runtime. This should be the 'same' construct for all file formats (so also for Text, Avro, Parquet, etc.). Also if I create a new file format that for something that does not support splitting (like some specific datastructures using for example XML) then that should be cleanly possible. - The codecs must be overridable. For this https://github.com/nielsbasjes/splittablegzip to work the default gzip decompressor must be disabled. The current setup seems to make this possible but I'm not sure: https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L118 I intend to leave the way the splits are calculated to be as-is. Niels On Thu, Apr 23, 2020 at 11:33 AM Jingsong Li wrote: > Hi Niels, > > Thanks for start this discussion. Share some thought about your > questions/proposals. > > > Judge whether splittable for each individual file > Looks good to me. > > > BZIP2 support splittable > Looks good to me. > > > the Flink implementation is controlled by the number of splits > Can you check again? I think Flink is also affected by block size [1]. > > > Looking at how Hadoop does this I see that the FileInputFormat has a > method isSplitable > Now Flink do this in FileInputFormat, but also can do this like Hadoop > > I can see the split strategy in Hadoop orc and parquet are quite complex, I > don't have a lot of in-depth research, but I think these implementations > should be meaningful. > > [1] > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L644 > > Best, > Jingsong Lee > > On Thu, Apr 23, 2020 at 4:13 PM Niels Basjes wrote: > > > Hi, > > > > I wanted to spend some time on this idea I asked about a year ago. > > > > So I had a look at the actual code and found this where the file splits > are > > calculated > > > > > > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L601 > > > > What I realized is that apparently the current system for determining the > > FileInputSplit s > > 1) Lists the collection of input files. > > 2) If a file is NOT splittable set a global flag to false. > > 3) ONLY if all of them are splittable then will they be split. > > > > So (if I understand correctly) if I have an input directory with a 100GB > > uncompressed text file and a 1 KB gzipped file then BOTH will NOT be > split. > > Why is that done this way? > > > > I also found that the determination if a file is NOT splittable > completely > > rests on the fact that the decompression that is to be used is to be > > provided by an InflaterInputStreamFactory. > > Which is funny because the BZIP2 as a compression IS splittable but > > according to this implementation it isn't. > > > > I also noticed that the Hadoop split calculation is controlled by a split > > size, the Flink implementation is controlled by the number of splits. > > This seems logical to me as in Hadoop (MapReduce) the number of tasks > > running is dynamic. In contrast with Flink where the number of parallel > > tasks is a given setting. > > > > Looking at how Hadoop does this I see that the FileInputFormat has a > > method isSplitable > > (yes with typo, and with a terribly dangerous default value: > > MAPREDUCE-2094) > > which gives the answer on a per file basis. > > Some file formats (like TextInputFormat) look at the codec that was found > > to see if it implements the SplittableCompressionCodec interface to > > determine if the file is splittable. > > Other file formats (like Avro and Parquet) use something else to > determine > > this. > > > > Overall I currently think the way this has been implemented in Flink is > not > > very good. > > > > However looking at the gap to bridge to make it similar to what Hadoop > has > > seems like a huge step. > > > > I expect that many classes and interfaces will need to change > dramatically > > to make this happen. > > > > My main question to you guys: Do you think it is worth it? > > > > Niels Basjes > > > > > > On Mon, Feb 19, 2018 at 10:50 AM Fabian Hueske > wrote: > > > > > Hi Niels, > > > > > > Jörn is right, although offering different methods, Flink's InputFormat > > is > > > very similar to Hadoop's InputFormat interface. > > > The InputFormat.createInputSplits() method generates splits that can be > > > read in parallel. > > > The FileInputFormat splits files by fixed boundaries (usually HDFS > > > blocksize) and expects the InputFormat to find the right place to start > > and > > > end. > > > For line-wise read files (TextInputFormat) or
Re: [DISCUSS] Exact feature freeze date
Hi Stephan, As release manager I’ve seen that quite a bit of features could use of the extra couple of weeks. This also includes some features that I’m involved with, like FLIP-76, or limiting the in-flight buffers. +1 From my side for extending the feature freeze until May 15th. Piotrek > On 23 Apr 2020, at 10:10, Stephan Ewen wrote: > > Hi all! > > I want to bring up a discussion about when we want to do the feature freeze > for 1.11. > > When kicking off the release cycle, we tentatively set the date to end of > April, which would be in one week. > > I can say from the features I am involved with (FLIP-27, FLIP-115, > reviewing some state backend improvements, etc.) that it would be helpful > to have two additional weeks. > > When looking at various other feature threads, my feeling is that there are > more contributors and committers that could use a few more days. > The last two months were quite exceptional in and we did lose a bit of > development speed here and there. > > How do you think about making *May 15th* the feature freeze? > > Best, > Stephan
[GitHub] [flink-web] dianfu commented on issue #329: Add Apache Flink release 1.9.3
dianfu commented on issue #329: URL: https://github.com/apache/flink-web/pull/329#issuecomment-618330787 @carp84 @uce Thanks a lot for the review. I will update the post date and release date before merging this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17347) upgrade to latest angular version
Yadong Xie created FLINK-17347: -- Summary: upgrade to latest angular version Key: FLINK-17347 URL: https://issues.apache.org/jira/browse/FLINK-17347 Project: Flink Issue Type: Improvement Components: Runtime / Web Frontend Affects Versions: 1.11.0 Reporter: Yadong Xie angular 9.0 is released, upgrade to the latest angular version to provide better UI performance and make the web easier to maintain no break change -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release 1.9.3, release candidate #1
+1 (binding) - checked release notes ✅ - verified sums and hashes ✅ - verified no binary artifacts in source release ✅ - reviewed website PR ✅ - built from source ✅ - built an internal Flink distribution based on the 1.9.3-rc1 commit ✅ - built internal jobs against the staging repo ✅ - deployed those jobs to a 1.9.3 job cluster on Kubernetes and tested checkpointing, SSL ✅ On Thu, Apr 23, 2020 at 10:58 AM Robert Metzger wrote: > +1 (binding) > > - for license documentation, I checked the diff between 1.9.2 and 1.9.3: > https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 > - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc was > updated in PR https://github.com/apache/flink/pull/11617/files > - one test dependency change that is not relevant > - checked the maven staging repo files > - 1.9.3 set correctly, java quickstart looks fine > - checked license files of some shaded jars > > > > On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: > > > +1 (non-binding) > > > > - checked/verified signatures and hashes > > - checked the release note > > - checked that there are no missing artifacts in staging area > > - built from source sing scala 2.11 and using Scala 2.12 succeeded > > - ran a couple of end-to-end tests locally and succeeded > > - went through all commits checked in between 1.9.3 and 1.9.2, make sure > > all issues set the "fixVersion" property > > - started a cluster, WebUI was accessible, submitted a wordcount job and > > ran succeeded, no suspicious log output > > - the web PR looks good > > > > Best, > > Leonard Xu > > > > > 在 2020年4月22日，17:58，Dian Fu 写道： > > > > > > +1 (non-binding) > > > > > > - verified the checksum and signature > > > - checked the release note > > > - checked that there are no new dependencies introduced since 1.9.2 > which > > > may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 which > > > doesn't affect the license) > > > - checked that there are no missing artifacts in staging area > > > - checked that the pyflink package could be pip installed > > > > > > Regards, > > > Dian > > > > > > On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < > > fabianp...@data-artisans.com> > > > wrote: > > > > > >> +1 (non-binding) > > >> > > >> - Verified signature > > >> - Built from source (Java8) > > >> - Run custom jobs on Kubernetes > > >> > > >> Regards, > > >> Fabian > > >> > > >>> On 18. Apr 2020, at 04:37, Dian Fu wrote: > > >>> > > >>> Hi everyone, > > >>> > > >>> Please review and vote on the release candidate #1 for the version > > 1.9.3, > > >>> as follows: > > >>> [ ] +1, Approve the release > > >>> [ ] -1, Do not approve the release (please provide specific comments) > > >>> > > >>> > > >>> The complete staging area is available for your review, which > includes: > > >>> * JIRA release notes [1], > > >>> * the official Apache source release and binary convenience releases > to > > >> be > > >>> deployed to dist.apache.org [2], which are signed with the key with > > >>> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], > > >>> * all artifacts to be deployed to the Maven Central Repository [4], > > >>> * source code tag "release-1.9.3-rc1" [5], > > >>> * website pull request listing the new release and adding > announcement > > >> blog > > >>> post [6]. > > >>> > > >>> The vote will be open for at least 72 hours. It is adopted by > majority > > >> approval, with at least 3 PMC affirmative votes. > > >>> > > >>> Thanks, > > >>> Dian > > >>> > > >>> [1] > > >> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 > > >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > > >>> [4] > > >> > https://repository.apache.org/content/repositories/orgapacheflink-1353/ > > >>> [5] > > >> > > > https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 > > >>> [6] https://github.com/apache/flink-web/pull/329 > > >> > > >> > > > > >
Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop
This would only work so long as all Hadoop APIs do not directly expose any transitive non-hadoop dependency. Otherwise the user code classloader might search for this transitive dependency in lib instead of the hadoop classpath (and possibly not find it). On 23/04/2020 11:34, Stephan Ewen wrote: True, connectors built on Hadoop make this a bit more complex. That is also the reason why Hadoop is on the "parent first" patterns. Maybe this is a bit of a wild thought, but what would happen if we had a "first class" notion of a Hadoop Classloader in the system, and the user code classloader would explicitly fall back to that one whenever a class whose name starts with "org.apache.hadoop" is not found? We could also generalize this by associating plugin loaders with class name prefixes. Then it would try to load from the user code jar, and if the class was not found, load it from the hadoop classpath. On Thu, Apr 23, 2020 at 10:56 AM Chesnay Schepler wrote: although, if you can load the HADOOP_CLASSPATH as a plugin, then you can also load it in the user-code classloader. On 23/04/2020 10:50, Chesnay Schepler wrote: @Stephan I'm not aware of anyone having tried that; possibly since we have various connectors that require hadoop (hadoop-compat, hive, orc/parquet/hbase, hadoop inputformats). This would require connectors to be loaded as plugins (or having access to the plugin classloader) to be feasible. On 23/04/2020 09:59, Stephan Ewen wrote: Hi all! +1 for the simplification of dropping hadoop-shaded Have we ever investigated how much work it would be to load the HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy dependency footprint would not spoil the main classpath. - HDFS might be very simple, because file systems are already Plugin aware - Yarn would need some extra work. In essence, we would need to discover executors also through plugins - Kerberos is the other remaining bit. We would need to switch security modules to ServiceLoaders (which we should do anyways) and also pull them from plugins. Best, Stephan On Thu, Apr 23, 2020 at 4:05 AM Xintong Song wrote: +1 for supporting Hadoop 3. I'm not familiar with the shading efforts, thus no comment on dropping the flink-shaded-hadoop. Correct me if I'm wrong. Despite currently the default Hadoop version for compiling is 2.4.1 in Flink, I think this does not mean Flink should support only Hadoop 2.4+. So no matter which Hadoop version we use for compiling by default, we need to use reflection for the Hadoop features/APIs that are not supported in all versions anyway. There're already many such reflections in `YarnClusterDescriptor` and `YarnResourceManager`, and might be more in future. I'm wondering whether we should have a unified mechanism (an interface / abstract class or so) that handles all these kind of Hadoop API reflections at one place. Not necessarily in the scope to this discussion though. Thank you~ Xintong Song On Wed, Apr 22, 2020 at 8:32 PM Chesnay Schepler wrote: 1) Likely not, as this again introduces a hard-dependency on flink-shaded-hadoop. 2) Indeed; this will be something the user/cloud providers have to deal with now. 3) Yes. As a small note, we can still keep the hadoop-2 version of flink-shaded around for existing users. What I suggested was to just not release hadoop-3 versions. On 22/04/2020 14:19, Yang Wang wrote: Thanks Robert for starting this significant discussion. Since hadoop3 has been released for long time and many companies have already put it in production. No matter you are using flink-shaded-hadoop2 or not, currently Flink could already run in yarn3(not sure about HDFS). Since the yarn api is always backward compatible. The difference is we could not benefit from the new features because we are using hadoop-2.4 as compile dependency. So then we need to use reflector for new features(node label, tags, etc.). All in all, i am in in favour of dropping the flink-shaded-hadoop. Just have some questions. 1. Do we still support "-include-hadoop" profile? If yes, what we will get in the lib dir? 2. I am not sure whether dropping the flink-shaded-hadoop will take some class conflicts problems. If we use "export HADOOP_CLASSPATH=`hadoop classpath`" for the hadoop env setup, then many jars will be appended to the Flink client classpath. 3. The compile hadoop version is still 2.4.1. Right? Best, Yang Sivaprasanna 于2020年4月22日周三 下午4:18写道： I agree with Aljoscha. Otherwise I can see a lot of tickets getting created saying the application is not running on YARN. Cheers, Sivaprasanna On Wed, Apr 22, 2020 at 1:00 PM Aljoscha Krettek +1 to getting rid of flink-shaded-hadoop. But we need to document how people can now get a Flink dist that works with Hadoop. Currently, when you download the single shaded jar you immediately get support for submitting to YARN via bin/flink run. Aljoscha On 22.04.20 09:08, Till Rohrmann wrote: Hi Robert,
Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop
True, connectors built on Hadoop make this a bit more complex. That is also the reason why Hadoop is on the "parent first" patterns. Maybe this is a bit of a wild thought, but what would happen if we had a "first class" notion of a Hadoop Classloader in the system, and the user code classloader would explicitly fall back to that one whenever a class whose name starts with "org.apache.hadoop" is not found? We could also generalize this by associating plugin loaders with class name prefixes. Then it would try to load from the user code jar, and if the class was not found, load it from the hadoop classpath. On Thu, Apr 23, 2020 at 10:56 AM Chesnay Schepler wrote: > although, if you can load the HADOOP_CLASSPATH as a plugin, then you can > also load it in the user-code classloader. > > On 23/04/2020 10:50, Chesnay Schepler wrote: > > @Stephan I'm not aware of anyone having tried that; possibly since we > > have various connectors that require hadoop (hadoop-compat, hive, > > orc/parquet/hbase, hadoop inputformats). This would require connectors > > to be loaded as plugins (or having access to the plugin classloader) > > to be feasible. > > > > On 23/04/2020 09:59, Stephan Ewen wrote: > >> Hi all! > >> > >> +1 for the simplification of dropping hadoop-shaded > >> > >> > >> Have we ever investigated how much work it would be to load the > >> HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy > >> dependency > >> footprint would not spoil the main classpath. > >> > >>- HDFS might be very simple, because file systems are already > >> Plugin aware > >>- Yarn would need some extra work. In essence, we would need to > >> discover > >> executors also through plugins > >>- Kerberos is the other remaining bit. We would need to switch > >> security > >> modules to ServiceLoaders (which we should do anyways) and also pull > >> them > >> from plugins. > >> > >> Best, > >> Stephan > >> > >> > >> > >> On Thu, Apr 23, 2020 at 4:05 AM Xintong Song > >> wrote: > >> > >>> +1 for supporting Hadoop 3. > >>> > >>> I'm not familiar with the shading efforts, thus no comment on > >>> dropping the > >>> flink-shaded-hadoop. > >>> > >>> > >>> Correct me if I'm wrong. Despite currently the default Hadoop > >>> version for > >>> compiling is 2.4.1 in Flink, I think this does not mean Flink should > >>> support only Hadoop 2.4+. So no matter which Hadoop version we use for > >>> compiling by default, we need to use reflection for the Hadoop > >>> features/APIs that are not supported in all versions anyway. > >>> > >>> > >>> There're already many such reflections in `YarnClusterDescriptor` and > >>> `YarnResourceManager`, and might be more in future. I'm wondering > >>> whether > >>> we should have a unified mechanism (an interface / abstract class or > >>> so) > >>> that handles all these kind of Hadoop API reflections at one place. Not > >>> necessarily in the scope to this discussion though. > >>> > >>> > >>> Thank you~ > >>> > >>> Xintong Song > >>> > >>> > >>> > >>> On Wed, Apr 22, 2020 at 8:32 PM Chesnay Schepler > >>> wrote: > >>> > 1) Likely not, as this again introduces a hard-dependency on > flink-shaded-hadoop. > 2) Indeed; this will be something the user/cloud providers have to > deal > with now. > 3) Yes. > > As a small note, we can still keep the hadoop-2 version of > flink-shaded > around for existing users. > What I suggested was to just not release hadoop-3 versions. > > On 22/04/2020 14:19, Yang Wang wrote: > > Thanks Robert for starting this significant discussion. > > > > Since hadoop3 has been released for long time and many companies have > > already > > put it in production. No matter you are using flink-shaded-hadoop2 or > not, > > currently > > Flink could already run in yarn3(not sure about HDFS). Since the yarn > >>> api > > is always > > backward compatible. The difference is we could not benefit from the > >>> new > > features > > because we are using hadoop-2.4 as compile dependency. So then we > > need > >>> to > > use > > reflector for new features(node label, tags, etc.). > > > > All in all, i am in in favour of dropping the flink-shaded-hadoop. > > Just > > have some questions. > > 1. Do we still support "-include-hadoop" profile? If yes, what we > > will > get > > in the lib dir? > > 2. I am not sure whether dropping the flink-shaded-hadoop will take > >>> some > > class conflicts > > problems. If we use "export HADOOP_CLASSPATH=`hadoop classpath`" for > >>> the > > hadoop > > env setup, then many jars will be appended to the Flink client > >>> classpath. > > 3. The compile hadoop version is still 2.4.1. Right? > > > > > > Best, > > Yang > > > > > > Sivaprasanna 于2020年4月22日周三 > > 下午4:18写道： > > > >> I agree with Aljoscha. Otherwise I can see a lot of
Re: Reading a single input file in parallel?
Hi Niels, Thanks for start this discussion. Share some thought about your questions/proposals. > Judge whether splittable for each individual file Looks good to me. > BZIP2 support splittable Looks good to me. > the Flink implementation is controlled by the number of splits Can you check again? I think Flink is also affected by block size [1]. > Looking at how Hadoop does this I see that the FileInputFormat has a method isSplitable Now Flink do this in FileInputFormat, but also can do this like Hadoop I can see the split strategy in Hadoop orc and parquet are quite complex, I don't have a lot of in-depth research, but I think these implementations should be meaningful. [1] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L644 Best, Jingsong Lee On Thu, Apr 23, 2020 at 4:13 PM Niels Basjes wrote: > Hi, > > I wanted to spend some time on this idea I asked about a year ago. > > So I had a look at the actual code and found this where the file splits are > calculated > > > > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L601 > > What I realized is that apparently the current system for determining the > FileInputSplit s > 1) Lists the collection of input files. > 2) If a file is NOT splittable set a global flag to false. > 3) ONLY if all of them are splittable then will they be split. > > So (if I understand correctly) if I have an input directory with a 100GB > uncompressed text file and a 1 KB gzipped file then BOTH will NOT be split. > Why is that done this way? > > I also found that the determination if a file is NOT splittable completely > rests on the fact that the decompression that is to be used is to be > provided by an InflaterInputStreamFactory. > Which is funny because the BZIP2 as a compression IS splittable but > according to this implementation it isn't. > > I also noticed that the Hadoop split calculation is controlled by a split > size, the Flink implementation is controlled by the number of splits. > This seems logical to me as in Hadoop (MapReduce) the number of tasks > running is dynamic. In contrast with Flink where the number of parallel > tasks is a given setting. > > Looking at how Hadoop does this I see that the FileInputFormat has a > method isSplitable > (yes with typo, and with a terribly dangerous default value: > MAPREDUCE-2094) > which gives the answer on a per file basis. > Some file formats (like TextInputFormat) look at the codec that was found > to see if it implements the SplittableCompressionCodec interface to > determine if the file is splittable. > Other file formats (like Avro and Parquet) use something else to determine > this. > > Overall I currently think the way this has been implemented in Flink is not > very good. > > However looking at the gap to bridge to make it similar to what Hadoop has > seems like a huge step. > > I expect that many classes and interfaces will need to change dramatically > to make this happen. > > My main question to you guys: Do you think it is worth it? > > Niels Basjes > > > On Mon, Feb 19, 2018 at 10:50 AM Fabian Hueske wrote: > > > Hi Niels, > > > > Jörn is right, although offering different methods, Flink's InputFormat > is > > very similar to Hadoop's InputFormat interface. > > The InputFormat.createInputSplits() method generates splits that can be > > read in parallel. > > The FileInputFormat splits files by fixed boundaries (usually HDFS > > blocksize) and expects the InputFormat to find the right place to start > and > > end. > > For line-wise read files (TextInputFormat) or files with a record > delimiter > > (DelimiterInputFormat), the formats read the first record after they > found > > the first delimiter in their split and stop at the first delimiter after > > the split boundary. > > The BinaryInputFormat extends FileInputFormat but overrides the > > createInputSplits method. > > > > So, how exactly a file is read in parallel depends on the > > createInputSplits() method of the InputFormat. > > > > Hope this helps, > > Fabian > > > > > > 2018-02-18 13:36 GMT+01:00 Jörn Franke : > > > > > AFAIK Flink has a similar notion of splittable as Hadoop. Furthermore > you > > > can set for custom Fileibputformats the attribute unsplittable = true > if > > > your file format cannot be split > > > > > > > On 18. Feb 2018, at 13:28, Niels Basjes wrote: > > > > > > > > Hi, > > > > > > > > In Hadoop MapReduce there is the notion of "splittable" in the > > > > FileInputFormat. This has the effect that a single input file can be > > fed > > > > into multiple separate instances of the mapper that read the data. > > > > A lot has been documented (i.e. text is splittable per line, gzipped > > text > > > > is not splittable) and designed into the various file formats (like > > Avro > > > > and Parquet) to allow splittability. > > > > > > > > The goal is that reading and parsing files can
[DISCUSS] Add a custom label on MetricReporter like PrometheusPushGatewayReporter's groupingKey
Hi all We need add some custom labels on MetricReporter. We can add groupingKey to PrometheusPushGatewayReporter in version 1.10, but not in PrometheusReporter. Can we add several methods to the interface MetricReporter to support? Best Regards jinhai...@gmail.com
[jira] [Created] (FLINK-17346) Deduplicate process setup in docker tests
Chesnay Schepler created FLINK-17346: Summary: Deduplicate process setup in docker tests Key: FLINK-17346 URL: https://issues.apache.org/jira/browse/FLINK-17346 Project: Flink Issue Type: Improvement Components: Dockerfiles Reporter: Chesnay Schepler Assignee: Chesnay Schepler There's lots of duplication in how the docker processes is being setup, which would become even worse with FLINK-17164. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17345) Support registering and getting Python UDF in HiveCatalog
Wei Zhong created FLINK-17345: - Summary: Support registering and getting Python UDF in HiveCatalog Key: FLINK-17345 URL: https://issues.apache.org/jira/browse/FLINK-17345 Project: Flink Issue Type: Improvement Reporter: Wei Zhong Currently Python UDF can be registered via SQL DDL but can not be registered as catalog function because HiveCatalog is not support Python UDF yet. It is necessary to support registering and getting Python UDF in HiveCatalog. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17344) RecordWriterTest.testIdleTime possibly deadlocks on Travis
Andrey Zagrebin created FLINK-17344: --- Summary: RecordWriterTest.testIdleTime possibly deadlocks on Travis Key: FLINK-17344 URL: https://issues.apache.org/jira/browse/FLINK-17344 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.11.0 Reporter: Andrey Zagrebin Fix For: 1.11.0 https://travis-ci.org/github/apache/flink/jobs/678193214 The test was introduced in [FLINK-16864|https://jira.apache.org/jira/browse/FLINK-16864]. It may be an instability as it passed 2 times (core and core-scala) and failed in core-hadoop: https://travis-ci.org/github/apache/flink/builds/678193199 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [VOTE] Release 1.9.3, release candidate #1
+1 (binding) - for license documentation, I checked the diff between 1.9.2 and 1.9.3: https://github.com/apache/flink/compare/release-1.9.2...release-1.9.3-rc1 - Kafka dependency changed from 0.10.2.1 to 0.10.2.2 --> License doc was updated in PR https://github.com/apache/flink/pull/11617/files - one test dependency change that is not relevant - checked the maven staging repo files - 1.9.3 set correctly, java quickstart looks fine - checked license files of some shaded jars On Wed, Apr 22, 2020 at 3:22 PM Leonard Xu wrote: > +1 (non-binding) > > - checked/verified signatures and hashes > - checked the release note > - checked that there are no missing artifacts in staging area > - built from source sing scala 2.11 and using Scala 2.12 succeeded > - ran a couple of end-to-end tests locally and succeeded > - went through all commits checked in between 1.9.3 and 1.9.2, make sure > all issues set the "fixVersion" property > - started a cluster, WebUI was accessible, submitted a wordcount job and > ran succeeded, no suspicious log output > - the web PR looks good > > Best, > Leonard Xu > > > 在 2020年4月22日，17:58，Dian Fu 写道： > > > > +1 (non-binding) > > > > - verified the checksum and signature > > - checked the release note > > - checked that there are no new dependencies introduced since 1.9.2 which > > may affect the license (only bump kafka from 0.10.2.1 to 0.10.2.2 which > > doesn't affect the license) > > - checked that there are no missing artifacts in staging area > > - checked that the pyflink package could be pip installed > > > > Regards, > > Dian > > > > On Wed, Apr 22, 2020 at 3:35 PM Fabian Paul < > fabianp...@data-artisans.com> > > wrote: > > > >> +1 (non-binding) > >> > >> - Verified signature > >> - Built from source (Java8) > >> - Run custom jobs on Kubernetes > >> > >> Regards, > >> Fabian > >> > >>> On 18. Apr 2020, at 04:37, Dian Fu wrote: > >>> > >>> Hi everyone, > >>> > >>> Please review and vote on the release candidate #1 for the version > 1.9.3, > >>> as follows: > >>> [ ] +1, Approve the release > >>> [ ] -1, Do not approve the release (please provide specific comments) > >>> > >>> > >>> The complete staging area is available for your review, which includes: > >>> * JIRA release notes [1], > >>> * the official Apache source release and binary convenience releases to > >> be > >>> deployed to dist.apache.org [2], which are signed with the key with > >>> fingerprint 6B6291A8502BA8F0913AE04DDEB95B05BF075300 [3], > >>> * all artifacts to be deployed to the Maven Central Repository [4], > >>> * source code tag "release-1.9.3-rc1" [5], > >>> * website pull request listing the new release and adding announcement > >> blog > >>> post [6]. > >>> > >>> The vote will be open for at least 72 hours. It is adopted by majority > >> approval, with at least 3 PMC affirmative votes. > >>> > >>> Thanks, > >>> Dian > >>> > >>> [1] > >> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522=12346867 > >>> [2] https://dist.apache.org/repos/dist/dev/flink/flink-1.9.3-rc1/ > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS > >>> [4] > >> https://repository.apache.org/content/repositories/orgapacheflink-1353/ > >>> [5] > >> > https://github.com/apache/flink/commit/6d23b2c38c7a8fd8855063238c744923e1985a63 > >>> [6] https://github.com/apache/flink-web/pull/329 > >> > >> > >
Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop
although, if you can load the HADOOP_CLASSPATH as a plugin, then you can also load it in the user-code classloader. On 23/04/2020 10:50, Chesnay Schepler wrote: @Stephan I'm not aware of anyone having tried that; possibly since we have various connectors that require hadoop (hadoop-compat, hive, orc/parquet/hbase, hadoop inputformats). This would require connectors to be loaded as plugins (or having access to the plugin classloader) to be feasible. On 23/04/2020 09:59, Stephan Ewen wrote: Hi all! +1 for the simplification of dropping hadoop-shaded Have we ever investigated how much work it would be to load the HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy dependency footprint would not spoil the main classpath. - HDFS might be very simple, because file systems are already Plugin aware - Yarn would need some extra work. In essence, we would need to discover executors also through plugins - Kerberos is the other remaining bit. We would need to switch security modules to ServiceLoaders (which we should do anyways) and also pull them from plugins. Best, Stephan On Thu, Apr 23, 2020 at 4:05 AM Xintong Song wrote: +1 for supporting Hadoop 3. I'm not familiar with the shading efforts, thus no comment on dropping the flink-shaded-hadoop. Correct me if I'm wrong. Despite currently the default Hadoop version for compiling is 2.4.1 in Flink, I think this does not mean Flink should support only Hadoop 2.4+. So no matter which Hadoop version we use for compiling by default, we need to use reflection for the Hadoop features/APIs that are not supported in all versions anyway. There're already many such reflections in `YarnClusterDescriptor` and `YarnResourceManager`, and might be more in future. I'm wondering whether we should have a unified mechanism (an interface / abstract class or so) that handles all these kind of Hadoop API reflections at one place. Not necessarily in the scope to this discussion though. Thank you~ Xintong Song On Wed, Apr 22, 2020 at 8:32 PM Chesnay Schepler wrote: 1) Likely not, as this again introduces a hard-dependency on flink-shaded-hadoop. 2) Indeed; this will be something the user/cloud providers have to deal with now. 3) Yes. As a small note, we can still keep the hadoop-2 version of flink-shaded around for existing users. What I suggested was to just not release hadoop-3 versions. On 22/04/2020 14:19, Yang Wang wrote: Thanks Robert for starting this significant discussion. Since hadoop3 has been released for long time and many companies have already put it in production. No matter you are using flink-shaded-hadoop2 or not, currently Flink could already run in yarn3(not sure about HDFS). Since the yarn api is always backward compatible. The difference is we could not benefit from the new features because we are using hadoop-2.4 as compile dependency. So then we need to use reflector for new features(node label, tags, etc.). All in all, i am in in favour of dropping the flink-shaded-hadoop. Just have some questions. 1. Do we still support "-include-hadoop" profile? If yes, what we will get in the lib dir? 2. I am not sure whether dropping the flink-shaded-hadoop will take some class conflicts problems. If we use "export HADOOP_CLASSPATH=`hadoop classpath`" for the hadoop env setup, then many jars will be appended to the Flink client classpath. 3. The compile hadoop version is still 2.4.1. Right? Best, Yang Sivaprasanna 于2020年4月22日周三 下午4:18写道： I agree with Aljoscha. Otherwise I can see a lot of tickets getting created saying the application is not running on YARN. Cheers, Sivaprasanna On Wed, Apr 22, 2020 at 1:00 PM Aljoscha Krettek wrote: +1 to getting rid of flink-shaded-hadoop. But we need to document how people can now get a Flink dist that works with Hadoop. Currently, when you download the single shaded jar you immediately get support for submitting to YARN via bin/flink run. Aljoscha On 22.04.20 09:08, Till Rohrmann wrote: Hi Robert, I think it would be a helpful simplification of Flink's build setup if we can get rid of flink-shaded-hadoop. Moreover relying only on the vanilla Hadoop dependencies for the modules which interact with Hadoop/Yarn sounds like a good idea to me. Adding support for Hadoop 3 would also be nice. I'm not sure, though, how Hadoop's API's have changed between 2 and 3. It might be necessary to introduce some bridges in order to make it work. Cheers, Till On Tue, Apr 21, 2020 at 4:37 PM Robert Metzger wrote: Hi all, for the upcoming 1.11 release, I started looking into adding support for Hadoop 3[1] for Flink. I have explored a little bit already into adding a shaded hadoop 3 into “flink-shaded”, and some mechanisms for switching between Hadoop 2 and 3 dependencies in the Flink build. However, Chesnay made me aware that we could also go a different route: We let Flink depend on vanilla Hadoop
Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop
@Stephan I'm not aware of anyone having tried that; possibly since we have various connectors that require hadoop (hadoop-compat, hive, orc/parquet/hbase, hadoop inputformats). This would require connectors to be loaded as plugins (or having access to the plugin classloader) to be feasible. On 23/04/2020 09:59, Stephan Ewen wrote: Hi all! +1 for the simplification of dropping hadoop-shaded Have we ever investigated how much work it would be to load the HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy dependency footprint would not spoil the main classpath. - HDFS might be very simple, because file systems are already Plugin aware - Yarn would need some extra work. In essence, we would need to discover executors also through plugins - Kerberos is the other remaining bit. We would need to switch security modules to ServiceLoaders (which we should do anyways) and also pull them from plugins. Best, Stephan On Thu, Apr 23, 2020 at 4:05 AM Xintong Song wrote: +1 for supporting Hadoop 3. I'm not familiar with the shading efforts, thus no comment on dropping the flink-shaded-hadoop. Correct me if I'm wrong. Despite currently the default Hadoop version for compiling is 2.4.1 in Flink, I think this does not mean Flink should support only Hadoop 2.4+. So no matter which Hadoop version we use for compiling by default, we need to use reflection for the Hadoop features/APIs that are not supported in all versions anyway. There're already many such reflections in `YarnClusterDescriptor` and `YarnResourceManager`, and might be more in future. I'm wondering whether we should have a unified mechanism (an interface / abstract class or so) that handles all these kind of Hadoop API reflections at one place. Not necessarily in the scope to this discussion though. Thank you~ Xintong Song On Wed, Apr 22, 2020 at 8:32 PM Chesnay Schepler wrote: 1) Likely not, as this again introduces a hard-dependency on flink-shaded-hadoop. 2) Indeed; this will be something the user/cloud providers have to deal with now. 3) Yes. As a small note, we can still keep the hadoop-2 version of flink-shaded around for existing users. What I suggested was to just not release hadoop-3 versions. On 22/04/2020 14:19, Yang Wang wrote: Thanks Robert for starting this significant discussion. Since hadoop3 has been released for long time and many companies have already put it in production. No matter you are using flink-shaded-hadoop2 or not, currently Flink could already run in yarn3(not sure about HDFS). Since the yarn api is always backward compatible. The difference is we could not benefit from the new features because we are using hadoop-2.4 as compile dependency. So then we need to use reflector for new features(node label, tags, etc.). All in all, i am in in favour of dropping the flink-shaded-hadoop. Just have some questions. 1. Do we still support "-include-hadoop" profile? If yes, what we will get in the lib dir? 2. I am not sure whether dropping the flink-shaded-hadoop will take some class conflicts problems. If we use "export HADOOP_CLASSPATH=`hadoop classpath`" for the hadoop env setup, then many jars will be appended to the Flink client classpath. 3. The compile hadoop version is still 2.4.1. Right? Best, Yang Sivaprasanna 于2020年4月22日周三 下午4:18写道： I agree with Aljoscha. Otherwise I can see a lot of tickets getting created saying the application is not running on YARN. Cheers, Sivaprasanna On Wed, Apr 22, 2020 at 1:00 PM Aljoscha Krettek +1 to getting rid of flink-shaded-hadoop. But we need to document how people can now get a Flink dist that works with Hadoop. Currently, when you download the single shaded jar you immediately get support for submitting to YARN via bin/flink run. Aljoscha On 22.04.20 09:08, Till Rohrmann wrote: Hi Robert, I think it would be a helpful simplification of Flink's build setup if we can get rid of flink-shaded-hadoop. Moreover relying only on the vanilla Hadoop dependencies for the modules which interact with Hadoop/Yarn sounds like a good idea to me. Adding support for Hadoop 3 would also be nice. I'm not sure, though, how Hadoop's API's have changed between 2 and 3. It might be necessary to introduce some bridges in order to make it work. Cheers, Till On Tue, Apr 21, 2020 at 4:37 PM Robert Metzger wrote: Hi all, for the upcoming 1.11 release, I started looking into adding support for Hadoop 3[1] for Flink. I have explored a little bit already into adding a shaded hadoop 3 into “flink-shaded”, and some mechanisms for switching between Hadoop 2 and 3 dependencies in the Flink build. However, Chesnay made me aware that we could also go a different route: We let Flink depend on vanilla Hadoop dependencies and stop providing shaded fat jars for Hadoop through “flink-shaded”. Why? - Maintaining properly shaded Hadoop fat jars is a lot of work (we have insufficient test coverage for all kinds of
[jira] [Created] (FLINK-17343) Add doc for retraction behavior for all operators
Benchao Li created FLINK-17343: -- Summary: Add doc for retraction behavior for all operators Key: FLINK-17343 URL: https://issues.apache.org/jira/browse/FLINK-17343 Project: Flink Issue Type: Improvement Components: Documentation, Table SQL / Planner Reporter: Benchao Li Currently the documentation does not explain the retraction behavior explicitly, which makes it confused for users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17342) Schedule savepoint if max-inflight-checkpoints limit is reached isntead of forcing (in UC mode)
Roman Khachatryan created FLINK-17342: - Summary: Schedule savepoint if max-inflight-checkpoints limit is reached isntead of forcing (in UC mode) Key: FLINK-17342 URL: https://issues.apache.org/jira/browse/FLINK-17342 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing Affects Versions: 1.11.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17341) freeSlot in TaskExecutor.closeJobManagerConnection cause ConcurrentModificationException
huweihua created FLINK-17341:
Summary: freeSlot in TaskExecutor.closeJobManagerConnection cause
ConcurrentModificationException
Key: FLINK-17341
URL: https://issues.apache.org/jira/browse/FLINK-17341
Project: Flink
Issue Type: Bug
Reporter: huweihua
TaskExecutor may freeSlot when closeJobManagerConnection. freeSlot will modify
the TaskSlotTable.slotsPerJob. this modify will cause
ConcurrentModificationException.
{code:java}
Iterator activeSlots = taskSlotTable.getActiveSlots(jobId);
final FlinkException freeingCause = new FlinkException("Slot could not be
marked inactive.");
while (activeSlots.hasNext()) {
AllocationID activeSlot = activeSlots.next();
try {
if (!taskSlotTable.markSlotInactive(activeSlot,
taskManagerConfiguration.getTimeout())) {
freeSlotInternal(activeSlot, freeingCause);
}
} catch (SlotNotFoundException e) {
log.debug("Could not mark the slot {} inactive.", jobId, e);
}
}
{code}
error log：
{code:java}
2020-04-21 23:37:11,363 ERROR org.apache.flink.runtime.rpc.akka.AkkaRpcActor
- Caught exception while executing runnable in main thread.
java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
at java.util.HashMap$KeyIterator.next(HashMap.java:1461)
at
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable$TaskSlotIterator.hasNext(TaskSlotTable.java:698)
at
org.apache.flink.runtime.taskexecutor.slot.TaskSlotTable$AllocationIDIterator.hasNext(TaskSlotTable.java:652)
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.closeJobManagerConnection(TaskExecutor.java:1314)
at
org.apache.flink.runtime.taskexecutor.TaskExecutor.access$1300(TaskExecutor.java:149)
at
org.apache.flink.runtime.taskexecutor.TaskExecutor$JobLeaderListenerImpl.lambda$jobManagerLostLeadership$1(TaskExecutor.java:1726)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17340) Update docs which related to default planner changes
Kurt Young created FLINK-17340: -- Summary: Update docs which related to default planner changes Key: FLINK-17340 URL: https://issues.apache.org/jira/browse/FLINK-17340 Project: Flink Issue Type: Sub-task Components: Documentation Reporter: Kurt Young Assignee: Kurt Young Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17339) Change default planner to blink and update test cases in both planners
Kurt Young created FLINK-17339: -- Summary: Change default planner to blink and update test cases in both planners Key: FLINK-17339 URL: https://issues.apache.org/jira/browse/FLINK-17339 Project: Flink Issue Type: Sub-task Components: Table SQL / Legacy Planner, Table SQL / Planner Reporter: Kurt Young Assignee: Kurt Young Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Reading a single input file in parallel?
Hi, I wanted to spend some time on this idea I asked about a year ago. So I had a look at the actual code and found this where the file splits are calculated https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/io/FileInputFormat.java#L601 What I realized is that apparently the current system for determining the FileInputSplit s 1) Lists the collection of input files. 2) If a file is NOT splittable set a global flag to false. 3) ONLY if all of them are splittable then will they be split. So (if I understand correctly) if I have an input directory with a 100GB uncompressed text file and a 1 KB gzipped file then BOTH will NOT be split. Why is that done this way? I also found that the determination if a file is NOT splittable completely rests on the fact that the decompression that is to be used is to be provided by an InflaterInputStreamFactory. Which is funny because the BZIP2 as a compression IS splittable but according to this implementation it isn't. I also noticed that the Hadoop split calculation is controlled by a split size, the Flink implementation is controlled by the number of splits. This seems logical to me as in Hadoop (MapReduce) the number of tasks running is dynamic. In contrast with Flink where the number of parallel tasks is a given setting. Looking at how Hadoop does this I see that the FileInputFormat has a method isSplitable (yes with typo, and with a terribly dangerous default value: MAPREDUCE-2094) which gives the answer on a per file basis. Some file formats (like TextInputFormat) look at the codec that was found to see if it implements the SplittableCompressionCodec interface to determine if the file is splittable. Other file formats (like Avro and Parquet) use something else to determine this. Overall I currently think the way this has been implemented in Flink is not very good. However looking at the gap to bridge to make it similar to what Hadoop has seems like a huge step. I expect that many classes and interfaces will need to change dramatically to make this happen. My main question to you guys: Do you think it is worth it? Niels Basjes On Mon, Feb 19, 2018 at 10:50 AM Fabian Hueske wrote: > Hi Niels, > > Jörn is right, although offering different methods, Flink's InputFormat is > very similar to Hadoop's InputFormat interface. > The InputFormat.createInputSplits() method generates splits that can be > read in parallel. > The FileInputFormat splits files by fixed boundaries (usually HDFS > blocksize) and expects the InputFormat to find the right place to start and > end. > For line-wise read files (TextInputFormat) or files with a record delimiter > (DelimiterInputFormat), the formats read the first record after they found > the first delimiter in their split and stop at the first delimiter after > the split boundary. > The BinaryInputFormat extends FileInputFormat but overrides the > createInputSplits method. > > So, how exactly a file is read in parallel depends on the > createInputSplits() method of the InputFormat. > > Hope this helps, > Fabian > > > 2018-02-18 13:36 GMT+01:00 Jörn Franke : > > > AFAIK Flink has a similar notion of splittable as Hadoop. Furthermore you > > can set for custom Fileibputformats the attribute unsplittable = true if > > your file format cannot be split > > > > > On 18. Feb 2018, at 13:28, Niels Basjes wrote: > > > > > > Hi, > > > > > > In Hadoop MapReduce there is the notion of "splittable" in the > > > FileInputFormat. This has the effect that a single input file can be > fed > > > into multiple separate instances of the mapper that read the data. > > > A lot has been documented (i.e. text is splittable per line, gzipped > text > > > is not splittable) and designed into the various file formats (like > Avro > > > and Parquet) to allow splittability. > > > > > > The goal is that reading and parsing files can be done by multiple > > > cpus/systems in parallel. > > > > > > How is this handled in Flink? > > > Can Flink read a single file in parallel? > > > How does Flink administrate/handle the possibilities regarding the > > various > > > file formats? > > > > > > > > > The reason I ask is because I want to see if I can port this (now > Hadoop > > > specific) hobby project of mine to work with Flink: > > > https://github.com/nielsbasjes/splittablegzip > > > > > > Thanks. > > > > > > -- > > > Best regards / Met vriendelijke groeten, > > > > > > Niels Basjes > > > -- Best regards / Met vriendelijke groeten, Niels Basjes
[DISCUSS] Exact feature freeze date
Hi all! I want to bring up a discussion about when we want to do the feature freeze for 1.11. When kicking off the release cycle, we tentatively set the date to end of April, which would be in one week. I can say from the features I am involved with (FLIP-27, FLIP-115, reviewing some state backend improvements, etc.) that it would be helpful to have two additional weeks. When looking at various other feature threads, my feeling is that there are more contributors and committers that could use a few more days. The last two months were quite exceptional in and we did lose a bit of development speed here and there. How do you think about making *May 15th* the feature freeze? Best, Stephan
Re: [DISCUSS] Adding support for Hadoop 3 and removing flink-shaded-hadoop
Hi all! +1 for the simplification of dropping hadoop-shaded Have we ever investigated how much work it would be to load the HADOOP_CLASSPATH through the plugin loader? Then Hadoop's crazy dependency footprint would not spoil the main classpath. - HDFS might be very simple, because file systems are already Plugin aware - Yarn would need some extra work. In essence, we would need to discover executors also through plugins - Kerberos is the other remaining bit. We would need to switch security modules to ServiceLoaders (which we should do anyways) and also pull them from plugins. Best, Stephan On Thu, Apr 23, 2020 at 4:05 AM Xintong Song wrote: > +1 for supporting Hadoop 3. > > I'm not familiar with the shading efforts, thus no comment on dropping the > flink-shaded-hadoop. > > > Correct me if I'm wrong. Despite currently the default Hadoop version for > compiling is 2.4.1 in Flink, I think this does not mean Flink should > support only Hadoop 2.4+. So no matter which Hadoop version we use for > compiling by default, we need to use reflection for the Hadoop > features/APIs that are not supported in all versions anyway. > > > There're already many such reflections in `YarnClusterDescriptor` and > `YarnResourceManager`, and might be more in future. I'm wondering whether > we should have a unified mechanism (an interface / abstract class or so) > that handles all these kind of Hadoop API reflections at one place. Not > necessarily in the scope to this discussion though. > > > Thank you~ > > Xintong Song > > > > On Wed, Apr 22, 2020 at 8:32 PM Chesnay Schepler > wrote: > > > 1) Likely not, as this again introduces a hard-dependency on > > flink-shaded-hadoop. > > 2) Indeed; this will be something the user/cloud providers have to deal > > with now. > > 3) Yes. > > > > As a small note, we can still keep the hadoop-2 version of flink-shaded > > around for existing users. > > What I suggested was to just not release hadoop-3 versions. > > > > On 22/04/2020 14:19, Yang Wang wrote: > > > Thanks Robert for starting this significant discussion. > > > > > > Since hadoop3 has been released for long time and many companies have > > > already > > > put it in production. No matter you are using flink-shaded-hadoop2 or > > not, > > > currently > > > Flink could already run in yarn3(not sure about HDFS). Since the yarn > api > > > is always > > > backward compatible. The difference is we could not benefit from the > new > > > features > > > because we are using hadoop-2.4 as compile dependency. So then we need > to > > > use > > > reflector for new features(node label, tags, etc.). > > > > > > All in all, i am in in favour of dropping the flink-shaded-hadoop. Just > > > have some questions. > > > 1. Do we still support "-include-hadoop" profile? If yes, what we will > > get > > > in the lib dir? > > > 2. I am not sure whether dropping the flink-shaded-hadoop will take > some > > > class conflicts > > > problems. If we use "export HADOOP_CLASSPATH=`hadoop classpath`" for > the > > > hadoop > > > env setup, then many jars will be appended to the Flink client > classpath. > > > 3. The compile hadoop version is still 2.4.1. Right? > > > > > > > > > Best, > > > Yang > > > > > > > > > Sivaprasanna 于2020年4月22日周三 下午4:18写道： > > > > > >> I agree with Aljoscha. Otherwise I can see a lot of tickets getting > > created > > >> saying the application is not running on YARN. > > >> > > >> Cheers, > > >> Sivaprasanna > > >> > > >> On Wed, Apr 22, 2020 at 1:00 PM Aljoscha Krettek > > > >> wrote: > > >> > > >>> +1 to getting rid of flink-shaded-hadoop. But we need to document how > > >>> people can now get a Flink dist that works with Hadoop. Currently, > when > > >>> you download the single shaded jar you immediately get support for > > >>> submitting to YARN via bin/flink run. > > >>> > > >>> Aljoscha > > >>> > > >>> > > >>> On 22.04.20 09:08, Till Rohrmann wrote: > > Hi Robert, > > > > I think it would be a helpful simplification of Flink's build setup > if > > >> we > > can get rid of flink-shaded-hadoop. Moreover relying only on the > > >> vanilla > > Hadoop dependencies for the modules which interact with Hadoop/Yarn > > >>> sounds > > like a good idea to me. > > > > Adding support for Hadoop 3 would also be nice. I'm not sure, > though, > > >> how > > Hadoop's API's have changed between 2 and 3. It might be necessary > to > > introduce some bridges in order to make it work. > > > > Cheers, > > Till > > > > On Tue, Apr 21, 2020 at 4:37 PM Robert Metzger > > > >>> wrote: > > > Hi all, > > > > > > for the upcoming 1.11 release, I started looking into adding > support > > >> for > > > Hadoop 3[1] for Flink. I have explored a little bit already into > > >> adding > > >>> a > > > shaded hadoop 3 into “flink-shaded”, and some mechanisms for > > switching > > > between Hadoop 2 and 3 dependencies in the Flink build. > > > > >
Re: Flink dev blog
Hey all,
I wanted to remind everybody that we've started this nice initiative with
the "Flink Engine Room" / dev blog.
It would be great if we could start filling it with more content.
In my opinion, the blog posts can be fairly unpolished, as it's more
important to share the knowledge than to make a good impression :)
Looking forward to more contributions!
Best,
Robert
On Mon, Mar 9, 2020 at 11:48 AM Yu Li wrote:
> Hurray! Thanks Arvid and Robert! Will ask the team here to prepare some
> RocksDB backend related posts.
>
> Best Regards,
> Yu
>
>
> On Mon, 9 Mar 2020 at 17:51, tison wrote:
>
> > Thanks for your reply Robert. That sounds great.
> >
> > Best,
> > tison.
> >
> >
> > Robert Metzger 于2020年3月9日周一 下午5:46写道：
> >
> > > Hey Tison,
> > >
> > > only people we have manually given write permission to the Wiki are
> able
> > to
> > > add a blog post. If somebody is posting something we don't want there,
> we
> > > can just revoke that person's permission to write on the blog.
> > >
> > > It is definitely something we should keep an eye on, but I don't think
> we
> > > need to get active beforehand.
> > >
> > > Best,
> > > Robert
> > >
> > > On Mon, Mar 9, 2020 at 10:02 AM tison wrote:
> > >
> > > > Thank Arvid & Robert for the effort. Amazing!
> > > >
> > > > I'm curious the procedure a blog get posted. Follow the discussion so
> > far
> > > > it seems any contributor can post his blog under the directory as he
> > > > wishes,
> > > > is it the case?
> > > >
> > > > Best,
> > > > tison.
> > > >
> > > >
> > > > Arvid Heise 于2020年3月9日周一 下午4:54写道：
> > > >
> > > > > Dear all,
> > > > >
> > > > > Robert added a link to the engine room blog on the Apache wiki [1].
> > > It's
> > > > > currently empty except for one test post. To add a new post, you
> just
> > > > > create a new blog post from the top menu ("engine room" is just a
> > fancy
> > > > > link to the default blog).
> > > > >
> > > > > Robert would like to write about the migration to Azure Pipelines
> > and I
> > > > > would write a post about plugins in the next days. After having
> these
> > > > > ready, I'd start a shorter round of discussion about style
> guidelines
> > > > (more
> > > > > meant as a guidance than restriction). I'd explicitly ask for
> > feedback
> > > on
> > > > > user ML as well at this point. Of course, we are happy about any
> > early
> > > > > contribution of more blog posts.
> > > > >
> > > > > If we have more posts at a later point in time, we would discuss
> blog
> > > > > structure and find out easy ways/tools to migrate revised articles
> to
> > > the
> > > > > official Flink blog.
> > > > >
> > > > > [1]
> > > https://cwiki.apache.org/confluence/display/FLINK/Apache+Flink+Home
> > > > >
> > > > > On Sat, Mar 7, 2020 at 7:51 PM Rong Rong
> > wrote:
> > > > >
> > > > > > +1 on Arvid's proposal. looking forward to the "Engine Room" blog
> > > > series.
> > > > > > :-D
> > > > > >
> > > > > > --
> > > > > > Rong
> > > > > >
> > > > > > On Sat, Mar 7, 2020 at 12:08 AM Yu Li wrote:
> > > > > >
> > > > > > > +1 to Arvid's proposal, thanks for the efforts!
> > > > > > >
> > > > > > > Best Regards,
> > > > > > > Yu
> > > > > > >
> > > > > > >
> > > > > > > On Thu, 5 Mar 2020 at 23:04, Zhijiang <
> > wangzhijiang...@aliyun.com
> > > > > > .invalid>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks for this proposal Arvid!
> > > > > > > > +1 and looking forward to the wiki structure and more
> following
> > > > > blogs.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Zhijiang
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > --
> > > > > > > > From:Dian Fu
> > > > > > > > Send Time:2020 Mar. 5 (Thu.) 19:08
> > > > > > > > To:dev
> > > > > > > > Subject:Re: Flink dev blog
> > > > > > > >
> > > > > > > > +1 to Arvid's proposal
> > > > > > > >
> > > > > > > > > 在 2020年3月5日，下午6:49，Jark Wu 写道：
> > > > > > > > >
> > > > > > > > > +1 to Arvid's proposal.
> > > > > > > > >
> > > > > > > > > On Thu, 5 Mar 2020 at 18:13, Robert Metzger <
> > > rmetz...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> +1 to Arvid's proposal.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> On Thu, Mar 5, 2020 at 4:14 AM Xingbo Huang <
> > > hxbks...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Thanks a for this proposal.
> > > > > > > > >>>
> > > > > > > > >>> As a new contributor to Flink, it would be very helpful
> to
> > > have
> > > > > > such
> > > > > > > > >> blogs
> > > > > > > > >>> for us to understand the future of Flink and get involved
> > > > > > > > >>>
> > > > > > > > >>> BTW, I have a question whether the dev blog needs a
> > template
> > > > like
> > > > > > > FLIP.
> > > > > > > > >>>
> > > > > > > > >>> Of course, There is no doubt that dev blogs do not need
> to
> > be
> > > > as
> > > > > > > formal
> > > > > > > > >> as
> > > > > > > > >>> FLIP,
Re: [DISCUSS] Switch to Azure Pipelines as the primary CI tool / switch off Travis
FYI: I have moved the Flink PR and master builds from my personal Azure account to a PMC controlled account: https://dev.azure.com/apache-flink/apache-flink/_build On Fri, Apr 17, 2020 at 8:28 PM Robert Metzger wrote: > Thanks a lot for bringing up this topic again. > The reason why I was hesitant to decommission Travis was that we were > still facing some issues with the Azure infrastructure that I want to > resolve, so that we have a strong test coverage. > > In the last few weeks, we had the following issues: > - unstable e2e tests (we are running the e2e tests much more frequently, > thus we see more failures (and discover actual bugs!)) > - network issues (mostly around downloading maven artifacts. This is > solved at the cost of slower builds. I'm preparing a fix to have stable & > fast maven downloads) > - the private builds were never really stable (this is work in progress. > the situation is definitely better than the private Travis builds) > - I haven't followed the overall master stability closely before February, > but I have the feeling that April so far was a pretty unstable month on > master. Piotr is regularly reverting commits that somehow broke master. The > problem with unstable master is that is causes a "CI fatigue", were people > assume that failing builds are not worth investigating anymore, leading to > more instability. This is not a problem of the CI infrastructure itself, > but it makes me less confident switching systems :) > > > Unless something unexpected happens, I'm proposing to disable pull request > processing on Travis next week. > > > > On Fri, Apr 17, 2020 at 10:05 AM Gary Yao wrote: > >> I am in favour of decommissioning Travis. >> >> Moreover, I wanted to use this thread to raise another issue with Travis >> that I >> have discovered recently; many of the builds running in my private Travis >> account are timing out in the compilation stage (i.e., compilation takes >> more >> than 50 minutes). This means that I am not able to reliably run a full >> build on >> a CI server without creating a pull request. If other developers also >> experience >> this issue, it would speak for putting more effort into making Azure >> Pipelines >> the project-wide default. >> >> Best, >> Gary >> >> On Thu, Mar 26, 2020 at 12:26 PM Yu Li wrote: >> >> > Thanks for the clarification Robert. >> > >> > Since the first step plan is to replace the travis PR runs, I checked >> all >> > PR builds from 2020-01-01 (PR#10735-11526) [1], and below is the result: >> > >> > * Travis FAILURE: 298 >> > * Travis SUCCESS: 649 (68.5%) >> > * Azure FAILURE: 420 >> > * Azure SUCCESS: 571 (57.6%) >> > >> > Since the patch for each run is equivalent for Travis and Azure, there >> > seems to be slightly higher failure rate (~10%) when running in Azure. >> > >> > However, with the just-merged fix for uploading logs (FLINK-16480), I >> > believe the success rate of Azure could compete with Travis now >> (uploading >> > files contribute to 20% of the failures according to the report [2]). >> > >> > So I'm +1 to disable travis runs according to the numbers. >> > >> > Best Regards, >> > Yu >> > >> > [1] >> > >> https://github.com/apache/flink/pulls?q=is%3Apr+created%3A%3E%3D2020-01-01 >> > [2] >> > >> > >> https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=4 >> > >> > On Thu, 26 Mar 2020 at 03:28, Robert Metzger >> wrote: >> > >> > > Thank you for your responses. >> > > >> > > @Yu Li: In the current master, the log upload always fails, if the e2e >> > job >> > > failed. I just merged a PR that fixes this issue [1]. The problem was >> not >> > > really the network stability, rather a problem with the interaction of >> > the >> > > jobs in the pipeline (the e2e job did not set the right variables for >> the >> > > log upload) >> > > Secondly, you are looking at the report of the "flink-ci.flink" >> pipeline, >> > > where pull requests are build. Naturally, pull request builds fail all >> > the >> > > time, because the PRs are not yet perfect. >> > > >> > > "flink-ci.flink-master" is the right pipeline to look at: >> > > >> > > >> > >> https://dev.azure.com/rmetzger/Flink/_pipeline/analytics/stageawareoutcome?definitionId=8=build >> > > We have a fairly high number of failures there, because we currently >> have >> > > some issues downloading the maven artifacts [2]. I'm working already >> with >> > > Chesnay on merging a fix for that. >> > > >> > > >> > > [1] >> > > >> > > >> > >> https://github.com/apache/flink/commit/1c86b8b9dd05615a3b2600984db738a9bf388259 >> > > [2]https://issues.apache.org/jira/browse/FLINK-16720 >> > > >> > > >> > > >> > > On Wed, Mar 25, 2020 at 4:48 PM Chesnay Schepler >> > > wrote: >> > > >> > > > The easiest way to disable travis for pushes is to remove all builds >> > > > from the .travis.yml with a push/pr condition. >> > > > >> > > > On 25/03/2020 15:03, Robert Metzger wrote: >> > > > > Thank you for the feedback so far. >> > > > > >> > > > >
[jira] [Created] (FLINK-17338) LocalExecutorITCase.testBatchQueryCancel test timeout
Robert Metzger created FLINK-17338:
--
Summary: LocalExecutorITCase.testBatchQueryCancel test timeout
Key: FLINK-17338
URL: https://issues.apache.org/jira/browse/FLINK-17338
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Reporter: Robert Metzger
CI https://travis-ci.org/github/apache/flink/jobs/678185941
{code}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.flink.table.client.gateway.local.LocalExecutorITCase
[ERROR] Tests run: 70, Failures: 0, Errors: 1, Skipped: 5, Time elapsed:
226.359 s <<< FAILURE! - in
org.apache.flink.table.client.gateway.local.LocalExecutorITCase
[ERROR] testBatchQueryCancel[Planner:
old](org.apache.flink.table.client.gateway.local.LocalExecutorITCase) Time
elapsed: 30.009 s <<< ERROR!
org.junit.runners.model.TestTimedOutException: test timed out after 3
milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.flink.table.client.gateway.local.LocalExecutorITCase.testBatchQueryCancel(LocalExecutorITCase.java:733)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
[INFO]
[INFO] Results:
[INFO]
[ERROR] Errors:
[ERROR] LocalExecutorITCase.testBatchQueryCancel:733 » TestTimedOut test
timed out aft...
[INFO]
[ERROR] Tests run: 70, Failures: 0, Errors: 1, Skipped: 5
[INFO]
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] flink-table-common . SUCCESS [ 9.955 s]
[INFO] flink-table-api-java ... SUCCESS [ 4.615 s]
[INFO] flink-table-api-java-bridge SUCCESS [ 4.150 s]
[INFO] flink-table-api-scala .. SUCCESS [ 2.843 s]
[INFO] flink-table-api-scala-bridge ... SUCCESS [ 2.843 s]
[INFO] flink-cep .. SUCCESS [ 30.868 s]
[INFO] flink-table-planner SUCCESS [04:51 min]
[INFO] flink-cep-scala SUCCESS [ 8.361 s]
[INFO] flink-sql-client ... FAILURE [04:21 min]
[INFO] flink-state-processor-api .. SKIPPED
[INFO]
[INFO] BUILD FAILURE
[INFO]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17337) Send UPDATE messages instead of INSERT and DELETE in streaming join operator
Jark Wu created FLINK-17337:
---
Summary: Send UPDATE messages instead of INSERT and DELETE in
streaming join operator
Key: FLINK-17337
URL: https://issues.apache.org/jira/browse/FLINK-17337
Project: Flink
Issue Type: Improvement
Components: Table SQL / Runtime
Reporter: Jark Wu
Currently, streaming join operator always send INSERT and DELETE messages for
simplification if it's not inner join. However, we can send UPDATE_BEFORE and
UPDATE_AFTER messages instead of INSERT and DELETE. For example, when we
recieve right record "b", then we can send {{UB[a, null]}} and {{UA[a,b]}}
instead of {{D[a,null]}}, {{I[a,b]}}. This is an optimization, because UB can
be omitted in some cases to reduce IO cost and computation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17336) MVN exited with EXIT CODE: 143. in "libraries" test job
Robert Metzger created FLINK-17336:
--
Summary: MVN exited with EXIT CODE: 143. in "libraries" test job
Key: FLINK-17336
URL: https://issues.apache.org/jira/browse/FLINK-17336
Project: Flink
Issue Type: Bug
Components: Build System / Azure Pipelines, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger
CI:https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=89=logs=56781494-ebb0-5eae-f732-b9c397ec6ede=32b25b6b-f46f-5bca-b5eb-2c6936ee77a4
maven reports "build success", but the exit code is 143?
{code}
[INFO] flink-state-processor-api .. SUCCESS [ 0.273 s]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 19:24 min
[INFO] Finished at: 2020-04-23T00:46:43+00:00
[INFO] Final Memory: 246M/4214M
[INFO]
[WARNING] The requested profile "e2e-hadoop" could not be activated because it
does not exist.
MVN exited with EXIT CODE: 143.
Trying to KILL watchdog (265).
==
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
