Re: [DISCUSS] Planning Flink 1.13

2021-01-18 Thread Khachatryan Roman
Hey Dawid, Replying to your penultimate message: > We have never done something like this before I thought releasing usability features can be such an experiment. My reasoning is the following. As each release requires coordination of all the participating teams, it will not scale beyond some

[jira] [Created] (FLINK-21022) flink-connector-es add onSuccess handler after bulk process for sync success data to other third party system for data consistency checking

2021-01-18 Thread Zheng WEI (Jira)
Zheng WEI created FLINK-21022: - Summary: flink-connector-es add onSuccess handler after bulk process for sync success data to other third party system for data consistency checking Key: FLINK-21022 URL:

[jira] [Created] (FLINK-21021) Bump Beam to 2.27.0

2021-01-18 Thread Dian Fu (Jira)
Dian Fu created FLINK-21021: --- Summary: Bump Beam to 2.27.0 Key: FLINK-21021 URL: https://issues.apache.org/jira/browse/FLINK-21021 Project: Flink Issue Type: Improvement Components: API

[jira] [Created] (FLINK-21019) Bump Netty 4 to 4.1.58

2021-01-18 Thread Dian Fu (Jira)
Dian Fu created FLINK-21019: --- Summary: Bump Netty 4 to 4.1.58 Key: FLINK-21019 URL: https://issues.apache.org/jira/browse/FLINK-21019 Project: Flink Issue Type: Improvement Components:

[jira] [Created] (FLINK-21020) Bump Jackson to 2.12.1

2021-01-18 Thread Dian Fu (Jira)
Dian Fu created FLINK-21020: --- Summary: Bump Jackson to 2.12.1 Key: FLINK-21020 URL: https://issues.apache.org/jira/browse/FLINK-21020 Project: Flink Issue Type: Improvement Components:

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-18 Thread Xingbo Huang
Thanks Xintong for the great work! Best, Xingbo Peter Huang 于2021年1月19日周二 下午12:51写道: > Thanks for the great effort to make this happen. It paves us from using > 1.12 soon. > > Best Regards > Peter Huang > > On Mon, Jan 18, 2021 at 8:16 PM Yang Wang wrote: > > > Thanks Xintong for the great

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-18 Thread Peter Huang
Thanks for the great effort to make this happen. It paves us from using 1.12 soon. Best Regards Peter Huang On Mon, Jan 18, 2021 at 8:16 PM Yang Wang wrote: > Thanks Xintong for the great work as our release manager! > > > Best, > Yang > > Xintong Song 于2021年1月19日周二 上午11:53写道: > >> The Apache

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-01-18 Thread Khachatryan Roman
+1 (non-binding) Thanks for driving the FLIP! Regards, Roman On Tue, Jan 19, 2021 at 2:58 AM Guowei Ma wrote: > +1 non-binding > Best, > Guowei > > > On Fri, Jan 15, 2021 at 10:56 PM Yun Gao > wrote: > > > > > Hi all, > > > > I would like to start the vote for FLIP-147[1], which propose to

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Forward Xu
+1 Dian Fu 于2021年1月19日周二 上午11:40写道: > +1 > > > 在 2021年1月19日,上午11:34,Jark Wu 写道: > > > > +1 > > > > On Tue, 19 Jan 2021 at 01:59, Till Rohrmann > wrote: > > > >> +1, > >> > >> Cheers, > >> Till > >> > >> On Mon, Jan 18, 2021 at 4:12 PM Chesnay Schepler > >> wrote: > >> > >>> +1 > >>> On

Re: [ANNOUNCE] Apache Flink 1.12.1 released

2021-01-18 Thread Yang Wang
Thanks Xintong for the great work as our release manager! Best, Yang Xintong Song 于2021年1月19日周二 上午11:53写道: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.1, which is the first bugfix release for the Apache Flink 1.12 > series. > > Apache Flink® is an

[ANNOUNCE] Apache Flink 1.12.1 released

2021-01-18 Thread Xintong Song
The Apache Flink community is very happy to announce the release of Apache Flink 1.12.1, which is the first bugfix release for the Apache Flink 1.12 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: [DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Yang Wang
Hi Ingo, Thanks for your response. 1. Not distinguishing JM/TM is reasonable, but what about the client side. For Yarn/K8s deployment, the local flink-conf.yaml will be shipped to JM/TM. So I am just confused about where should the environment variables be replaced? IIUC, it is not an issue for

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Dian Fu
+1 > 在 2021年1月19日,上午11:34,Jark Wu 写道: > > +1 > > On Tue, 19 Jan 2021 at 01:59, Till Rohrmann wrote: > >> +1, >> >> Cheers, >> Till >> >> On Mon, Jan 18, 2021 at 4:12 PM Chesnay Schepler >> wrote: >> >>> +1 >>> On 1/18/2021 3:50 PM, Seth Wiesman wrote: Addendum, 72 hours from now is

[jira] [Created] (FLINK-21018) Follow up of FLINK-20488 to update checkpoint related documentation

2021-01-18 Thread Yuan Mei (Jira)
Yuan Mei created FLINK-21018: Summary: Follow up of FLINK-20488 to update checkpoint related documentation Key: FLINK-21018 URL: https://issues.apache.org/jira/browse/FLINK-21018 Project: Flink

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Jark Wu
+1 On Tue, 19 Jan 2021 at 01:59, Till Rohrmann wrote: > +1, > > Cheers, > Till > > On Mon, Jan 18, 2021 at 4:12 PM Chesnay Schepler > wrote: > > > +1 > > On 1/18/2021 3:50 PM, Seth Wiesman wrote: > > > Addendum, 72 hours from now is Thursday the 21st :) > > > > > > sorry for the mistake. > > >

[jira] [Created] (FLINK-21017) Fix missing backquote in table connectors docs

2021-01-18 Thread Han (Jira)
Han created FLINK-21017: --- Summary: Fix missing backquote in table connectors docs Key: FLINK-21017 URL: https://issues.apache.org/jira/browse/FLINK-21017 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-21016) Upload missing packages to PyPI for release 1.12.1

2021-01-18 Thread Xintong Song (Jira)
Xintong Song created FLINK-21016: Summary: Upload missing packages to PyPI for release 1.12.1 Key: FLINK-21016 URL: https://issues.apache.org/jira/browse/FLINK-21016 Project: Flink Issue

Re: [VOTE] FLIP-147: Support Checkpoint After Tasks Finished

2021-01-18 Thread Guowei Ma
+1 non-binding Best, Guowei On Fri, Jan 15, 2021 at 10:56 PM Yun Gao wrote: > > Hi all, > > I would like to start the vote for FLIP-147[1], which propose to support > checkpoints after > tasks finished and is discussed in [2]. > > The vote will last at least 72 hours (Jan 20th due to weekend),

Re: [DISCUSS] FLIP-156: Runtime Interfaces for Fine-Grained Resource Requirements

2021-01-18 Thread Stephan Ewen
Thanks a lot, Yangze and Xintong for this FLIP. I want to say, first of all, that this is super well written. And the points that the FLIP makes about how to expose the configuration to users is exactly the right thing to figure out first. So good job here! About how to let users specify the

[jira] [Created] (FLINK-21015) Wait for checkpoint cleanup completion during shutdown

2021-01-18 Thread Roman Khachatryan (Jira)
Roman Khachatryan created FLINK-21015: - Summary: Wait for checkpoint cleanup completion during shutdown Key: FLINK-21015 URL: https://issues.apache.org/jira/browse/FLINK-21015 Project: Flink

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Till Rohrmann
+1, Cheers, Till On Mon, Jan 18, 2021 at 4:12 PM Chesnay Schepler wrote: > +1 > On 1/18/2021 3:50 PM, Seth Wiesman wrote: > > Addendum, 72 hours from now is Thursday the 21st :) > > > > sorry for the mistake. > > > > Seth > > > > On Mon, Jan 18, 2021 at 8:41 AM Timo Walther wrote: > > > >> +1

[jira] [Created] (FLINK-21014) Different timeouts for savepoints and checkpoints

2021-01-18 Thread Rex Remind (Jira)
Rex Remind created FLINK-21014: -- Summary: Different timeouts for savepoints and checkpoints Key: FLINK-21014 URL: https://issues.apache.org/jira/browse/FLINK-21014 Project: Flink Issue Type:

Re: [DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Steven Wu
Variable substitution (proposed here) is definitely useful. For us, hierarchical override is more useful. E.g., we may have the default value of "state.checkpoints.dir=path1" defined in flink-conf.yaml. But maybe we want to override it to "state.checkpoints.dir=path2" via environment variable in

[jira] [Created] (FLINK-21013) Blink planner does not ingest timestamp into StreamRecord

2021-01-18 Thread Timo Walther (Jira)
Timo Walther created FLINK-21013: Summary: Blink planner does not ingest timestamp into StreamRecord Key: FLINK-21013 URL: https://issues.apache.org/jira/browse/FLINK-21013 Project: Flink

Re: [DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Ingo Bürk
Hi Yang, thanks for your questions! I'm glad to see this feature is being received positively. ad 1) We don't distinguish JM/TM, and I can't think of a good reason why a user would want to do so. I'm not very experienced with Flink, however, so please excuse me if I'm overlooking some obvious

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Chesnay Schepler
+1 On 1/18/2021 3:50 PM, Seth Wiesman wrote: Addendum, 72 hours from now is Thursday the 21st :) sorry for the mistake. Seth On Mon, Jan 18, 2021 at 8:41 AM Timo Walther wrote: +1 Thanks for upgrading our docs infrastructure. Regards, Timo On 18.01.21 15:29, Seth Wiesman wrote: Hi

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Dawid Wysakowicz
+1 On 18/01/2021 15:50, Seth Wiesman wrote: > Addendum, 72 hours from now is Thursday the 21st :) > > sorry for the mistake. > > Seth > > On Mon, Jan 18, 2021 at 8:41 AM Timo Walther wrote: > >> +1 >> >> Thanks for upgrading our docs infrastructure. >> >> Regards, >> Timo >> >> On 18.01.21

Re: [DISCUSS] Planning Flink 1.13

2021-01-18 Thread Dawid Wysakowicz
Hi all, Thank you all for a warm reception as release managers. We are also glad that the majority agrees we should aim for the end of March as the feature freeze date. 1. Till created a page in our wiki to gather potential features that we want to include in the release. Thank you Till!

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Seth Wiesman
Addendum, 72 hours from now is Thursday the 21st :) sorry for the mistake. Seth On Mon, Jan 18, 2021 at 8:41 AM Timo Walther wrote: > +1 > > Thanks for upgrading our docs infrastructure. > > Regards, > Timo > > On 18.01.21 15:29, Seth Wiesman wrote: > > Hi devs, > > > > The discussion of the

Re: [DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Yang Wang
Thanks for kicking off the discussion. I think supporting environment variables rendering in the Flink configuration yaml file is a good idea. Especially for the Kubernetes environment since we are using the secret resource to store the authentication information. But I have some questions for

Re: [Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Timo Walther
+1 Thanks for upgrading our docs infrastructure. Regards, Timo On 18.01.21 15:29, Seth Wiesman wrote: Hi devs, The discussion of the FLIP-157 [1] seems has reached a consensus through the mailing thread [2]. I would like to start a vote for it. The vote will be opened until 20th January

[Vote] FLIP-157 Migrate Flink Documentation from Jekyll to Hugo

2021-01-18 Thread Seth Wiesman
Hi devs, The discussion of the FLIP-157 [1] seems has reached a consensus through the mailing thread [2]. I would like to start a vote for it. The vote will be opened until 20th January (72h), unless there is an objection or no enough votes. Best, Seth [1]

[jira] [Created] (FLINK-21012) AvroFileFormatFactory uses non-deserializable lambda function

2021-01-18 Thread Jira
Ingo Bürk created FLINK-21012: - Summary: AvroFileFormatFactory uses non-deserializable lambda function Key: FLINK-21012 URL: https://issues.apache.org/jira/browse/FLINK-21012 Project: Flink

[jira] [Created] (FLINK-21011) Separete the implementation of StreamExecJoin

2021-01-18 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-21011: --- Summary: Separete the implementation of StreamExecJoin Key: FLINK-21011 URL: https://issues.apache.org/jira/browse/FLINK-21011 Project: Flink Issue Type:

[jira] [Created] (FLINK-21010) Separete the implementation of StreamExecJoin

2021-01-18 Thread Wenlong Lyu (Jira)
Wenlong Lyu created FLINK-21010: --- Summary: Separete the implementation of StreamExecJoin Key: FLINK-21010 URL: https://issues.apache.org/jira/browse/FLINK-21010 Project: Flink Issue Type:

Re: [DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Khachatryan Roman
Hi Ingo, Thanks a lot for this proposal! We had a related discussion recently in the context of FLINK-19520 (randomizing tests configuration) [1]. I believe other scenarios will benefit as well. For the end users, I think substitution in configuration files is preferable over parsing env vars

Re: [DISCUSS] Planning Flink 1.13

2021-01-18 Thread Dawid Wysakowicz
Hey Roman, Thanks for the idea! Personally I am a bit torn about this. We have never done something like this before. In principle I like the idea of shorter release cycles. However I am bit skeptical about the timing to do it now. Do we know what would be included in such a release? I am a bit

Re: [DISCUSS] Planning Flink 1.13

2021-01-18 Thread Piotr Nowojski
+1 for both proposals. Thanks! Piotrek pon., 18 sty 2021 o 11:04 Yun Tang napisał(a): > +1 for the proposed date and thanks for Dawid & Guowei as the release > manager. > > Best > Yun Tang > > From: Yangze Guo > Sent: Friday, January 15, 2021 9:50 > To: dev >

Re: Re: [DISCUSS] Dealing with deprecated and legacy code in Flink

2021-01-18 Thread Yun Gao
Hi, Very thanks for @Timo to initiate the discussion! I would also +1 for providing some informations to users via annotations or documents in advanced to not suprise users before we actually remove the legacy code. If we finally decide to change one functionality that user could sense,

[jira] [Created] (FLINK-21009) Can not disable certain options in Elasticsearch 7 connector

2021-01-18 Thread Dawid Wysakowicz (Jira)
Dawid Wysakowicz created FLINK-21009: Summary: Can not disable certain options in Elasticsearch 7 connector Key: FLINK-21009 URL: https://issues.apache.org/jira/browse/FLINK-21009 Project: Flink

Re: [DISCUSS] Dealing with deprecated and legacy code in Flink

2021-01-18 Thread Piotr Nowojski
Hi Timo, Thanks for starting this discussion. I'm not sure how we should approach this topic and what should be our final recommendation, but definitely clearing up a couple of things would be helpful. For starters, I agree it would be good to have some more information, besides just

[DISCUSS] Flink configuration from environment variables

2021-01-18 Thread Ingo Bürk
Hi everyone, in Ververica Platform we offer a feature to use environment variables in the Flink configuration¹, e.g. ``` s3.access-key: ${S3_ACCESS_KEY} ``` We've been discussing internally whether contributing such a feature to Flink directly would make sense and wanted to start a discussion

Re: [DISCUSS] Planning Flink 1.13

2021-01-18 Thread Yun Tang
+1 for the proposed date and thanks for Dawid & Guowei as the release manager. Best Yun Tang From: Yangze Guo Sent: Friday, January 15, 2021 9:50 To: dev Subject: Re: [DISCUSS] Planning Flink 1.13 Thanks for being the release managers, Guowei and Dawid. +1 for

[jira] [Created] (FLINK-21008) ClusterEntrypoint#shutDownAsync may not be fully executed

2021-01-18 Thread Yang Wang (Jira)
Yang Wang created FLINK-21008: - Summary: ClusterEntrypoint#shutDownAsync may not be fully executed Key: FLINK-21008 URL: https://issues.apache.org/jira/browse/FLINK-21008 Project: Flink Issue

Re: Flink performance issue with Google Cloud Storage- (Small Parquet Files)

2021-01-18 Thread Till Rohrmann
Hi Aswin, I am not aware of any particular problems. In order to pinpoint the problem, I would suggest to reduce the complexity of the job to see where it goes wrong. For example, you could try to not use the Hive connector and instead write directly to GCS. Then you could try to write to a

[ANNOUNCE] Stateful Functions Docker images are now hosted on Dockerhub at apache/flink-statefun

2021-01-18 Thread Tzu-Li (Gordon) Tai
Hi, We have created an "apache/flink-statefun" Dockerhub repository managed by the Flink PMC, at: https://hub.docker.com/r/apache/flink-statefun The images for the latest stable StateFun release, 2.2.2, have already been pushed there. Going forward, it will be part of the release process to make