Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Tzu-Li (Gordon) Tai
Vote thread for RC3 has been started: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-1-9-0-release-candidate-3-td31988.html On Mon, Aug 19, 2019 at 6:32 PM Tzu-Li (Gordon) Tai wrote: > Thanks for the comments and fast fixes. > > @Becket Qin I've quickly looked

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Tzu-Li (Gordon) Tai
Thanks for the comments and fast fixes. @Becket Qin I've quickly looked at the changes to the PubSub connector. Given that it is a API-breaking change and is quite local as a configuration change, I've decided to include that change in RC3. @Jark @Timo Walther I'll be adding FLINK-13699 as

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Stephan Ewen
Looking at FLINK-13699, it seems to be very local to Table API and HBase connector. We can cherry-pick that without re-running distributed tests. On Mon, Aug 19, 2019 at 1:46 PM Till Rohrmann wrote: > I've merged the fix for FLINK-13752. Hence we are good to go to create the > new RC. > >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Till Rohrmann
I've merged the fix for FLINK-13752. Hence we are good to go to create the new RC. Cheers, Till On Mon, Aug 19, 2019 at 1:30 PM Timo Walther wrote: > I support Jark's fix for FLINK-13699 because it would be disappointing > if both DDL and connectors are ready to handle DATE/TIME/TIMESTAMP but

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Timo Walther
I support Jark's fix for FLINK-13699 because it would be disappointing if both DDL and connectors are ready to handle DATE/TIME/TIMESTAMP but a little component in the middle of the stack is preventing an otherwise usable feature. The changes are minor. Thanks, Timo Am 19.08.19 um 13:24

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Jark Wu
Hi Gordon, I agree that we should pick the minimal set of changes to shorten the release testing time. However, I would like to include FLINK-13699 into RC3. FLINK-13699 is a critical DDL issue, and is a small change to flink table (won't affect the runtime feature and stability). I will do some

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Stephan Ewen
+1 for Gordon's approach. If we do that, we can probably skip re-testing everything and mainly need to verify the release artifacts (signatures, build from source, etc.). If we open the RC up for changes, I fear a lot of small issues will rush in and destabilize the candidate again, meaning we

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Till Rohrmann
+1 for only cherry picking FLINK-13752 and the LICENSE fixes into RC 3. Cheers, Till On Mon, Aug 19, 2019 at 9:48 AM Becket Qin wrote: > Hi Gordon, > > I remember we mentioned earlier that if there is an additional RC, we can > piggyback the GCP PubSub API change ( >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Becket Qin
Hi Gordon, I remember we mentioned earlier that if there is an additional RC, we can piggyback the GCP PubSub API change ( https://issues.apache.org/jira/browse/FLINK-13231). It is a small patch to avoid future API change. So should be able to merge it very shortly. Would it be possible to

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-19 Thread Tzu-Li (Gordon) Tai
Hi, https://issues.apache.org/jira/browse/FLINK-13752 turns out to be an actual blocker, so we would have to close this RC now in favor of a new one. Since we are already quite past the planned release time for 1.9.0, I would like to limit the new changes included in RC3 to only the following: -

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-18 Thread Zili Chen
We should investigate the performance regression but regardless the regression I vote +1 Have verified following things - Jobs running on YARN x (Session & Per Job) with high-availability enabled. - Simulate JM and TM failures. - Simulate temporary network partition. Best, tison. Stephan Ewen

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-18 Thread Stephan Ewen
For reference, this is the JIRA issue about the regression in question: https://issues.apache.org/jira/browse/FLINK-13752 On Fri, Aug 16, 2019 at 10:57 AM Guowei Ma wrote: > Hi, till > I can send the job to you offline. > It is just a datastream job and does not use

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-16 Thread Gyula Fóra
Hi all, I agree with Till that we should investigate the suspected performance regression issue before proceeding with the release. If we do not find any problem I vote +1 I have verified the following behaviour: - Built flink with custom hadoop version - YARN Deployment with and without

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-16 Thread Guowei Ma
Hi, till I can send the job to you offline. It is just a datastream job and does not use TwoInputSelectableStreamTask. A->B \ C / D->E Best, Guowei Till Rohrmann 于2019年8月16日周五 下午4:34写道: > Thanks for reporting this issue Guowei. Could you share a bit more details >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-16 Thread Till Rohrmann
Thanks for reporting this issue Guowei. Could you share a bit more details what the job exactly does and which operators it uses? Does the job uses the new `TwoInputSelectableStreamTask` which might cause the performance regression? I think it is important to understand where the problem comes

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-16 Thread Guowei Ma
Hi, -1 We have a benchmark job, which includes a two-input operator. This job has a big performance regression using 1.9 compared to 1.8. It's still not very clear why this regression happens. Best, Guowei Yu Li 于2019年8月16日周五 下午3:27写道: > +1 (non-binding) > > - checked release notes: OK > -

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-16 Thread Yu Li
+1 (non-binding) - checked release notes: OK - checked sums and signatures: OK - source release - contains no binaries: OK - contains no 1.9-SNAPSHOT references: OK - build from source: OK (8u102) - mvn clean verify: OK (8u102) - binary release - no examples appear to be

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Bowen Li
Hi Jark, Thanks for letting me know that it's been like this in previous releases. Though I don't think that's the right behavior, it can be discussed for later release. Thus I retract my -1 for RC2. Bowen On Thu, Aug 15, 2019 at 7:49 PM Jark Wu wrote: > Hi Bowen, > > Thanks for reporting

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Jark Wu
Hi Bowen, Thanks for reporting this. However, I don't think this is an issue. IMO, it is by design. The `tEnv.listUserDefinedFunctions()` in Table API and `show functions;` in SQL CLI are intended to return only the registered UDFs, not including built-in functions. This is also the behavior in

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Bowen Li
-1 for RC2. I found a bug https://issues.apache.org/jira/browse/FLINK-13741, and I think it's a blocker. The bug means currently if users call `tEnv.listUserDefinedFunctions()` in Table API or `show functions;` thru SQL would not be able to see Flink's built-in functions. I'm preparing a fix

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Tzu-Li (Gordon) Tai
Thanks for all the test efforts, verifications and votes so far. So far, things are looking good, but we still require one more PMC binding vote for this RC to be the official release, so I would like to extend the vote time for 1 more day, until *Aug. 16th 17:00 CET*. In the meantime, the

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Kurt Young
Great, then I have no other comments on legal check. Best, Kurt On Thu, Aug 15, 2019 at 9:56 PM Chesnay Schepler wrote: > The licensing items aren't a problem; we don't care about Flink modules > in NOTICE files, and we don't have to update the source-release > licensing since we don't have a

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Chesnay Schepler
The licensing items aren't a problem; we don't care about Flink modules in NOTICE files, and we don't have to update the source-release licensing since we don't have a pre-built version of the WebUI in the source. On 15/08/2019 15:22, Kurt Young wrote: After going through the licenses, I

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Dawid Wysakowicz
Thanks Kurt for checking that. The mentioned problem with table-examples is that, when working on FLINK-13558, I forgot to add dependency on flink-examples-table to flink-dist. So this module is not built if only the flink-dist with its dependencies is built (this happens in the release scripts:

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Kurt Young
After going through the licenses, I found 2 suspicions but not sure if they are valid or not. 1. flink-state-processing-api is packaged in to flink-dist jar, but not included in NOTICE-binary file (the one under the root directory) like other modules. 2. flink-runtime-web distributed some

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Kurt Young
Hi Gordon & Timo, Thanks for the feedback, and I agree with it. I will document this in the release notes. Best, Kurt On Thu, Aug 15, 2019 at 6:14 PM Tzu-Li (Gordon) Tai wrote: > Hi Kurt, > > With the same argument as before, given that it is mentioned in the release > announcement that it

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Tzu-Li (Gordon) Tai
Hi Kurt, With the same argument as before, given that it is mentioned in the release announcement that it is a preview feature, I would not block this release because of it. Nevertheless, it would be important to mention this explicitly in the release notes [1]. Regards, Gordon [1]

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Andrey Zagrebin
+1 (non-binding) Tested in AWS EMR Yarn: 1 master and 4 worker nodes (m5.xlarge: 4 vCore, 16 GiB). EMR runs only on Java 8. Fine-grained recovery is enabled by default. Modified E2E test scripts can be found here (asserting output): https://github.com/azagrebin/flink/commits/FLINK-13597 Batch

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Timo Walther
Hi Kurt, I agree that this is a serious bug. However, I would not block the release because of this. As you said, there is a workaround and the `execute()` works in the most common case of a single execution. We can fix this in a minor release shortly after. What do others think? Regards,

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Kurt Young
HI, We just find a serious bug around blink planner: https://issues.apache.org/jira/browse/FLINK-13708 When user reused the table environment instance, and call `execute` method multiple times for different sql, the later call will trigger the earlier ones to be re-executed. It's a serious bug

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-15 Thread Gary Yao
+1 (non-binding) Jepsen test suite passed 10 times consecutively On Wed, Aug 14, 2019 at 5:31 PM Aljoscha Krettek wrote: > +1 > > I did some testing on a Google Cloud Dataproc cluster (it gives you a > managed YARN and Google Cloud Storage (GCS)): > - tried both YARN session mode and YARN

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Kurt Young
Hi Robert, I will do it today. Best, Kurt On Wed, Aug 14, 2019 at 11:55 PM Robert Metzger wrote: > Has anybody verified the inclusion of all bundled dependencies into the > NOTICE files? > > I'm asking because we had some issues with that in the last release(s). > > On Wed, Aug 14, 2019 at

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Robert Metzger
Has anybody verified the inclusion of all bundled dependencies into the NOTICE files? I'm asking because we had some issues with that in the last release(s). On Wed, Aug 14, 2019 at 5:31 PM Aljoscha Krettek wrote: > +1 > > I did some testing on a Google Cloud Dataproc cluster (it gives you a >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Aljoscha Krettek
+1 I did some testing on a Google Cloud Dataproc cluster (it gives you a managed YARN and Google Cloud Storage (GCS)): - tried both YARN session mode and YARN per-job mode, also using bin/flink list/cancel/etc. against a YARN session cluster - ran examples that write to GCS, both with the

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Kurt Young
Hi, Thanks for preparing this release candidate. I have verified the following: - verified the checksums and GPG files match the corresponding release files - verified that the source archives do not contains any binaries - build the source release with Scala 2.11 successfully. - ran `mvn

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-14 Thread Jark Wu
Hi Gordon, I have verified the following things: - build the source release with Scala 2.12 and Scala 2.11 successfully - checked/verified signatures and hashes - checked that all POM files point to the same version - ran some flink table related end-to-end tests locally and succeeded (except

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-13 Thread Till Rohrmann
Hi Richard, although I can see that it would be handy for users who have PubSub set up, I would rather not include examples which require an external dependency into the Flink distribution. I think examples should be self-contained. My concern is that we would bloat the distribution for many

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-13 Thread Jark Wu
Hi Till, After thinking about we can use VARCHAR as an alternative of timestamp/time/date. I'm fine with not recognize it as a blocker issue. We can fix it into 1.9.1. Thanks, Jark On Tue, 13 Aug 2019 at 15:10, Richard Deurwaarder wrote: > Hello all, > > I noticed the PubSub example jar is

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-13 Thread Richard Deurwaarder
Hello all, I noticed the PubSub example jar is not included in the examples/ dir of flink-dist. I've created https://issues.apache.org/jira/browse/FLINK-13700 + https://github.com/apache/flink/pull/9424/files to fix this. I will leave it up to you to decide if we want to add this to 1.9.0.

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-13 Thread Till Rohrmann
Hi Jark, thanks for reporting this issue. Could this be a documented limitation of Blink's preview version? I think we have agreed that the Blink SQL planner will be rather a preview feature than production ready. Hence it could still contain some bugs. My concern is that there might be still

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Jark Wu
Hi all, I just find an issue when testing connector DDLs against blink planner for rc2. This issue lead to the DDL doesn't work when containing timestamp/date/time type. I have created an issue FLINK-13699[1] and a pull request for this. IMO, this can be a blocker issue of 1.9 release. Because

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Becket Qin
Thanks Gordon, will do that. On Mon, Aug 12, 2019 at 4:42 PM Tzu-Li (Gordon) Tai wrote: > Concerning FLINK-13231: > > Since this is a @PublicEvolving interface, technically it is ok to break > it across releases (including across bugfix releases?). > So, @Becket if you do merge it now, please

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Becket Qin
That sounds good to me. I was initially trying to piggyback it into an RC, but fell behind and was not able to catch the last one. Thanks, Jiangjie (Becket) Qin On Mon, Aug 12, 2019 at 4:25 PM Till Rohrmann wrote: > I agree that it would be nicer. Not sure whether we should cancel the RC >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Tzu-Li (Gordon) Tai
Concerning FLINK-13231: Since this is a @PublicEvolving interface, technically it is ok to break it across releases (including across bugfix releases?). So, @Becket if you do merge it now, please mark the fix version as 1.9.1. During the voting process, in the case a new RC is created, we

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Till Rohrmann
I agree that it would be nicer. Not sure whether we should cancel the RC for this issue given that it is open for quite some time and hasn't been addressed until very recently. Maybe we could include it on the shortlist of nice-to-do things which we do in case that the RC gets cancelled. Cheers,

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Becket Qin
Hi Till, Yes, I think we have already documented in that way. So technically speaking it is fine to change it later. It is just better if we could avoid doing that. Thanks, Jiangjie (Becket) Qin On Mon, Aug 12, 2019 at 4:09 PM Till Rohrmann wrote: > Could we say that the PubSub connector is

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Till Rohrmann
Could we say that the PubSub connector is public evolving instead? Cheers, Till On Mon, Aug 12, 2019 at 3:18 PM Becket Qin wrote: > Hi all, > > FLINK-13231(palindrome!) has a minor Google PubSub connector API change > regarding how to config rate limiting. The GCP PubSub connector is a newly >

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Becket Qin
Hi all, FLINK-13231(palindrome!) has a minor Google PubSub connector API change regarding how to config rate limiting. The GCP PubSub connector is a newly introduced connector in 1.9, so it would be nice to include this change into 1.9 rather than later to avoid a public API change. I am thinking

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Zili Chen
Hi Kurt, Thanks for your explanation. For [1] I think at least we should change the JIRA issue field, like unset the fixed version. For [2] I can see the change is all in test scope but wonder if such a commit still invalid the release candidate. IIRC previous RC VOTE threads would contain a

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Kurt Young
Hi Zili, Thanks for the heads up. The 2 issues you mentioned were opened by me. We have found the reason of the second issue and a PR was opened for it. As said in jira, the issue was just a testing problem, should not be blocker of 1.9.0 release. However, we will still merge it into 1.9 branch.

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Zili Chen
Hi, I just noticed that a few hours ago there were two new issues filed and marked as blockers to 1.9.0[1][2]. Now [1] is closed as duplication but still marked as a blocker to 1.9.0, while [2] is downgrade to "Major" priority but still target to be fixed in 1.9.0. It would be worth to have

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Gyula Fóra
Thanks Stephan :) That looks easy enough, will try! Gyula On Mon, Aug 12, 2019 at 11:00 AM Stephan Ewen wrote: > Hi Gyula! > > Thanks for reporting this. > > Can you try to simply build Flink without Hadoop and then exporting > HADOOP_CLASSPATH to your CloudEra libs? > That is the recommended

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Stephan Ewen
Hi Gyula! Thanks for reporting this. Can you try to simply build Flink without Hadoop and then exporting HADOOP_CLASSPATH to your CloudEra libs? That is the recommended way these days. Best, Stephan On Mon, Aug 12, 2019 at 10:48 AM Gyula Fóra wrote: > Thanks Dawid, > > In the meantime I

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Gyula Fóra
Thanks Dawid, In the meantime I also figured out that I need to build the https://github.com/apache/flink-shaded project locally with -Dhadoop.version set to the specific hadoop version if I want something different. Cheers, Gyula On Mon, Aug 12, 2019 at 9:54 AM Dawid Wysakowicz wrote: > Hi

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-12 Thread Dawid Wysakowicz
Hi Gyula, As for the issues with mapr maven repository, you might have a look at this message: https://lists.apache.org/thread.html/77f4db930216e6da0d6121065149cef43ff3ea33c9ffe9b1a3047210@%3Cdev.flink.apache.org%3E Try using the "unsafe-mapr-repo" profile. Best, Dawid On 11/08/2019 19:31,

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-11 Thread Gyula Fóra
Hi again, How do I build the RC locally with the hadoop version specified? Seems like no matter what I do I run into dependency problems with the shaded hadoop dependencies. This seems to have worked in the past. There might be some documentation somewhere that I couldnt find, so I would

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-11 Thread Gyula Fóra
Hi! I am trying to build 1.9.0-rc2 with the -Pvendor-repos profile enabled. I get the following error: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.6.0 -Pinclude-hadoop (ignore that the hadoop version is not a vendor hadoop version) [ERROR] Failed to execute goal on project

Re: [VOTE] Apache Flink Release 1.9.0, release candidate #2

2019-08-09 Thread Tzu-Li (Gordon) Tai
Please note that the unresolved issues that are still tagged with a fix version "1.9.0", as seen in the JIRA release notes [1], are issues to update documents for new features. I've left them still associated with 1.9.0 since these should still be updated for 1.9.0 soon along with the official