[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047339#comment-17047339 ] Maximilian Michels commented on BEAM-9299: -- I think we'll move this to 2.21.0 to allow time for these changes to stabilize. Looking at the PR, I'm in favor of approach (1), removing 1.7 first. There is too much code complexity added now and we're planning to remove 1.7 anyways. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17043975#comment-17043975 ] Rui Wang commented on BEAM-9299: I am going to cut 2.20.0 branch on 02/26/2020. Can this Jira be finished by that date? Will you able to push this back to 2.21.0? > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041755#comment-17041755 ] Ismaël Mejía commented on BEAM-9299: [~sunjincheng121]it is probably a good idea that you take a look at BEAM-9345 since like an issue worth to fix as part of this. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17039584#comment-17039584 ] sunjincheng commented on BEAM-9299: --- Currently we always test the last version for flink runners，and I think it's by design for now, and I agree with you [~iemejia], would be better to avoid test all versions for every PR. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038959#comment-17038959 ] Ismaël Mejía commented on BEAM-9299: +1 to go ahead with the updates. For the tests on all supported versions I think that currently all the tests are run with the oldest version (or is it the most recent)? We should probably have a sort of postcommit for all the versions but not do this by default for every PR otherwise it would be nightmarish and too slow. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038898#comment-17038898 ] sunjincheng commented on BEAM-9299: --- As BEAM-5396 and FLINK-11048 have been completed so far, we have a cleaner solution 3, which is to remove `BeamFlinkRemoteStreamEnvironment` class. I have updated the PR, but the content does not include the changes of `cover all of the version for test` which [~angoenka] mentioned before, I think we can handle it in another PR. What do you think? > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038240#comment-17038240 ] sunjincheng commented on BEAM-9299: --- Thanks for your feedback [~iemejia] [~angoenka]! I will update PR according to solution 2. And I agree that it's better to let the test cover all of versions of runner. But I'm curious. Why only test the latest one before, Is this by design？ > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 50m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17037271#comment-17037271 ] Ankur Goenka commented on BEAM-9299: +1 for option 2. We should not make deprecation of old flink version a blocker for adding support for new flink versions. As a side item, I think we should have easy ways to test old versions of flink. the beam precommit tests primarily use latest flink version. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036790#comment-17036790 ] Ismaël Mejía commented on BEAM-9299: +1 for solution 2, it does not look like a big burden to maintain that copy and it will allow users to catch the latest and greatest of the previous releases. I might be wrong but if I remember correctly `JobWithJars.buildUserCodeClassLoader` was removed as part of 1.10.x so being isolated from it could prove useful for the 1.10 upgrade. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 40m > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17036700#comment-17036700 ] sunjincheng commented on BEAM-9299: --- I found that the issue FLINK-15844 also exists in Flink 1.8.3. I noticed that FLINK-15844 has provided a fix for 1.9.3. However, AFAIK, there will be no new 1.8 releases any more according to FLink's [release policy|[https://flink.apache.org/downloads.html#update-policy-for-old-releases]]. There are two solutions in my mind: - Solution1: As the signature of `JobWithJars.buildUserCodeClassLoader `has changed in both Flink 1.8.3 and 1.9.2 and Flink 1.7.2 still uses the old signature, we could drop the Flink 1.7 support firstly and then update the implementation of `FlinkExecutionEnvironments` to use the new signature. -Solution2: We could make a copy of `FlinkExecutionEnvironments` in each version of Flink runner and update the implementation for each copy according to the Flink version. This solution decouples the drop of Flink 1.7 support and the upgrades of 1.8 and 1.9. Besides, Flink community has made big change for the job submission path(FLINK-13954) in 1.10, e.g. `JobWithJars` has been removed in 1.10. It means that the job submission logic will be separate for 1.8/1.9 and 1.10 anyway. Personally I prefer solution2 and what's your thought? :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035945#comment-17035945 ] sunjincheng commented on BEAM-9299: --- Thanks for add this input [~iemejia] , I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released due to this upgrade is not urgent. :) > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035941#comment-17035941 ] sunjincheng commented on BEAM-9299: --- Thanks for add this input [~iemejia] I think we can upgrade the 1.9.1 to 1.9.3 when Flink 1.9.3 released. :) and do not upgrade the 1.9.1 in this JIRA. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9299) Upgrade Flink Runner to 1.8.3 and 1.9.2
[ https://issues.apache.org/jira/browse/BEAM-9299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17034796#comment-17034796 ] Ismaël Mejía commented on BEAM-9299: For ref Flink 1.9.2 introduced a bacwards incompatible change FLINK-15844 so we need a workaround or just to avoid the hassle.. > Upgrade Flink Runner to 1.8.3 and 1.9.2 > --- > > Key: BEAM-9299 > URL: https://issues.apache.org/jira/browse/BEAM-9299 > Project: Beam > Issue Type: Task > Components: runner-flink >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.20.0 > > > I would like to Upgrade Flink Runner to 18.3 and 1.9.2 due to both the Apache > Flink 1.8.3 and Apache Flink 1.9.2 have been released [1]. > What do you think? > [1] https://dist.apache.org/repos/dist/release/flink/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
