Hello, we fixed the issue and are ready to test :) - is there a RC already available?
Best, Tobi On Wed, Feb 26, 2020 at 12:59 PM Kaymak, Tobias <[email protected]> wrote: > Hello, > > happy to help testing! I am currently fixing a networking issue between > our dev cluster for integration tests and the Kafka it is consuming from. > After that I would be ready to spin it up and test > > Best, > Tobi > > On Mon, Feb 24, 2020 at 10:13 PM Maximilian Michels <[email protected]> > wrote: > >> Thank you for reporting / filing / collecting the issues. >> >> There is a fix pending: https://github.com/apache/beam/pull/10950 >> >> As for the upgrade issues, the 1.8 and 1.9 upgrade is trivial. I will >> check out the Flink 1.10 PR tomorrow. >> >> Cheers, >> Max >> >> On 24.02.20 09:26, Ismaël Mejía wrote: >> > We are cutting the release branch for 2.20.0 next wednesday, so not >> sure >> > if these tickets will make it, but hopefully. >> > >> > For ref, >> > BEAM-9295 Add Flink 1.10 build target and Make FlinkRunner compatible >> > with Flink 1.10 >> > BEAM-9299 Upgrade Flink Runner to 1.8.3 and 1.9.2 >> > >> > In any case if you have cycles to help test any of the related tickets >> > PRs that would help too. >> > >> > >> > On Mon, Feb 24, 2020 at 8:47 AM Kaymak, Tobias < >> [email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi Kyle, >> > >> > thank you for creating the JIRA ticket, I think my best option right >> > now is to wait for a Beam version that is running on Flink 1.10 then >> > - unless there is a new Beam release around the corner :) >> > >> > Best, >> > Tobi >> > >> > On Thu, Feb 20, 2020 at 11:52 PM Kyle Weaver <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > Hi Tobi, >> > >> > This seems like a bug with Beam 2.19. I filed >> > https://issues.apache.org/jira/browse/BEAM-9345 to track the >> issue. >> > >> > > What puzzles me is that the session cluster should be allowed >> > to have multiple environments in detached mode - or am I wrong? >> > >> > It looks like that check is removed in Flink 1.10: >> > https://issues.apache.org/jira/browse/FLINK-15201 >> > >> > Thanks for reporting. >> > Kyle >> > >> > On Thu, Feb 20, 2020 at 4:10 AM Kaymak, Tobias >> > <[email protected] <mailto:[email protected]>> >> wrote: >> > >> > Hello, >> > >> > I am trying to upgrade from a Flink session cluster 1.8 to >> > 1.9 and from Beam 2.16.0 to 2.19.0. >> > Everything went quite smoothly, the local runner and the >> > local Flink runner work flawlessly. >> > >> > However when I: >> > 1. Generate a Beam jar for the FlinkRunner via maven (mvn >> > package -PFlinkRunner) >> > 2. Glue that into a Flink 1.9 docker image >> > 3. Start the image as a Standalone Session Cluster >> > >> > When I try to launch the first pipeline I get the following >> > exception >> > >> > org.apache.flink.client.program.ProgramInvocationException: >> > The main method caused an error: Failed to construct >> > instance from factory method >> > FlinkRunner#fromOptions(interface >> > org.apache.beam.sdk.options.PipelineOptions) >> > at >> > >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593) >> > at >> > >> >> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438) >> > at >> > >> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274) >> > at >> > >> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746) >> > at >> > >> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273) >> > at >> > >> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205) >> > at >> > >> >> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010) >> > at >> > >> >> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083) >> > at >> > >> >> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) >> > at >> > >> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083) >> > Caused by: java.lang.RuntimeException: Failed to construct >> > instance from factory method >> > FlinkRunner#fromOptions(interface >> > org.apache.beam.sdk.options.PipelineOptions) >> > at >> > >> >> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224) >> > at >> > >> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155) >> > at >> > >> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55) >> > at >> org.apache.beam.sdk.Pipeline.run(Pipeline.java:309) >> > at >> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301) >> > at >> > >> ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180) >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> > >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >> > >> >> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576) >> > ... 9 more >> > Caused by: java.lang.reflect.InvocationTargetException >> > at >> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> > at >> > >> >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> > at >> > >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> > at java.lang.reflect.Method.invoke(Method.java:498) >> > at >> > >> >> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214) >> > ... 19 more >> > Caused by: >> > org.apache.flink.api.common.InvalidProgramException: >> > Multiple environments cannot be created in detached mode >> > at >> > >> >> org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67) >> > at java.util.Optional.map(Optional.java:215) >> > at >> > >> >> org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068) >> > at >> > >> >> org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43) >> > at >> > >> org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96) >> > at >> > >> org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90) >> > ... 24 more >> > >> > I've checked the release notes and the issues and couldn't >> > find anything that relates to this. What puzzles me is that >> > the session cluster should be allowed to have multiple >> > environments in detached mode - or am I wrong? >> > >> > Best, >> > Tobi >> > >> > >> > >> > -- >> > >> > Tobias Kaymak >> > Data Engineer >> > Data Intelligence >> > >> > [email protected] <mailto:[email protected]> >> > www.ricardo.ch <http://www.ricardo.ch/> >> > Theilerstrasse 1a, 6300 Zug >> > > >
