Thank you for reporting / filing / collecting the issues.
There is a fix pending: https://github.com/apache/beam/pull/10950
As for the upgrade issues, the 1.8 and 1.9 upgrade is trivial. I will
check out the Flink 1.10 PR tomorrow.
Cheers,
Max
On 24.02.20 09:26, Ismaël Mejía wrote:
We are cutting the release branch for 2.20.0 next wednesday, so not sure
if these tickets will make it, but hopefully.
For ref,
BEAM-9295 Add Flink 1.10 build target and Make FlinkRunner compatible
with Flink 1.10
BEAM-9299 Upgrade Flink Runner to 1.8.3 and 1.9.2
In any case if you have cycles to help test any of the related tickets
PRs that would help too.
On Mon, Feb 24, 2020 at 8:47 AM Kaymak, Tobias <[email protected]
<mailto:[email protected]>> wrote:
Hi Kyle,
thank you for creating the JIRA ticket, I think my best option right
now is to wait for a Beam version that is running on Flink 1.10 then
- unless there is a new Beam release around the corner :)
Best,
Tobi
On Thu, Feb 20, 2020 at 11:52 PM Kyle Weaver <[email protected]
<mailto:[email protected]>> wrote:
Hi Tobi,
This seems like a bug with Beam 2.19. I filed
https://issues.apache.org/jira/browse/BEAM-9345 to track the issue.
> What puzzles me is that the session cluster should be allowed
to have multiple environments in detached mode - or am I wrong?
It looks like that check is removed in Flink 1.10:
https://issues.apache.org/jira/browse/FLINK-15201
Thanks for reporting.
Kyle
On Thu, Feb 20, 2020 at 4:10 AM Kaymak, Tobias
<[email protected] <mailto:[email protected]>> wrote:
Hello,
I am trying to upgrade from a Flink session cluster 1.8 to
1.9 and from Beam 2.16.0 to 2.19.0.
Everything went quite smoothly, the local runner and the
local Flink runner work flawlessly.
However when I:
1. Generate a Beam jar for the FlinkRunner via maven (mvn
package -PFlinkRunner)
2. Glue that into a Flink 1.9 docker image
3. Start the image as a Standalone Session Cluster
When I try to launch the first pipeline I get the following
exception
org.apache.flink.client.program.ProgramInvocationException:
The main method caused an error: Failed to construct
instance from factory method
FlinkRunner#fromOptions(interface
org.apache.beam.sdk.options.PipelineOptions)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
at
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
at
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
at
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
at
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
at
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
Caused by: java.lang.RuntimeException: Failed to construct
instance from factory method
FlinkRunner#fromOptions(interface
org.apache.beam.sdk.options.PipelineOptions)
at
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
at
org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
at
org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
at
ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
... 9 more
Caused by: java.lang.reflect.InvocationTargetException
at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
... 19 more
Caused by:
org.apache.flink.api.common.InvalidProgramException:
Multiple environments cannot be created in detached mode
at
org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
at java.util.Optional.map(Optional.java:215)
at
org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
at
org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
at
org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
at
org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
... 24 more
I've checked the release notes and the issues and couldn't
find anything that relates to this. What puzzles me is that
the session cluster should be allowed to have multiple
environments in detached mode - or am I wrong?
Best,
Tobi
--
Tobias Kaymak
Data Engineer
Data Intelligence
[email protected] <mailto:[email protected]>
www.ricardo.ch <http://www.ricardo.ch/>
Theilerstrasse 1a, 6300 Zug
