[ 
https://issues.apache.org/jira/browse/BEAM-9345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kyle Weaver updated BEAM-9345:
------------------------------
    Description: 
Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when 
running in standalone session cluster. Since FLINK-15504 is resolved in Flink 
1.10, maybe we can remove the workaround in Beam?

>From user@ 
>(https://lists.apache.org/thread.html/r1fb9456055ae51f998be67531f0ac55d3da9cf2647238b991eee7f97%40%3Cuser.beam.apache.org%3E):

I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 
2.16.0 to 2.19.0. 
Everything went quite smoothly, the local runner and the local Flink runner 
work flawlessly.

However when I:
  1. Generate a Beam jar for the FlinkRunner via maven (mvn package 
-PFlinkRunner)
  2. Glue that into a Flink 1.9 docker image
  3. Start the image as a Standalone Session Cluster

When I try to launch the first pipeline I get the following exception

org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: Failed to construct instance from factory method 
FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
        at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
        at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
        at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
        at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
        at 
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
        at 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
        at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
        at 
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
Caused by: java.lang.RuntimeException: Failed to construct instance from 
factory method FlinkRunner#fromOptions(interface 
org.apache.beam.sdk.options.PipelineOptions)
        at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
        at 
org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
        at 
org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
        at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
        at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
        at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
        ... 19 more
Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple 
environments cannot be created in detached mode
        at 
org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
        at java.util.Optional.map(Optional.java:215)
        at 
org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
        at 
org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
        at org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
        at 
org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
        ... 24 more 


  was:
Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when 
running in standalone session cluster. Since FLINK-15504 is resolved in Flink 
1.10, maybe we can remove the workaround in Beam?

>From user@:

I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 
2.16.0 to 2.19.0. 
Everything went quite smoothly, the local runner and the local Flink runner 
work flawlessly.

However when I:
  1. Generate a Beam jar for the FlinkRunner via maven (mvn package 
-PFlinkRunner)
  2. Glue that into a Flink 1.9 docker image
  3. Start the image as a Standalone Session Cluster

When I try to launch the first pipeline I get the following exception

org.apache.flink.client.program.ProgramInvocationException: The main method 
caused an error: Failed to construct instance from factory method 
FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
        at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
        at 
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
        at 
org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
        at 
org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
        at 
org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
        at 
org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
        at 
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
        at 
org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
Caused by: java.lang.RuntimeException: Failed to construct instance from 
factory method FlinkRunner#fromOptions(interface 
org.apache.beam.sdk.options.PipelineOptions)
        at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
        at 
org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
        at 
org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
        at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
        at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
        at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
        ... 9 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
        ... 19 more
Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple 
environments cannot be created in detached mode
        at 
org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
        at java.util.Optional.map(Optional.java:215)
        at 
org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
        at 
org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
        at org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
        at 
org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
        ... 24 more 



> "Multiple environments cannot be created in detached mode"
> ----------------------------------------------------------
>
>                 Key: BEAM-9345
>                 URL: https://issues.apache.org/jira/browse/BEAM-9345
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-flink
>            Reporter: Kyle Weaver
>            Assignee: Maximilian Michels
>            Priority: Major
>
> Workarounds.restoreOriginalStdOutAndStdErrIfApplicable throws exception when 
> running in standalone session cluster. Since FLINK-15504 is resolved in Flink 
> 1.10, maybe we can remove the workaround in Beam?
> From user@ 
> (https://lists.apache.org/thread.html/r1fb9456055ae51f998be67531f0ac55d3da9cf2647238b991eee7f97%40%3Cuser.beam.apache.org%3E):
> I am trying to upgrade from a Flink session cluster 1.8 to 1.9 and from Beam 
> 2.16.0 to 2.19.0. 
> Everything went quite smoothly, the local runner and the local Flink runner 
> work flawlessly.
> However when I:
>   1. Generate a Beam jar for the FlinkRunner via maven (mvn package 
> -PFlinkRunner)
>   2. Glue that into a Flink 1.9 docker image
>   3. Start the image as a Standalone Session Cluster
> When I try to launch the first pipeline I get the following exception
> org.apache.flink.client.program.ProgramInvocationException: The main method 
> caused an error: Failed to construct instance from factory method 
> FlinkRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
>         at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>         at 
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>         at 
> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>         at 
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>         at 
> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>         at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>         at 
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>         at 
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>         at 
> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>         at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
> Caused by: java.lang.RuntimeException: Failed to construct instance from 
> factory method FlinkRunner#fromOptions(interface 
> org.apache.beam.sdk.options.PipelineOptions)
>         at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
>         at 
> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
>         at 
> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
>         at org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
>         at org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
>         at ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>         ... 9 more
> Caused by: java.lang.reflect.InvocationTargetException
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at 
> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
>         ... 19 more
> Caused by: org.apache.flink.api.common.InvalidProgramException: Multiple 
> environments cannot be created in detached mode
>         at 
> org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
>         at java.util.Optional.map(Optional.java:215)
>         at 
> org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
>         at 
> org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
>         at 
> org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
>         at 
> org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
>         ... 24 more 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to