I can confirm that the pipeline behaves as expected with 2.20.0-SNAPSHOT
and Flink 1.9.1 - I also tried Flink 1.9.2 but the webinterface didn't show
up (just a blank page - javascript was being loaded though).
I emptied my cache and investigated the log and asked on the Flink mailing
list if this is known - maybe it's also because of one of the
dependencies in my fat Beam jar. I am still investigating this.
How can I test the Flink 1.10 runners? (The following POM is not resolvable
by maven)
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-flink-1.10</artifactId>
<version>2.20-SNAPSHOT</version>
</dependency>
Best,
Tobi
On Wed, Feb 26, 2020 at 5:07 PM Ismaël Mejía <[email protected]> wrote:
> Since it was merged yesterday you can test with the 2.20.0-SNAPSHOT until
> the first candidate is out.
>
> On Wed, Feb 26, 2020 at 4:37 PM Kaymak, Tobias <[email protected]>
> wrote:
>
>> If I am not running in detached mode (so that my pipeline starts) I am
>> unable to Stop it in the webinterface. The only option available is to
>> cancel it. Is this expected?
>>
>> [image: Screenshot 2020-02-26 at 16.34.08.png]
>>
>> On Wed, Feb 26, 2020 at 4:16 PM Kaymak, Tobias <[email protected]>
>> wrote:
>>
>>> Hello,
>>>
>>> we fixed the issue and are ready to test :) - is there a RC already
>>> available?
>>>
>>> Best,
>>> Tobi
>>>
>>> On Wed, Feb 26, 2020 at 12:59 PM Kaymak, Tobias <
>>> [email protected]> wrote:
>>>
>>>> Hello,
>>>>
>>>> happy to help testing! I am currently fixing a networking issue between
>>>> our dev cluster for integration tests and the Kafka it is consuming from.
>>>> After that I would be ready to spin it up and test
>>>>
>>>> Best,
>>>> Tobi
>>>>
>>>> On Mon, Feb 24, 2020 at 10:13 PM Maximilian Michels <[email protected]>
>>>> wrote:
>>>>
>>>>> Thank you for reporting / filing / collecting the issues.
>>>>>
>>>>> There is a fix pending: https://github.com/apache/beam/pull/10950
>>>>>
>>>>> As for the upgrade issues, the 1.8 and 1.9 upgrade is trivial. I will
>>>>> check out the Flink 1.10 PR tomorrow.
>>>>>
>>>>> Cheers,
>>>>> Max
>>>>>
>>>>> On 24.02.20 09:26, Ismaël Mejía wrote:
>>>>> > We are cutting the release branch for 2.20.0 next wednesday, so not
>>>>> sure
>>>>> > if these tickets will make it, but hopefully.
>>>>> >
>>>>> > For ref,
>>>>> > BEAM-9295 Add Flink 1.10 build target and Make FlinkRunner
>>>>> compatible
>>>>> > with Flink 1.10
>>>>> > BEAM-9299 Upgrade Flink Runner to 1.8.3 and 1.9.2
>>>>> >
>>>>> > In any case if you have cycles to help test any of the related
>>>>> tickets
>>>>> > PRs that would help too.
>>>>> >
>>>>> >
>>>>> > On Mon, Feb 24, 2020 at 8:47 AM Kaymak, Tobias <
>>>>> [email protected]
>>>>> > <mailto:[email protected]>> wrote:
>>>>> >
>>>>> > Hi Kyle,
>>>>> >
>>>>> > thank you for creating the JIRA ticket, I think my best option
>>>>> right
>>>>> > now is to wait for a Beam version that is running on Flink 1.10
>>>>> then
>>>>> > - unless there is a new Beam release around the corner :)
>>>>> >
>>>>> > Best,
>>>>> > Tobi
>>>>> >
>>>>> > On Thu, Feb 20, 2020 at 11:52 PM Kyle Weaver <
>>>>> [email protected]
>>>>> > <mailto:[email protected]>> wrote:
>>>>> >
>>>>> > Hi Tobi,
>>>>> >
>>>>> > This seems like a bug with Beam 2.19. I filed
>>>>> > https://issues.apache.org/jira/browse/BEAM-9345 to track
>>>>> the issue.
>>>>> >
>>>>> > > What puzzles me is that the session cluster should be
>>>>> allowed
>>>>> > to have multiple environments in detached mode - or am I
>>>>> wrong?
>>>>> >
>>>>> > It looks like that check is removed in Flink 1.10:
>>>>> > https://issues.apache.org/jira/browse/FLINK-15201
>>>>> >
>>>>> > Thanks for reporting.
>>>>> > Kyle
>>>>> >
>>>>> > On Thu, Feb 20, 2020 at 4:10 AM Kaymak, Tobias
>>>>> > <[email protected] <mailto:[email protected]>>
>>>>> wrote:
>>>>> >
>>>>> > Hello,
>>>>> >
>>>>> > I am trying to upgrade from a Flink session cluster 1.8
>>>>> to
>>>>> > 1.9 and from Beam 2.16.0 to 2.19.0.
>>>>> > Everything went quite smoothly, the local runner and the
>>>>> > local Flink runner work flawlessly.
>>>>> >
>>>>> > However when I:
>>>>> > 1. Generate a Beam jar for the FlinkRunner via maven
>>>>> (mvn
>>>>> > package -PFlinkRunner)
>>>>> > 2. Glue that into a Flink 1.9 docker image
>>>>> > 3. Start the image as a Standalone Session Cluster
>>>>> >
>>>>> > When I try to launch the first pipeline I get the
>>>>> following
>>>>> > exception
>>>>> >
>>>>> >
>>>>> org.apache.flink.client.program.ProgramInvocationException:
>>>>> > The main method caused an error: Failed to construct
>>>>> > instance from factory method
>>>>> > FlinkRunner#fromOptions(interface
>>>>> > org.apache.beam.sdk.options.PipelineOptions)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:593)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:438)
>>>>> > at
>>>>> >
>>>>> org.apache.flink.client.program.ClusterClient.run(ClusterClient.java:274)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:746)
>>>>> > at
>>>>> >
>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:273)
>>>>> > at
>>>>> >
>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:205)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1010)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1083)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
>>>>> > at
>>>>> >
>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1083)
>>>>> > Caused by: java.lang.RuntimeException: Failed to
>>>>> construct
>>>>> > instance from factory method
>>>>> > FlinkRunner#fromOptions(interface
>>>>> > org.apache.beam.sdk.options.PipelineOptions)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:224)
>>>>> > at
>>>>> >
>>>>> org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:155)
>>>>> > at
>>>>> >
>>>>> org.apache.beam.sdk.PipelineRunner.fromOptions(PipelineRunner.java:55)
>>>>> > at
>>>>> org.apache.beam.sdk.Pipeline.run(Pipeline.java:309)
>>>>> > at
>>>>> org.apache.beam.sdk.Pipeline.run(Pipeline.java:301)
>>>>> > at
>>>>> >
>>>>> ch.ricardo.di.beam.KafkaToBigQuery.main(KafkaToBigQuery.java:180)
>>>>> > at
>>>>> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> > at
>>>>> >
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> > at
>>>>> >
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> > at
>>>>> java.lang.reflect.Method.invoke(Method.java:498)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:576)
>>>>> > ... 9 more
>>>>> > Caused by: java.lang.reflect.InvocationTargetException
>>>>> > at
>>>>> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>>> Method)
>>>>> > at
>>>>> >
>>>>>
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>>> > at
>>>>> >
>>>>>
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>>> > at
>>>>> java.lang.reflect.Method.invoke(Method.java:498)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:214)
>>>>> > ... 19 more
>>>>> > Caused by:
>>>>> > org.apache.flink.api.common.InvalidProgramException:
>>>>> > Multiple environments cannot be created in detached mode
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.client.program.ContextEnvironmentFactory.createExecutionEnvironment(ContextEnvironmentFactory.java:67)
>>>>> > at java.util.Optional.map(Optional.java:215)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.flink.api.java.ExecutionEnvironment.getExecutionEnvironment(ExecutionEnvironment.java:1068)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.beam.runners.flink.translation.utils.Workarounds.restoreOriginalStdOutAndStdErrIfApplicable(Workarounds.java:43)
>>>>> > at
>>>>> >
>>>>> org.apache.beam.runners.flink.FlinkRunner.<init>(FlinkRunner.java:96)
>>>>> > at
>>>>> >
>>>>>
>>>>> org.apache.beam.runners.flink.FlinkRunner.fromOptions(FlinkRunner.java:90)
>>>>> > ... 24 more
>>>>> >
>>>>> > I've checked the release notes and the issues and
>>>>> couldn't
>>>>> > find anything that relates to this. What puzzles me is
>>>>> that
>>>>> > the session cluster should be allowed to have multiple
>>>>> > environments in detached mode - or am I wrong?
>>>>> >
>>>>> > Best,
>>>>> > Tobi
>>>>> >
>>>>> >
>>>>> >
>>>>> > --
>>>>> >
>>>>> > Tobias Kaymak
>>>>> > Data Engineer
>>>>> > Data Intelligence
>>>>> >
>>>>> > [email protected] <mailto:[email protected]>
>>>>> > www.ricardo.ch <http://www.ricardo.ch/>
>>>>> > Theilerstrasse 1a, 6300 Zug
>>>>> >
>>>>
>>>>
