Hi Sujay,
can you please provide more information about how you run the job?
FlinkRunner is definitely compiled against Flink 1.12.2, but that should
not be an issue. FlinkRunner contains flink-runtime, you should be fine
simply excluding it and replacing with the version 1.12.7 (maybe better
behavior here would be not to include the Flink dependencies into
FlinkRunner and let the user explicitly provide them). Additionally,
Flink classes are usually loaded using "parent-first" classloader [1],
which should mean, that any classes in org.apache.flink.* should be
loaded from the server classpath. You can verify if you haven't
accidentally changed the default and that your JM and TM are all at the
same versions.
Best,
Jan
[1]
https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#classloader-parent-first-patterns-default
On 1/18/22 07:35, sujay kulkarni wrote:
Hi,
I just dug into the code where this issue is getting highlighted, I
observed 2 different implementations of the class are getting loaded
in classpath.
Beam-flink runner is bringing the 1.12.2 version of the flink_clients
which has this class definition. And at runtime Flink Server is
getting a 1.12.7 version of the same class definition.
1.12.2 method implementation coming from beam-flink-runner
private Builder(String[] programArguments) {
this.programArguments =
Objects.<String[]>requireNonNull(programArguments);
}
1.12.7 method implementation present in FlinkServer
private Builder(String[] programArguments, Configuration
configuration) {
this.programArguments =
Objects.<String[]>requireNonNull(programArguments);
this.configuration =
Objects.<Configuration>requireNonNull(configuration);
}
But this should not happen right as Beam flinkRunner in version 2.29.0
supports 1.12.x of Flink version. Is it bug or am I missing something?
If it's a bug any jira is raised against this issue.?
References:
https://mvnrepository.com/artifact/org.apache.beam/beam-runners-flink-1.12/2.29.0
https://beam.apache.org/documentation/runners/flink/#flink-version-compatibility
Thanks & Regards,
Sujay
On Mon, Jan 17, 2022 at 11:58 PM Luke Cwik <[email protected]> wrote:
+user <mailto:[email protected]> -dev
This looks like you have a dependency conflict issue. Is that the
whole stack trace or is there more?
Also, in my experience Java only emits the full failure reason the
first time this dependency issue exists. Future failures are
effectively cached and any suppressed exceptions and other
failures are typically no longer available to help debug this
issue so finding the first failure can provide more detail.
Finally, https://jlbp.dev/ has great information about Java
dependency management best practices. It will only help slightly
with your immediate problem as it can explain concepts and why
things fail this way but it also provides lots of helpful
information to improve dependency management in general within
your projects.
On Mon, Jan 17, 2022 at 1:44 AM sujay kulkarni
<[email protected]> wrote:
Hi all,
I am trying a sample beam job using flinkRunner.
*Beam-version -- 2.29.0*
*Flink server -- 1.12.7*
When running the job, I am getting below issue in the
jobmanager logs
*Exception in thread "main" java.lang.NoSuchMethodError:
'org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever$Builder*
*org.apache.flink.client.deployment.application.ClassPathPackagedProgramRetriever.newBuilder(java.lang.String[],
org.apache.flink.configuration.Configuration)'*
* at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgramRetriever(StandaloneApplicationClusterEntryPoint.java:117)*
* at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.getPackagedProgram(StandaloneApplicationClusterEntryPoint.java:103)*
* at
org.apache.flink.container.entrypoint.StandaloneApplicationClusterEntryPoint.main(StandaloneApplicationClusterEntryPoint.java:69)*
*
*
Seems like the issue at classpath. I checked the compatibility
matrix of beam flink runner and in version 2.29.0, Flink
1.12.x is supported,
Please help me with the issue I am facing.
Thanks and Regards,
Sujay
