Hi, When I execute a PyFlink-Job locally in a mini cluster, then everything works fine:
(env)
user@bla /cygdrive/c/flink1_13_3/examples/python/table/batch
$ python word_count.py
Results directory: C:\cygwin64\tmp/result
But when I try to execute the PyFlink-Job in a remote cluster, the job
execution fails:
user@bla /cygdrive/c/flink1_13_3/examples/python/table/batch
$ flink run -pyarch env.zip -pyexec env.zip/env/Scripts/python -py word_count.py
Although I set the environment variable (FLINK_HOME = C:\flink1_13_3), Flink is
trying to fetch the flink-python_2.11-1.13.3.jar from the wrong directory
(\cygdrive\c\flink1_13_3\opt).
The stacktrace in the logfiles looks as follow:
java.lang.RuntimeException: Exception encountered during finding the
flink-python jar. This should not happen.
at
org.apache.flink.client.program.PackagedProgramUtils.getPythonJar(PackagedProgramUtils.java:222)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.ProgramOptionsUtils.getPythonClassLoader(ProgramOptionsUtils.java:98)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.ProgramOptionsUtils.createPythonProgramOptions(ProgramOptionsUtils.java:76)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.ProgramOptions.create(ProgramOptions.java:186)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:236)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
[flink-dist_2.11-1.13.3.jar:1.13.3]
Caused by: java.nio.file.NoSuchFileException: \cygdrive\c\flink1_13_3\opt
at
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
~[?:1.8.0_302]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
~[?:1.8.0_302]
at
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
~[?:1.8.0_302]
at
sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:53)
~[?:1.8.0_302]
at
sun.nio.fs.WindowsFileAttributeViews$Basic.readAttributes(WindowsFileAttributeViews.java:38)
~[?:1.8.0_302]
at
sun.nio.fs.WindowsFileSystemProvider.readAttributes(WindowsFileSystemProvider.java:193)
~[?:1.8.0_302]
at java.nio.file.Files.readAttributes(Files.java:1737)
~[?:1.8.0_302]
at
java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219)
~[?:1.8.0_302]
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
~[?:1.8.0_302]
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322)
~[?:1.8.0_302]
at java.nio.file.Files.walkFileTree(Files.java:2662)
~[?:1.8.0_302]
at java.nio.file.Files.walkFileTree(Files.java:2742)
~[?:1.8.0_302]
at
org.apache.flink.client.program.PackagedProgramUtils.getPythonJar(PackagedProgramUtils.java:208)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
... 8 more
2021-10-28 09:20:35,663 ERROR org.apache.flink.client.cli.CliFrontend
[] - Invalid command line arguments.
org.apache.flink.client.cli.CliArgsException: Python command line option
detected but the flink-python module seems to be missing or not working as
expected.
at
org.apache.flink.client.cli.ProgramOptionsUtils.createPythonProgramOptions(ProgramOptionsUtils.java:88)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.ProgramOptions.create(ProgramOptions.java:186)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:236)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1054)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:1132)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:28)
[flink-dist_2.11-1.13.3.jar:1.13.3]
at
org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1132)
[flink-dist_2.11-1.13.3.jar:1.13.3]
Caused by: java.lang.ClassNotFoundException:
org.apache.flink.client.cli.PythonProgramOptions
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
~[?:1.8.0_302]
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
~[?:1.8.0_302]
at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) ~[?:1.8.0_302]
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
~[?:1.8.0_302]
at java.lang.Class.forName0(Native Method) ~[?:1.8.0_302]
at java.lang.Class.forName(Class.java:348) ~[?:1.8.0_302]
at
org.apache.flink.client.cli.ProgramOptionsUtils.createPythonProgramOptions(ProgramOptionsUtils.java:78)
~[flink-dist_2.11-1.13.3.jar:1.13.3]
... 6 more
How can I fix this?
I'm using Flink Version 1.13.3 (Windows – Cygwin)
Any help would be highly appreciated.
Diese E-Mail ist ausschliesslich für den Adressaten bestimmt. Sollten Sie diese
E-Mail irrtümlich erhalten haben oder wünschen Sie künftig keine Kontakte mehr
per E-Mail, bitten wir Sie, die Bank Cler hierüber sofort zu orientieren. Die
irrtümlich erhaltene E-Mail ist mit allen Anhängen unwiderruflich zu löschen,
allfällige Ausdrucke sind zu vernichten und auf die Verwendung des Inhalts ist
zu verzichten. Der Versand unverschlüsselter E-Mail birgt erhebliche Risiken in
sich (mangelnde Vertraulichkeit, Manipulation von Inhalt/Absender, Fehlleitung,
Viren etc.). Bank Cler lehnt jede Haftung für Schäden hieraus ab. Bank Cler
akzeptiert grundsätzlich keine per E-Mail übermittelten Aufträge, Widerrufe von
Aufträgen oder sonstige Weisungen etc., ohne verpflichtet zu sein, diese
ausdrücklich zurück zu weisen. Kündigungen von Verträgen per E-Mail sind nicht
rechtswirksam.
smime.p7s
Description: S/MIME cryptographic signature
