Hi Jeppe, Just for my understanding, what is the motivation for specifying all jars individually with the -cp flag? Also, can you tell me in which directory you execute the wayang-submit? There is one in wayang-assembly/target/wayang-0.7.1 (after assembly has been built – USE THIS ONE) and one in /bin/wayang-submit (only needed for the build process – DO NOT USE THIS ONE TO RUN STUFF).
Best, Juri From: Jeppe Nordvi Halvorsen <h...@itu.dk.INVALID> Date: Sunday, 15 September 2024 at 16.15 To: dev@wayang.apache.org <dev@wayang.apache.org> Cc: Christoffer Emil Kristensen <c...@itu.dk> Subject: Problems with classpath when attempting to run wordcount You don't often get email from h...@itu.dk.invalid. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hello all, I have some problems when attempting to run wordcount. For some reason, when attempting to run it, it cannot find the classes. Looking in the wayang-submit file, it seems to set the WAYANG_CLASSPATH to /jars and /libs, but it will not find it when ran. [cid:6671470c-6b1f-4e09-a6c4-13685266f2e1] I have tried a couple things: 1. When running it with a manually specified classpath it does find the class. Though, then it cannot find the next one, and so on. (I added about 6 paths manually before i stopped trying this) ex: ./bin/wayang-submit -cp /c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-benchmark-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-core-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/log4j-core-2.17.0.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/log4j-api-2.17.0.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/commons-io-2.7.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-java-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/commons-lang3-3.12.0.jar org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md 1. I attempted to run with all .jars in the /libs and /jars, but the classpath becomes too long, so java wouldnt run it. (using jars/* and libs/*) * To solve this I attempted to load all the classpaths into a classpath.txt file and formatted it, though now it would not find any classpaths. export HADOOP_HOME="/c/Users/jeppe/researchProject/hadoop-3.3.6" export JAVA_HOME="/c/Java/jdk-11" export SPARK_HOME="/c/Users/jeppe/researchProject/spark-3.5.2-bin-hadoop3" export MAVEN_HOME="/c/Users/jeppe/researchProject/apache-maven-3.9.9-bin" export WAYANG_HOME="/c/Users/jeppe/researchproject/incubator-wayang/wayang-assembly/target/wayang-0.7.1" export PATH="/c/Users/jeppe/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/jeppe/bin:/c/Program Files/Common Files/Oracle/Java/javapath:/c/Program Files/Eclipse Adoptium/jdk-11.0.16.101-hotspot/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/cmd:/c/Gradle/gradle-8.4/bin:/c/Program Files/dotnet:/c/Program Files/PostgreSQL/16/bin:/c/Program Files/PowerShell/7:/c/Users/jeppe/OneDrive/Skrivebord/researchProject/apache-maven-3.9.9-bin/apache-maven-3.9.9/bin:/c/Users/jeppe/researchProject/hadoop-3.3.6/bin:/c/Java/jdk-11/bin:/c/Users/jeppe/researchProject/apache-maven-3.9.9-bin/bin:/c/Users/jeppe/researchProject/spark-3.5.2-bin-hadoop3/bin:/c/Users/jeppe/AppData/Local/Programs/Python/Launcher:/c/Users/jeppe/AppData/Roaming/TinyTeX/bin/win32:/c/Users/jeppe/Downloads/gradle-8.4-bin/gradle-8.4/bin:/c/Users/jeppe/AppData/Local/Programs/Microsoft VS Code/bin:/c/Users/jeppe/AppData/Local/Microsoft/WindowsApps:/c/Users/jeppe/.dotnet/tools:/c/Users/jeppe/AppData/Local/Coursier/data/bin:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/Users/jeppe/researchproject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/bin" One weird thing though is when running the submit with only -cp /jars* [cid:9fedee50-b746-4963-a9b4-fedc3eba3ba3] it still has issues finding the api-python-0.7.1.jar file even though it lies in the /jars folder. Best regards, Jeppe Halvorsen
