Hello all,
I have some problems when attempting to run wordcount. For some reason, when
attempting to run it, it cannot find the classes.
Looking in the wayang-submit file, it seems to set the WAYANG_CLASSPATH to
/jars and /libs, but it will not find it when ran.
[cid:6671470c-6b1f-4e09-a6c4-13685266f2e1]
I have tried a couple things:
1.
When running it with a manually specified classpath it does find the class.
Though, then it cannot find the next one, and so on. (I added about 6 paths
manually before i stopped trying this)
ex:
./bin/wayang-submit -cp
/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-benchmark-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-core-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/log4j-core-2.17.0.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/log4j-api-2.17.0.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/commons-io-2.7.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/jars/wayang-java-0.7.1.jar:/c/Users/jeppe/researchProject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/libs/commons-lang3-3.12.0.jar
org.apache.wayang.apps.wordcount.Main java file://$(pwd)/README.md
2.
I attempted to run with all .jars in the /libs and /jars, but the classpath
becomes too long, so java wouldnt run it. (using jars/* and libs/*)
*
To solve this I attempted to load all the classpaths into a classpath.txt file
and formatted it, though now it would not find any classpaths.
export HADOOP_HOME="/c/Users/jeppe/researchProject/hadoop-3.3.6"
export JAVA_HOME="/c/Java/jdk-11"
export SPARK_HOME="/c/Users/jeppe/researchProject/spark-3.5.2-bin-hadoop3"
export MAVEN_HOME="/c/Users/jeppe/researchProject/apache-maven-3.9.9-bin"
export
WAYANG_HOME="/c/Users/jeppe/researchproject/incubator-wayang/wayang-assembly/target/wayang-0.7.1"
export
PATH="/c/Users/jeppe/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/jeppe/bin:/c/Program
Files/Common Files/Oracle/Java/javapath:/c/Program Files/Eclipse
Adoptium/jdk-11.0.16.101-hotspot/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/cmd:/c/Gradle/gradle-8.4/bin:/c/Program
Files/dotnet:/c/Program Files/PostgreSQL/16/bin:/c/Program
Files/PowerShell/7:/c/Users/jeppe/OneDrive/Skrivebord/researchProject/apache-maven-3.9.9-bin/apache-maven-3.9.9/bin:/c/Users/jeppe/researchProject/hadoop-3.3.6/bin:/c/Java/jdk-11/bin:/c/Users/jeppe/researchProject/apache-maven-3.9.9-bin/bin:/c/Users/jeppe/researchProject/spark-3.5.2-bin-hadoop3/bin:/c/Users/jeppe/AppData/Local/Programs/Python/Launcher:/c/Users/jeppe/AppData/Roaming/TinyTeX/bin/win32:/c/Users/jeppe/Downloads/gradle-8.4-bin/gradle-8.4/bin:/c/Users/jeppe/AppData/Local/Programs/Microsoft
VS
Code/bin:/c/Users/jeppe/AppData/Local/Microsoft/WindowsApps:/c/Users/jeppe/.dotnet/tools:/c/Users/jeppe/AppData/Local/Coursier/data/bin:/usr/bin/vendor_perl:/usr/bin/core_perl:/c/Users/jeppe/researchproject/incubator-wayang/wayang-assembly/target/wayang-0.7.1/bin"
One weird thing though is when running the submit with only -cp /jars*
[cid:9fedee50-b746-4963-a9b4-fedc3eba3ba3]
it still has issues finding the api-python-0.7.1.jar file even though it lies
in the /jars folder.
Best regards,
Jeppe Halvorsen
