Hello, I'm trying to run the WordCount example using SparkRunner.
In the client-mode, the child jobs are kicked off and the output is also produced. However in the cluster mode, the child jobs are not starting and no output is produced. I'm using Beam 2.9.0 and Spark 2.3.0 Here are the commands that I ran:- Cluster-mode (No output produced) spark2-submit --conf spark.driver.userClassPathFirst=true --class org.wordcount.WordCountSpark --master yarn --queue test --deploy-mode cluster wordcount-0.1.0-SNAPSHOT-shaded.jar --runner=SparkRunner Client-mode (Output produced) spark2-submit --conf spark.driver.userClassPathFirst=true --class org.wordcount.WordCountSpark --master yarn --queue test --deploy-mode client wordcount-0.1.0-SNAPSHOT-shaded.jar --runner=SparkRunner Here's the code https://gist.github.com/shrijitpillai/852a3c737c922af9e9ba4cd4c53b8530 Here's the mvn dependency tree:- https://gist.github.com/shrijitpillai/f06797238cf6db9bf350e6a835fb34d6 Cluster mode yarn log https://gist.github.com/shrijitpillai/2f1963ece26c152a2d68bc6ae8913fea Can somebody help in identifying what is that I'm missing here? Thanks, Shrijit Pillai
