Hi, Was anyone able to run Beam application on Spark at all??
I tried all possible options and still no luck. No executors getting assigned for the job submitted by below command even though explicitly specified, $ ~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --master yarn --executor-memory 2G --num-executors 2 target/word-count-beam-0.1-shaded.jar --runner=SparkRunner --inputFile=pom.xml --output=counts Please someone help or point me to the right forum. Attached job log from YARN. Job is stuck at 'INFO spark.SparkRunner$Evaluator: Evaluating ParMultiDo(ExtractWords)’. Regards, Sathish. J On 01-Aug-2017, at 4:16 PM, Sathish Jayaraman <[email protected]<mailto:[email protected]>> wrote: Hi JB, Not when I submit is using the option —master local. But when I submit with with spark master’s port, The job executes and hangs at step 'Registering block manager 192.168.0.2:58956 with 366.3 MB RAM, BlockManagerId(0, 192.168.0.2, 58956, None)’. Attached screenshots of the HistoryUI & Dashboard. Also raised a question in Stackoverflow<https://stackoverflow.com/questions/45435238/apache-beam-job-hangs-up-when-submitted-via-spark-submit> too. Command: ~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --master spark://Sathish-MacBook-Pro.local:7077 target/word-count-beam-0.1.jar --runner=SparkRunner --inputFile=pom.xml --output=counts <log2.txt> <Screen Shot 2017-08-01 at 4.10.17 PM.png> <Screen Shot 2017-08-01 at 4.08.18 PM.png> Regards, Sathish. J On 01-Aug-2017, at 4:02 PM, Jean-Baptiste Onofré <[email protected]<mailto:[email protected]>> wrote: Hi Sathish, Do you see the tasks submitted on the history server ? Regards JB On 08/01/2017 11:51 AM, Sathish Jayaraman wrote: Hi, I am trying to execute Beam example in local spark setup. When I try to submit the sample WordCount jar via spark-submit, the job just hangs at 'INFO SparkRunner$Evaluator: Evaluating ParMultiDo(ExtractWords)’. But it runs fine when executed directly. Below is the command I used to submit the job in spark local, $ ~/spark/bin/spark-submit --class "org.apache.beam.examples.WordCount" --master local[4] target/word-count-beam-0.1.jar —inputFile=./pom.xml --output=csvout --runner=SparkRunner Have attached log file for reference. Can anyone please help me find out whats going on? Regards, Sathish. J -- Jean-Baptiste Onofré [email protected]<mailto:[email protected]> http://blog.nanthrax.net Talend - http://www.talend.com
17/08/03 13:00:33 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030 17/08/03 13:00:33 INFO yarn.YarnRMClient: Registering the ApplicationMaster 17/08/03 13:00:34 INFO yarn.YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 2432 MB memory (including 384 MB of overhead) 17/08/03 13:00:34 INFO yarn.YarnAllocator: Submitted 2 unlocalized container requests. 17/08/03 13:00:34 INFO yarn.ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 17/08/03 13:00:35 INFO impl.AMRMClientImpl: Received new token for : 192.168.0.7:50173 17/08/03 13:00:35 INFO yarn.YarnAllocator: Launching container container_1501744514957_0003_01_000002 on host 192.168.0.7 17/08/03 13:00:35 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/08/03 13:00:35 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/08/03 13:00:35 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 192.168.0.7:50173 17/08/03 13:00:37 INFO yarn.YarnAllocator: Launching container container_1501744514957_0003_01_000003 on host 192.168.0.7 17/08/03 13:00:37 INFO yarn.YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/08/03 13:00:37 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/08/03 13:00:37 INFO impl.ContainerManagementProtocolProxy: Opening proxy : 192.168.0.7:50173
