Hi,
yes, but not with Yarn.
Let me bootstrap a quick cluster with Yarn to test it.
Regards
JB
On 08/03/2017 09:34 AM, Sathish Jayaraman wrote:
Hi,
Was anyone able to run Beam application on Spark at all??
I tried all possible options and still no luck. No executors getting assigned
for the job submitted by below command even though explicitly specified,
$ ~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --master
yarn --executor-memory 2G --num-executors 2
target/word-count-beam-0.1-shaded.jar --runner=SparkRunner --inputFile=pom.xml
--output=counts
Please someone help or point me to the right forum.
Attached job log from YARN. Job is stuck at 'INFO spark.SparkRunner$Evaluator:
Evaluating ParMultiDo(ExtractWords)’.
Regards,
Sathish. J
On 01-Aug-2017, at 4:16 PM, Sathish Jayaraman <[email protected]
<mailto:[email protected]>> wrote:
Hi JB,
Not when I submit is using the option —master local. But when I submit with
with spark master’s port, The job executes and hangs at step 'Registering
block manager 192.168.0.2:58956 with 366.3 MB RAM, BlockManagerId(0,
192.168.0.2, 58956, None)’. Attached screenshots of the HistoryUI & Dashboard.
Also raised a question in Stackoverflow
<https://stackoverflow.com/questions/45435238/apache-beam-job-hangs-up-when-submitted-via-spark-submit> too.
Command:
~/spark/bin/spark-submit --class org.apache.beam.examples.WordCount --master
spark://Sathish-MacBook-Pro.local:7077 target/word-count-beam-0.1.jar
--runner=SparkRunner --inputFile=pom.xml --output=counts
<log2.txt>
<Screen Shot 2017-08-01 at 4.10.17 PM.png>
<Screen Shot 2017-08-01 at 4.08.18 PM.png>
Regards,
Sathish. J
On 01-Aug-2017, at 4:02 PM, Jean-Baptiste Onofré <[email protected]
<mailto:[email protected]>> wrote:
Hi Sathish,
Do you see the tasks submitted on the history server ?
Regards
JB
On 08/01/2017 11:51 AM, Sathish Jayaraman wrote:
Hi,
I am trying to execute Beam example in local spark setup. When I try to
submit the sample WordCount jar via spark-submit, the job just hangs at
'INFO SparkRunner$Evaluator: Evaluating ParMultiDo(ExtractWords)’. But it
runs fine when executed directly. Below is the command I used to submit the
job in spark local,
$ ~/spark/bin/spark-submit --class "org.apache.beam.examples.WordCount"
--master local[4] target/word-count-beam-0.1.jar —inputFile=./pom.xml
--output=csvout --runner=SparkRunner
Have attached log file for reference. Can anyone please help me find out
whats going on?
Regards,
Sathish. J
--
Jean-Baptiste Onofré
[email protected] <mailto:[email protected]>
http://blog.nanthrax.net
Talend - http://www.talend.com
--
Jean-Baptiste Onofré
[email protected]
http://blog.nanthrax.net
Talend - http://www.talend.com
