Hi, @Brown. The error looks like you need to pass the flag `--master mesos://leader.mesos:5050` to spark-submit.
On Thu, Jun 16, 2016 at 6:57 PM, Rodrick Brown <[email protected]> wrote: > We use Chronos extensively for running our batch based spark jobs and > recently started using sparkr however I'm seeing an issue trying to get > sparkr jobs to run successfully when launched through Chronos basically > always failing with a generic error below > > Launching java with spark-submit command /opt/spark-1.6.1/bin/spark-submit > --jars > /data/orchard/etc/config/load-tradedata-accumulo-prod.jar,/data/orchard/jars/dataloader-library-626d9b10bef28a163ad61d8a2e87bb71d455ea34-assembled.jar > --driver-memory "6000M" sparkr-shell > /tmp/RtmpzJUGsJ/backend_port11c5c2f3ca40b > Error: Master must start with yarn, spark, mesos, or local > Run with --help for usage help or --verbose for debug output Error in > sparkR.init(master = sparkr_master, appName = job_name, sparkJars = > sparkr_jars, : > JVM is not ready after 10 seconds > Calls: orchard_sparkR_init -> sparkR.init > In addition: Warning message: > In processSparkJars(sparkJars) : > sparkJars as a comma-separated string is deprecated, use character vector > instead > Execution halted > > I believe it may have something to do with the sparkr-shell that's forked > from R > If I run this job outside of Chronos directly from the CLI everything runs > fine it connects to my mesos cluster and succeeds. > > I'm not sure what could be causing this issue. > > I've tried using R CMD BATCH myscript.R and Rscript --vanilla myscript.R > both suffer from the same issue. > > Here is the job definition > > { > "name": "PROD_applepie_loan_detail_R", > "command": "env && exec /data/orchard/bin/orchard_job_executor.py -j > PROD_applepie_loan_detail_R", > "shell": true, > "epsilon": "PT15M", > "executor": "", > "executorFlags": "", > "retries": 3, > "owner": "[email protected]", > "ownerName": "", > "runAsUser": "orchard", > "description": "", > "async": false, > "successCount": 0, > "errorCount": 0, > "lastSuccess": "", > "lastError": "", > "cpus": 0.1, > "disk": 10, > "mem": 6000, > "disabled": false, > "softError": false, > "dataProcessingJobType": false, > "errorsSinceLastSuccess": 0, > "uris": [ "file:///data/orchard/R/sparkr_env.R", > "file:///data/orchard/R/applepie_loan_detail.R"], > "environmentVariables": [ > { > "name": "SPARKR_MASTER", > "value": "mesos://leader.mesos:5050" > } > ], > "arguments": [], > "highPriority": false, > "runAsUser": "fedora", > "constraints": [[ "rack", "EQUALS", "spark" ]], > "schedule": "R/2016-06-09T02:00:00.000Z/P1W", > "scheduleTimeZone": "UTC" > } > > -- > > *Rodrick Brown* / Systems Engineer > > +1 917 445 6839 / [email protected] > <[email protected]> > > *Orchard Platform* > > 101 5th Avenue, 4th Floor, New York, NY 10003 > > http://www.orchardplatform.com > > Orchard Blog <http://www.orchardplatform.com/blog/> | Marketplace Lending > Meetup <http://www.meetup.com/Peer-to-Peer-Lending-P2P/> > > *NOTICE TO RECIPIENTS*: This communication is confidential and intended > for the use of the addressee only. If you are not an intended recipient of > this communication, please delete it immediately and notify the sender by > return email. Unauthorized reading, dissemination, distribution or copying > of this communication is prohibited. This communication does not constitute > an offer to sell or a solicitation of an indication of interest to purchase > any loan, security or any other financial product or instrument, nor is it > an offer to sell or a solicitation of an indication of interest to purchase > any products or services to any persons who are prohibited from receiving > such information under applicable law. The contents of this communication > may not be accurate or complete and are subject to change without notice. > As such, Orchard App, Inc. (including its subsidiaries and affiliates, > "Orchard") makes no representation regarding the accuracy or completeness > of the information contained herein. The intended recipient is advised to > consult its own professional advisors, including those specializing in > legal, tax and accounting matters. Orchard does not provide legal, tax or > accounting advice. > -- Best Regards, Haosdent Huang
