sc.applicationId gives the yarn appid. On 11 May 2015 at 08:13, Mridul Muralidharan <mri...@gmail.com> wrote:
> We had a similar requirement, and as a stopgap, I currently use a > suboptimal impl specific workaround - parsing it out of the > stdout/stderr (based on log config). > A better means to get to this is indeed required ! > > Regards, > Mridul > > On Sun, May 10, 2015 at 7:33 PM, Ron's Yahoo! > <zlgonza...@yahoo.com.invalid> wrote: > > Hi, > > I used to submit my Spark yarn applications by using > org.apache.spark.yarn.deploy.Client api so I can get the application id > after I submit it. The following is the code that I have, but after > upgrading to 1.3.1, the yarn Client class was made into a private class. Is > there a particular reason why this Client class was made private? > > I know that there’s a new SparkSubmit object that can be used, but > it’s not clear to me how I can use it to get the application id after > submitting to the cluster. > > Thoughts? > > > > Thanks, > > Ron > > > > class SparkLauncherServiceImpl extends SparkLauncherService { > > > > override def runApp(conf: Configuration, appName: String, queue: > String): ApplicationId = { > > val ws = SparkLauncherServiceImpl.getWorkspace() > > val params = Array("--class", // > > "com.xyz.sparkdb.service.impl.AssemblyServiceImpl", // > > "--name", appName, // > > "--queue", queue, // > > "--driver-memory", "1024m", // > > "--addJars", > getListOfDependencyJars(s"$ws/ledp/le-sparkdb/target/dependency"), // > > "--jar", > s"file:$ws/ledp/le-sparkdb/target/le-sparkdb-1.0.3-SNAPSHOT.jar") > > System.setProperty("SPARK_YARN_MODE", "true") > > System.setProperty("spark.driver.extraJavaOptions", > "-XX:PermSize=128m -XX:MaxPermSize=128m > -Dsun.io.serialization.extendedDebugInfo=true") > > val sparkConf = new SparkConf() > > val args = new ClientArguments(params, sparkConf) > > new Client(args, conf, sparkConf).runApp() > > } > > > > private def getListOfDependencyJars(baseDir: String): String = { > > val files = new > File(baseDir).listFiles().filter(!_.getName().startsWith("spark-assembly")) > > val prependedFiles = files.map(x => "file:" + x.getAbsolutePath()) > > val result = ((prependedFiles.tail.foldLeft(new > StringBuilder(prependedFiles.head))) {(acc, e) => acc.append(", > ").append(e)}).toString() > > result > > } > > } > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >