We have the same issue. As result, we are stuck back on 1.0.2.
Not being able to programmatically interface directly with the Yarn
client to obtain the application id is a show stopper for us, which is a
real shame given the Yarn enhancements in 1.2, 1.3, and 1.4.
I understand that SparkLauncher was supposed to address these issues,
but it really doesn't. Yarn already provides indirection and an arm's
length transaction for starting Spark on a cluster. The launcher
introduces yet another layer of indirection and dissociates the Yarn
Client from the application that launches it.
I am still reading the newest code, and we are still researching options
to move forward. If there are alternatives, we'd like to know.
Kevin Markey
On 05/11/2015 01:36 AM, Mridul Muralidharan wrote:
That works when it is launched from same process - which is
unfortunately not our case :-)
- Mridul
On Sun, May 10, 2015 at 9:05 PM, Manku Timma <manku.tim...@gmail.com> wrote:
sc.applicationId gives the yarn appid.
On 11 May 2015 at 08:13, Mridul Muralidharan <mri...@gmail.com> wrote:
We had a similar requirement, and as a stopgap, I currently use a
suboptimal impl specific workaround - parsing it out of the
stdout/stderr (based on log config).
A better means to get to this is indeed required !
Regards,
Mridul
On Sun, May 10, 2015 at 7:33 PM, Ron's Yahoo!
<zlgonza...@yahoo.com.invalid> wrote:
Hi,
I used to submit my Spark yarn applications by using
org.apache.spark.yarn.deploy.Client api so I can get the application id
after I submit it. The following is the code that I have, but after
upgrading to 1.3.1, the yarn Client class was made into a private class. Is
there a particular reason why this Client class was made private?
I know that there’s a new SparkSubmit object that can be used, but
it’s not clear to me how I can use it to get the application id after
submitting to the cluster.
Thoughts?
Thanks,
Ron
class SparkLauncherServiceImpl extends SparkLauncherService {
override def runApp(conf: Configuration, appName: String, queue:
String): ApplicationId = {
val ws = SparkLauncherServiceImpl.getWorkspace()
val params = Array("--class", //
"com.xyz.sparkdb.service.impl.AssemblyServiceImpl", //
"--name", appName, //
"--queue", queue, //
"--driver-memory", "1024m", //
"--addJars",
getListOfDependencyJars(s"$ws/ledp/le-sparkdb/target/dependency"), //
"--jar",
s"file:$ws/ledp/le-sparkdb/target/le-sparkdb-1.0.3-SNAPSHOT.jar")
System.setProperty("SPARK_YARN_MODE", "true")
System.setProperty("spark.driver.extraJavaOptions",
"-XX:PermSize=128m -XX:MaxPermSize=128m
-Dsun.io.serialization.extendedDebugInfo=true")
val sparkConf = new SparkConf()
val args = new ClientArguments(params, sparkConf)
new Client(args, conf, sparkConf).runApp()
}
private def getListOfDependencyJars(baseDir: String): String = {
val files = new
File(baseDir).listFiles().filter(!_.getName().startsWith("spark-assembly"))
val prependedFiles = files.map(x => "file:" + x.getAbsolutePath())
val result = ((prependedFiles.tail.foldLeft(new
StringBuilder(prependedFiles.head))) {(acc, e) => acc.append(",
").append(e)}).toString()
result
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org
