Problems when submitting a spark job via the REST API

[Kristinn Rúnarsson]

Hi,

I am trying to submit spark jobs via the "hidden" REST API(
http://spark-cluster-ip:6066/v1/submissions/..), but I am getting
ErrorResponse and I cant find what I am doing wrong.

I have been following the instructions from this blog post:
http://arturmkrtchyan.com/apache-spark-hidden-rest-api

When I try to send a CreateSubmissionRequest action I get the following
ErrorResponse:
{
  "action" : "ErrorResponse",
  "message" : "Malformed request:
org.apache.spark.deploy.rest.SubmitRestProtocolException: Validation of
message CreateSubmissionRequest
failed!\n\torg.apache.spark.deploy.rest.SubmitRestProtocolMessage.validate(SubmitRestProtocolMessage.scala:70)\n\torg.apache.spark.deploy.rest.SubmitRequestServlet.doPost(RestSubmissionServer.scala:272)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\torg.spark_project.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)\n\torg.spark_project.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)\n\torg.spark_project.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\n\torg.spark_project.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\n\torg.spark_project.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\n\torg.spark_project.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\torg.spark_project.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\n\torg.spark_project.jetty.server.Server.handle(Server.java:499)\n\torg.spark_project.jetty.server.HttpChannel.handle(HttpChannel.java:311)\n\torg.spark_project.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\n\torg.spark_project.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\n\torg.spark_project.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\n\tjava.lang.Thread.run(Thread.java:745)",
  "serverSparkVersion" : "2.1.0"
}

This is how my request looks like:

curl -X POST http://kristinn:6066/v1/submissions/create --header
"Content-Type:application/json;charset=UTF-8" --data '{
  "action": "CreateSubmissionRequest",
  "appArgs": [
    100
  ],
  "appResource":
"/opt/spark-2.1.0/examples/jars/spark-examples_2.11-2.1.0.jar",
  "clientSparkVersion": "2.1.0",
  "environmentVariables": {
    "SPARK_ENV_LOADED": "1"
  },
  "mainClass": "org.apache.spark.examples.SparkPi",
  "sparkProperties": {
    "spark.master": "spark://kristinn:6066",
    "spark.submit.deployMode": "cluster",
    "spark.jars":
"/opt/spark-2.1.0/examples/jars/spark-examples_2.11-2.1.0.jar"
  }
}'

I can not see what is causing this by looking at the source code.

Is something wrong in my request or has anyone have a solution for this
issue?

Best,
Kristinn Rúnarrson.