gaborgsomogyi opened a new pull request #26295: [SPARK-29637][CORE] Add description to Job SHS web API URL: https://github.com/apache/spark/pull/26295 ### Why are the changes needed? Starting from Spark 2.3, the SHS REST API endpoint `/applications/<app_id>/jobs/` is not including `description` in the JobData returned. This is not the case until Spark 2.2. In this PR I've added the mentioned field. ### Does this PR introduce any user-facing change? Yes. Old API response: ``` [ { "jobId" : 0, "name" : "foreach at <console>:26", "submissionTime" : "2019-10-28T12:41:54.301GMT", "completionTime" : "2019-10-28T12:41:54.731GMT", "stageIds" : [ 0 ], "jobGroup" : "test", "status" : "SUCCEEDED", "numTasks" : 1, "numActiveTasks" : 0, "numCompletedTasks" : 1, "numSkippedTasks" : 0, "numFailedTasks" : 0, "numKilledTasks" : 0, "numCompletedIndices" : 1, "numActiveStages" : 0, "numCompletedStages" : 1, "numSkippedStages" : 0, "numFailedStages" : 0, "killedTasksSummary" : { } } ] ``` New API response: ``` [ { "jobId" : 0, "name" : "foreach at <console>:26", "description" : "job", "submissionTime" : "2019-10-28T13:37:24.107GMT", "completionTime" : "2019-10-28T13:37:24.613GMT", "stageIds" : [ 0 ], "jobGroup" : "test", "status" : "SUCCEEDED", "numTasks" : 1, "numActiveTasks" : 0, "numCompletedTasks" : 1, "numSkippedTasks" : 0, "numFailedTasks" : 0, "numKilledTasks" : 0, "numCompletedIndices" : 1, "numActiveStages" : 0, "numCompletedStages" : 1, "numSkippedStages" : 0, "numFailedStages" : 0, "killedTasksSummary" : { } } ] ``` ### How was this patch tested? Extended + existing unit tests. Manually: * Open spark-shell ``` scala> sc.setJobGroup("test", "job", false); scala> val foo = sc.textFile("/user/foo.txt"); foo: org.apache.spark.rdd.RDD[String] = /user/foo.txt MapPartitionsRDD[1] at textFile at <console>:24 scala> foo.foreach(println); ``` * Access REST API `http://SHS-host:port/api/v1/applications/<app-id>/jobs/`
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
