Github user rekhajoshm commented on a diff in the pull request: https://github.com/apache/spark/pull/21684#discussion_r200784938 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/RestSubmissionClient.scala --- @@ -233,30 +233,43 @@ private[spark] class RestSubmissionClient(master: String) extends Logging { private[rest] def readResponse(connection: HttpURLConnection): SubmitRestProtocolResponse = { import scala.concurrent.ExecutionContext.Implicits.global val responseFuture = Future { - val dataStream = - if (connection.getResponseCode == HttpServletResponse.SC_OK) { - connection.getInputStream - } else { - connection.getErrorStream + val responseCode = connection.getResponseCode + + if (responseCode != HttpServletResponse.SC_OK) { + val errString = Some(Source.fromInputStream(connection.getErrorStream()) + .getLines().mkString("\n")) + logError(s"Server responded with error:\n${errString}") + if (responseCode == HttpServletResponse.SC_INTERNAL_SERVER_ERROR && connection.getRequestMethod == "GET") { --- End diff -- Thanks @srowen for the reviews. Fixed. As our intention is to also comply with existing tests, updated to verify with content_type. The reason is we have two tests in "client handles faulty server" both of which return SC_INTERNAL_SERVER_ERROR but one is expected by tests to return ErrorMessage while other needs to be an exception.I found that strange. Anyhow, debugged and found one(KillSubmissionResponse) returns a valid json and other(StatusSubmissionResponse) is an html response.This is due to the internal behavior of KillRequestServlet and StatusRequestServlet. This PR keeps tests behavior consistent, while addressing the objective of SPARK-24470 ``` KillSubmissionResponse for KillSubmissionResponse is json/500 { "action" : "ErrorResponse", "message" : "Internal server error: org.apache.spark.deploy.rest.SubmitRestProtocolException: Validation of message KillSubmissionResponse failed!\n\torg.apache.spark.deploy.rest.SubmitRestProtocolMessage.validate(SubmitRestProtocolMessage.scala:70)\n\torg.apache.spark.deploy.rest.RestServlet.validateResponse(RestSubmissionServer.scala:196)\n\torg.apache.spark.deploy.rest.RestServlet.sendResponse(RestSubmissionServer.scala:134)\n\torg.apache.spark.deploy.rest.KillRequestServlet.doPost(RestSubmissionServer.scala:223)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:707)\n\tjavax.servlet.http.HttpServlet.service(HttpServlet.java:790)\n\torg.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)\n\torg.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:584)\n\torg.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)\n\torg.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)\n\torg.eclipse.jetty.server .handler.ContextHandler.doScope(ContextHandler.java:1112)\n\torg.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\torg.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)\n\torg.eclipse.jetty.server.Server.handle(Server.java:534)\n\torg.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)\n\torg.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)\n\torg.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)\n\torg.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)\n\torg.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)\n\torg.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)\n\torg.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)\n\torg.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.ja va:136)\n\torg.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)\n\torg.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)\n\tjava.lang.Thread.run(Thread.java:748)", "serverSparkVersion" : "2.4.0-SNAPSHOT" } ``` ``` StatusSubmissionResponse from /submissions/status/anything is html/500 <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 500 </title> </head> <body> <h2>HTTP ERROR: 500</h2> <p>Problem accessing /v1/submissions/status/anything. Reason: <pre> java.lang.ArithmeticException: / by zero</pre></p> <hr /><a href="http://eclipse.org/jetty">Powered by Jetty:// 9.3.20.v20170531</a><hr/> </body> </html> ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org