[jira] [Comment Edited] (SPARK-2100) Allow users to disable Jetty Spark UI in local mode
[ https://issues.apache.org/jira/browse/SPARK-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033049#comment-14033049 ] DB Tsai edited comment on SPARK-2100 at 6/16/14 9:50 PM: - [~sowen] You are right. The servlet api is pulled by jetty's dependency. If the jetty is included with intransitive annotation, the tomcat can start. However, when I start a new SparkContext, it will hang forever without any error message. val sc = new SparkContext(local, appName, sparkConf) was (Author: dbtsai): [~sowen] You are right. The servlet api is pulled by jetty's dependency. If the jetty is included with intransitive annotation, the tomcat can start. However, when I start a new SparkContext, it will hang forever without any error message. val sc = new SparkContext(deployMode, appName, sparkConf) Allow users to disable Jetty Spark UI in local mode --- Key: SPARK-2100 URL: https://issues.apache.org/jira/browse/SPARK-2100 Project: Spark Issue Type: Improvement Reporter: DB Tsai Since we want to use Spark hadoop APIs in local mode for design time to explore the first couple hundred lines of data in HDFS. Also, we want to use Spark in our tomcat application, so starting a jetty UI will make our tomcat unhappy. In those scenarios, Spark UI is not necessary, and wasting resource. As a result, for local mode, it's desirable that users are able to disable the spark UI. Couple places I found where the jetty will be started. In SparkEnv.scala 1) val broadcastManager = new BroadcastManager(isDriver, conf, securityManager) 2) val httpFileServer = new HttpFileServer(securityManager) httpFileServer.initialize() I don't know if broadcastManager is needed in local mode tho. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (SPARK-2100) Allow users to disable Jetty Spark UI in local mode
[ https://issues.apache.org/jira/browse/SPARK-2100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14033055#comment-14033055 ] Sean Owen edited comment on SPARK-2100 at 6/16/14 9:55 PM: --- Yes, the Maven build has to do a little work to exclude copies of the Servlet 2.x API. Spark ends up including one copy of the Servlet 3.0 APIs, which should make everybody happy. But if your build brings back in something else, and it's bringing its own Servlet API, you may need to exclude it. (This dependency is super annoying because different containers have distributed the same classes in different artifacts.) Advert break: SPARK-1949 fixes this type of issue for Spark's own SBT-based build. Not exactly the issue here but related, and would be cool to get it committed. https://issues.apache.org/jira/browse/SPARK-1949 was (Author: srowen): Yes, the Maven build has to do a little work to exclude copies of the Servlet 2.x API. Spark ends up including one copy of the Servlet 3.0 APIs, which should everybody happing. But if your build brings back in something else, and it's bringing its own Servlet API, you may need to exclude it. (This dependency is super annoying because different containers have distributed the same classes in different artifacts.) Advert break: SPARK-1949 fixes this type of issue for Spark's own SBT-based build. Not exactly the issue here but related, and would be cool to get it committed. https://issues.apache.org/jira/browse/SPARK-1949 Allow users to disable Jetty Spark UI in local mode --- Key: SPARK-2100 URL: https://issues.apache.org/jira/browse/SPARK-2100 Project: Spark Issue Type: Improvement Reporter: DB Tsai Since we want to use Spark hadoop APIs in local mode for design time to explore the first couple hundred lines of data in HDFS. Also, we want to use Spark in our tomcat application, so starting a jetty UI will make our tomcat unhappy. In those scenarios, Spark UI is not necessary, and wasting resource. As a result, for local mode, it's desirable that users are able to disable the spark UI. Couple places I found where the jetty will be started. In SparkEnv.scala 1) val broadcastManager = new BroadcastManager(isDriver, conf, securityManager) 2) val httpFileServer = new HttpFileServer(securityManager) httpFileServer.initialize() I don't know if broadcastManager is needed in local mode tho. -- This message was sent by Atlassian JIRA (v6.2#6252)
