[
https://issues.apache.org/jira/browse/SPARK-26414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin resolved SPARK-26414.
------------------------------------
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 3.0.0
> Race between SparkContext and YARN AM can cause NPE in UI setup code
> --------------------------------------------------------------------
>
> Key: SPARK-26414
> URL: https://issues.apache.org/jira/browse/SPARK-26414
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 2.4.0
> Reporter: Marcelo Vanzin
> Assignee: Marcelo Vanzin
> Priority: Minor
> Fix For: 3.0.0
>
>
> There's a super narrow race between the SparkContext and the AM startup code:
> - SC starts the AM and waits for it to go into running state
> - AM goes into running state, unblocking SC
> - AM sends AmIpFilter config to SC, adds the filter to the list and then the
> filter configs
> - unblocked SC is in the middle of setting up the UI and sees only the
> filter, but not the configs
> Then you get this:
> {noformat}
> ERROR org.apache.spark.SparkContext - Error initializing SparkContext.
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.init(AmIpFilter.java:81)
> at
> org.spark_project.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
> at
> org.spark_project.jetty.servlet.ServletHandler.initialize(ServletHandler.java:881)
> at
> org.spark_project.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)
> at
> org.spark_project.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
> at
> org.spark_project.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
> at
> org.spark_project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:520)
> at
> org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
> at
> org.apache.spark.ui.WebUI$$anonfun$attachHandler$1.apply(WebUI.scala:96)
> at scala.Option.foreach(Option.scala:257)
> at org.apache.spark.ui.WebUI.attachHandler(WebUI.scala:96)
> at
> org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
> at
> org.apache.spark.SparkContext$$anonfun$22$$anonfun$apply$8.apply(SparkContext.scala:522)
> at scala.Option.foreach(Option.scala:257)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
