[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+

2019-09-15 Thread Dongjoon Hyun (Jira) 


[ 
https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16930019#comment-16930019
 ] 

Dongjoon Hyun commented on SPARK-27122:
---

This is backported to `branch-2.4` via 
https://github.com/apache/spark/pull/25793 .
Although this helps JDK11, but this is a general issue with JDK8+. So, I'll 
remove this from this JDK11 umbrella like SPARK-29087 .

> YARN test failures in Java 9+
> -
>
> Key: SPARK-27122
> URL: https://issues.apache.org/jira/browse/SPARK-27122
> Project: Spark
>  Issue Type: Sub-task
>  Components: YARN
>Affects Versions: 3.0.0
>Reporter: Sean Owen
>Assignee: Ajith S
>Priority: Major
> Fix For: 2.4.5, 3.0.0
>
> Attachments: image-2019-03-14-09-34-20-592.png, 
> image-2019-03-14-09-35-23-046.png
>
>
> Currently on Java 11:
> {code}
> YarnSchedulerBackendSuite:
> - RequestExecutors reflects node blacklist and is serializable
> - Respect user filters when adding AM IP filter *** FAILED ***
>   java.lang.ClassCastException: 
> org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to 
> org.eclipse.jetty.servlet.ServletContextHandler
>   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>   at scala.collection.TraversableLike.map(TraversableLike.scala:237)
>   at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174)
>   at scala.Option.foreach(Option.scala:274)
>   ...
> {code}
> This looks like a classpath issue, probably ultimately related to the same 
> classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+

2019-03-13 Thread Ajith S (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792355#comment-16792355
 ] 

Ajith S commented on SPARK-27122:
-

ping [~srowen] [~dongjoon] [~Gengliang.Wang]

> YARN test failures in Java 9+
> -
>
> Key: SPARK-27122
> URL: https://issues.apache.org/jira/browse/SPARK-27122
> Project: Spark
>  Issue Type: Sub-task
>  Components: YARN
>Affects Versions: 3.0.0
>Reporter: Sean Owen
>Priority: Major
> Attachments: image-2019-03-14-09-34-20-592.png, 
> image-2019-03-14-09-35-23-046.png
>
>
> Currently on Java 11:
> {code}
> YarnSchedulerBackendSuite:
> - RequestExecutors reflects node blacklist and is serializable
> - Respect user filters when adding AM IP filter *** FAILED ***
>   java.lang.ClassCastException: 
> org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to 
> org.eclipse.jetty.servlet.ServletContextHandler
>   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>   at scala.collection.TraversableLike.map(TraversableLike.scala:237)
>   at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174)
>   at scala.Option.foreach(Option.scala:274)
>   ...
> {code}
> This looks like a classpath issue, probably ultimately related to the same 
> classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+

2019-03-13 Thread Ajith S (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792318#comment-16792318
 ] 

Ajith S commented on SPARK-27122:
-

The problem seems to be shading of jetty package.

When we run test, the class path seems to be made from the classes 
folder(resource-managers/yarn/target/scala-2.12/classes) instead of jar 
(resource-managers/yarn/target/spark-yarn_2.12-3.0.0-SNAPSHOT.jar) 

Here is test classpath info:

!image-2019-03-14-09-34-20-592.png!

 

And here is the javap command which shows the difference between 
org.apache.spark.scheduler.cluster.YarnSchedulerBackend present in jar folder 
and classes folder

!image-2019-03-14-09-35-23-046.png!

> YARN test failures in Java 9+
> -
>
> Key: SPARK-27122
> URL: https://issues.apache.org/jira/browse/SPARK-27122
> Project: Spark
>  Issue Type: Sub-task
>  Components: YARN
>Affects Versions: 3.0.0
>Reporter: Sean Owen
>Priority: Major
> Attachments: image-2019-03-14-09-34-20-592.png, 
> image-2019-03-14-09-35-23-046.png
>
>
> Currently on Java 11:
> {code}
> YarnSchedulerBackendSuite:
> - RequestExecutors reflects node blacklist and is serializable
> - Respect user filters when adding AM IP filter *** FAILED ***
>   java.lang.ClassCastException: 
> org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to 
> org.eclipse.jetty.servlet.ServletContextHandler
>   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>   at scala.collection.TraversableLike.map(TraversableLike.scala:237)
>   at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174)
>   at scala.Option.foreach(Option.scala:274)
>   ...
> {code}
> This looks like a classpath issue, probably ultimately related to the same 
> classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-27122) YARN test failures in Java 9+

2019-03-13 Thread Ajith S (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-27122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16792299#comment-16792299
 ] 

Ajith S commented on SPARK-27122:
-

I can reproduce this issue even in Java8. I would like to work on this.

 

> YARN test failures in Java 9+
> -
>
> Key: SPARK-27122
> URL: https://issues.apache.org/jira/browse/SPARK-27122
> Project: Spark
>  Issue Type: Sub-task
>  Components: YARN
>Affects Versions: 3.0.0
>Reporter: Sean Owen
>Priority: Major
>
> Currently on Java 11:
> {code}
> YarnSchedulerBackendSuite:
> - RequestExecutors reflects node blacklist and is serializable
> - Respect user filters when adding AM IP filter *** FAILED ***
>   java.lang.ClassCastException: 
> org.spark_project.jetty.servlet.ServletContextHandler cannot be cast to 
> org.eclipse.jetty.servlet.ServletContextHandler
>   at 
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237)
>   at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
>   at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
>   at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
>   at scala.collection.TraversableLike.map(TraversableLike.scala:237)
>   at scala.collection.TraversableLike.map$(TraversableLike.scala:230)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:108)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2(YarnSchedulerBackend.scala:183)
>   at 
> org.apache.spark.scheduler.cluster.YarnSchedulerBackend.$anonfun$addWebUIFilter$2$adapted(YarnSchedulerBackend.scala:174)
>   at scala.Option.foreach(Option.scala:274)
>   ...
> {code}
> This looks like a classpath issue, probably ultimately related to the same 
> classloader issues in https://issues.apache.org/jira/browse/SPARK-26839 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org