[GitHub] [spark] dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency
dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency URL: https://github.com/apache/spark/pull/26742#issuecomment-561387746 Thank you for review. Merged to master. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency
dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency URL: https://github.com/apache/spark/pull/26742#issuecomment-561256563 Before this PR, I tested with `make-distribution.sh` and `master/worker/historyserver/sparkthriftserver/shell` (I didn't test the yarn suffle service tough). > This seems to remove it entirely from the distribution by marking it provided. 1. This PR does not remove it because we embedded `jetty` into `spark-core` library already. The following is the result of `make-distribution.sh`. ``` $ jar tvf spark-core_2.12-3.0.0-SNAPSHOT.jar | grep jetty | tail -n5 11996 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/PushCacheFilter.class 9569 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/PutFilter.class 1302 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/CGI$3.class 108 Mon Apr 29 15:50:58 PDT 2019 META-INF/maven/org.eclipse.jetty/jetty-servlets/pom.properties 2142 Mon Dec 02 22:25:20 PST 2019 org/sparkproject/jetty/servlets/CGI$EnvList.class ``` 2. Jetty project consists of multi-modules like Apache Spark. `jetty-webapp` is something like `graphx`. `jetty-webapp` has `WebAppContext` which loads `web.xml` and builds web apps. > A WebAppContext is a derivation of ServletContextHandler that supports the standardized layout of a web application and configuration of session, security, listeners, filter, servlets, and JSP via a web.xml descriptor normally found in the /WEB-INF directory of a web application. https://github.com/eclipse/jetty.project/tree/jetty-9.4.x/jetty-webapp AFAIK, Apache Spark don't use `web.xml` or the standardized-layouted web applications style. I guess yarn `shuffle-service` also doesn't use this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency
dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency URL: https://github.com/apache/spark/pull/26742#issuecomment-561023253 If I misunderstand something, I'll close this PR~ Please let me know. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency
dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency URL: https://github.com/apache/spark/pull/26742#issuecomment-561022246 `spark-core` already embeds the required `jetty` libraries into `spark-core_2.13_*.jar`, and this is not used for even in our UTs. Ur, are we using these libraries? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency
dongjoon-hyun commented on issue #26742: [SPARK-30051][BUILD] Clean up hadoop-3.2 dependency URL: https://github.com/apache/spark/pull/26742#issuecomment-560993217 Hi, @srowen and @HyukjinKwon . Could you review this PR? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
