[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 It's great to have this in Spark 2.4! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Thank you for merging, @srowen . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22340 Merged to master (which looks like is still 2.4) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Thank you, @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22340 Seems okay to me too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Could you review this Scala-2.12 PR again, @cloud-fan and @gatorsmile ? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Thank you for approval, @srowen . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 - Since `runSparkSubmit` is a part of `SparkSubmitTestUtils`, some test case will fail if we always remove the env variables like `SPARK_DIST_CLASSPATH`. It gives the child the correct test-time class-path. - For executing old Spark distribution, yes. `SPARK_DIST_CLASSPATH` was the direct root cause and `SPARK_PREPEND_CLASSES` also doesn't look safe to me. I added the others in order to be proactively preventive. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22340 I see. It does seem like we don't want to run with test env variables in this context. I was going to ask if we ever do? should this function always strip the env variables for testing? I can see being conservative and restricting it to this case. It seems like just stripping `SPARK_DIST_CLASSPATH` and maybe `SPARK_PREPEND_CLASSES` is the issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 Also, cc @gatorsmile . --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 `HiveExternalCatalogVersionsSuite` downloads and executes old Spark in their directory. However, it gave the child the followings. None of them is expected here. Especially, `SPARK_DIST_CLASSPATH` is related to this bug. - SPARK_TESTING - SPARK_SQL_TESTING - SPARK_PREPEND_CLASSES - SPARK_DIST_CLASSPATH Previously, in the class path, new Spark classes are behind the old Spark classes. So, new ones are unseen. However, Spark 2.4.0 reveals this bug due to the recent data source class changes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22340 Yeah same question, but I see why that could cause a problem. Is the point here that while this is a test, the spark-submit run by the test should be run 'normally'? I am happy for a solution just what to understand the implications. I wonder why the classpath stuff hasn't caused a problem before if so, but who knows? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22340 How does the non-test mode resolve the class path issue? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22340 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22340 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95705/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22340 **[Test build #95705 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95705/testReport)** for PR 22340 at commit [`d451f98`](https://github.com/apache/spark/commit/d451f989bb3c06304d2b962678f1cab7b561df10). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #22340: [SPARK-25337][SQL][TEST] `runSparkSubmit` should provide...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/22340 cc @srowen and @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
