[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 Merged to master/2.0 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 I think this would have to go into 2.0.x as I think it's a regression from 2.0.0, in that the default location moved from a local file to HDFS for many users, and, it's an HDFS path that can't be created. I'll leave it open for a few more days. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/15382 cc @yhuai --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/15382 Great. Is it going to 2.0.x, too? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/15382 LGTM. This should restore the behavior pre- 11a6844 and be compatible with Windows. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/67191/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #67191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67191/consoleFull)** for PR 15382 at commit [`69c5383`](https://github.com/apache/spark/commit/69c538333ba02fc806fe7e8f794f269936ebdc28). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #67191 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/67191/consoleFull)** for PR 15382 at commit [`69c5383`](https://github.com/apache/spark/commit/69c538333ba02fc806fe7e8f794f269936ebdc28). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 @vanzin this may be more like what you're suggesting. I think the Windows issue makes this a little bit more complex than what you're describing but I think this ends up being workable. @avulanov FYI @mridulm I think the change in behavior from what's in `master` now is: - Default to a local filesystem path (again) regardless of Hadoop config - Default to `spark-warehouse` in working dir not user dir (as in the docs) The net change is actually pretty minor after all. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66810/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66810 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66810/consoleFull)** for PR 15382 at commit [`ef7a141`](https://github.com/apache/spark/commit/ef7a14102c1269fabf10667947adb8e314d775a0). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66810 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66810/consoleFull)** for PR 15382 at commit [`ef7a141`](https://github.com/apache/spark/commit/ef7a14102c1269fabf10667947adb8e314d775a0). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3322/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3322 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3322/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3314 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3314/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3314 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3314/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3310 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3310/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #3310 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3310/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66580/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66580 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66580/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66580 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66580/consoleFull)** for PR 15382 at commit [`71f124a`](https://github.com/apache/spark/commit/71f124a94afb49f6a3ec6717f4110b5231403398). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 OK, now using `Utils.resolveURI`. This means all inputs without a scheme resolve as `file:` URIs, including the default "spark-warehouse". This will continue to resolve to the local working dir by default, as per docs. Inputs with a scheme can be specified, still, to point to HDFS or S3. I adjusted the tests to, I think, further standardize the handling of URIs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/15382 i don't think there is such a thing as a HDFS working directory, but that probably means it just uses the home dir on hdfs (/user/) for any relative paths --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66501/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66501 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66501/consoleFull)** for PR 15382 at commit [`cd4b4b9`](https://github.com/apache/spark/commit/cd4b4b96f4ee1df8265d696e21231101f7fad826). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66500/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66500 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66500/consoleFull)** for PR 15382 at commit [`67c2b58`](https://github.com/apache/spark/commit/67c2b58170ea91fa1e4b1ee72ffd42a1bb4cc373). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66501 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66501/consoleFull)** for PR 15382 at commit [`cd4b4b9`](https://github.com/apache/spark/commit/cd4b4b96f4ee1df8265d696e21231101f7fad826). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 Update on my thinking: Docs say the default should be "spark-warehouse" in the local working directory. The original code did establish a default on the local file system... though in the home dir, not working dir. So I still favor fixing that, but that's a small, orthogonal question. Although the default appears to be intended to be on the local FS, I don't think it must be. @avulanov mentions it might be on S3; the Hive warehouse dir is on HDFS too by default, in comparison. I don't have evidence that this has been prohibited so I guess we shouldn't assume this is a local path. It seems simple to default to "spark-warehouse" in the local working dir of whatever file system is being used, be it local or HDFS. That means SQLConf shouldn't specify a default with any URI scheme or path. SQLConf should then resolve it, though, at the moment there's no good way to plumb through a Hadoop Configuration. (I just made a new default one in here for the moment.) Then we're back to an interesting detail, that Hadoop Path's toURI won't add a trailing slash in the case of a local file. Maybe it doesn't in general since it can't always tell whether a path is a directory or not. Local URIs however will have a trailing slash. It's probably best to leave the behavior and adjust the tests to not expect a trailing slash, as the original code seemed to (?) Does this still work on Windows? As far as I can tell it would. Is it OK to instantiate `FileSystem.get(new Configuration())` in `SQLConf`? there's no easy access to the actual conf, but, it is only trying to read the default file system which shouldn't depend on Spark-specific Hadoop conf. I think this change actually _restores_ the ability to specify a warehouse path with a URI scheme. This second, it will be interpreted as a file path only if you write "file:/foo". However there's no way to write a relative file URI, so no way to guarantee that a default of "spark-warehouse" is a relative path on the local FS. After this change, HDFS users would see that this correctly defaults to "spark-warehouse" in an HDFS working directory. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66500 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66500/consoleFull)** for PR 15382 at commit [`67c2b58`](https://github.com/apache/spark/commit/67c2b58170ea91fa1e4b1ee72ffd42a1bb4cc373). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 Good point. The docs all say it should be working dir, so I was following the apparent intent along the way here. I don't know if one is more surprising than the other. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/15382 i think working dir makes more sense than home dir. but could this catch people by surprise because we now expect write permission in the working dir? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66459 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66459/consoleFull)** for PR 15382 at commit [`4fb0e58`](https://github.com/apache/spark/commit/4fb0e584aa9711780f89f6309d1c386fb501c336). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15382 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/66459/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15382 **[Test build #66459 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/66459/consoleFull)** for PR 15382 at commit [`4fb0e58`](https://github.com/apache/spark/commit/4fb0e584aa9711780f89f6309d1c386fb501c336). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #15382: [SPARK-17810] [SQL] Default spark.sql.warehouse.dir is r...
Github user srowen commented on the issue: https://github.com/apache/spark/pull/15382 CC @vanzin -- I think your solution at https://github.com/apache/spark/pull/13868#discussion_r82252926 is better, will probably switch to it. CC @avulanov and @koertkuipers --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
