[GitHub] spark issue #14124: [SPARK-16472][SQL] Inconsistent nullability in schema af...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14124 **[Test build #64631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64631/consoleFull)** for PR 14124 at commit [`ffacb55`](https://github.com/apache/spark/commit/ffacb55a9a13fc3144683d9dad8f2da21705a613). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14745: [SPARK-16896][SQL] Handle duplicated field names in head...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14745 **[Test build #64630 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64630/consoleFull)** for PR 14745 at commit [`0c02581`](https://github.com/apache/spark/commit/0c02581579a8c21ce3a3a992835646e52478e19b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14859 **[Test build #64629 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64629/consoleFull)** for PR 14859 at commit [`1b7b5f3`](https://github.com/apache/spark/commit/1b7b5f31e3eb833675045cad79aa2e73e77adc21). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/14871 LGTM Josh, how long does your microbenchmark take if you comment out the call to abortIfCompletelyBlacklisted? Wondering how much that continues to affect performance. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64622/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14531 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14531 **[Test build #64622 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64622/consoleFull)** for PR 14531 at commit [`1f70f9a`](https://github.com/apache/spark/commit/1f70f9ae2fdf2651889c574c0d29a99564685443). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14712 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14712 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64626/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14712 **[Test build #64626 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64626/consoleFull)** for PR 14712 at commit [`7e39a86`](https://github.com/apache/spark/commit/7e39a86030e45f10ae0c171a475c054b7c208d20). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14801 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64619/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14801 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14801 **[Test build #64619 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64619/consoleFull)** for PR 14801 at commit [`439db0b`](https://github.com/apache/spark/commit/439db0bb86179d376874c83e89f7cf3981fb1b6a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14531 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64620/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14531 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14531 **[Test build #64620 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64620/consoleFull)** for PR 14531 at commit [`cf747f8`](https://github.com/apache/spark/commit/cf747f80d0756591e1ed801b89a6ef18257ed1df). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14859 **[Test build #64628 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64628/consoleFull)** for PR 14859 at commit [`e7addc9`](https://github.com/apache/spark/commit/e7addc938691c2da420c2fef2e7b9131fdec00b2). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14859 **[Test build #64627 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64627/consoleFull)** for PR 14859 at commit [`97f3ea7`](https://github.com/apache/spark/commit/97f3ea766680f0cf2b73fb77a35bf79c63728bdf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14872: [SPARK-3162][MLlib][WIP] Add local tree training for dec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14872 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64624/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14872: [SPARK-3162][MLlib][WIP] Add local tree training for dec...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14872 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14872: [SPARK-3162][MLlib][WIP] Add local tree training for dec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14872 **[Test build #64624 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64624/consoleFull)** for PR 14872 at commit [`aa4fcc8`](https://github.com/apache/spark/commit/aa4fcc8d401385f38fe0cdfdb9fe39062c3a9f96). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64617/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #64617 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64617/consoleFull)** for PR 13599 at commit [`31cb42c`](https://github.com/apache/spark/commit/31cb42cfceb937941373caa76fe602abdfca3c9a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14204 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64615/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14204 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14204: [SPARK-16520] [WEBUI] Link executors to corresponding wo...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14204 **[Test build #64615 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64615/consoleFull)** for PR 14204 at commit [`bed0310`](https://github.com/apache/spark/commit/bed03100f93bfa96d1fa027f9ec9060ca14b7311). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #8880: [SPARK-5682][Core] Add encrypted shuffle in spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/8880 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #8880: [SPARK-5682][Core] Add encrypted shuffle in spark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/8880 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64614/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #8880: [SPARK-5682][Core] Add encrypted shuffle in spark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/8880 **[Test build #64614 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64614/consoleFull)** for PR 8880 at commit [`928a59b`](https://github.com/apache/spark/commit/928a59bc4566ec40e6caeccbc628369f050c31c9). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14710: [SPARK-16533][CORE] resolve deadlocking in driver when e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14710 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14710: [SPARK-16533][CORE] resolve deadlocking in driver when e...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14710 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64613/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14710: [SPARK-16533][CORE] resolve deadlocking in driver when e...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14710 **[Test build #64613 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64613/consoleFull)** for PR 14710 at commit [`3eb34fd`](https://github.com/apache/spark/commit/3eb34fd0190809b3dbfe981561210130f1743090). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14859 To cut it short, my suggestion is, - Filter commits via (maybe via message?) https://www.appveyor.com/docs/how-to/filtering-commits/ - Enable [Rolling build](https://www.appveyor.com/docs/build-configuration/#rolling-builds) - Run the build periodically as described above. If it sounds good, I will go ahead and test. But before proceeding, it'd be great if I can hear other opinions. I think it'd be okay just to filter commits via message. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14859 ### How to set up 1. Sign-up https://ci.appveyor.com 2. Go to profile https://cloud.githubusercontent.com/assets/6477701/18074997/093bf91e-6eac-11e6-99c0-80f75da9192d.png;> 3. Enable the link with GitHub https://cloud.githubusercontent.com/assets/6477701/18075004/19071450-6eac-11e6-98aa-edb4d7f7ca86.png;> 4. Add a project (Go to the PROJECTS menu and than add a new project) - https://cloud.githubusercontent.com/assets/6477701/18075017/2e572ffc-6eac-11e6-8e72-1531c81717a0.png;> - https://cloud.githubusercontent.com/assets/6477701/18075026/3ee57bc6-6eac-11e6-826e-5dd09aeb0e7c.png;> 5. Click Github project list and then click - https://cloud.githubusercontent.com/assets/6477701/18075047/6ffe59f8-6eac-11e6-8e60-b0595e2daed3.png;> - https://cloud.githubusercontent.com/assets/6477701/18075059/88763730-6eac-11e6-9107-1fdb34a4074f.png;> 6. Push any commit and check the build is running. - Click PROJECTS menu https://cloud.githubusercontent.com/assets/6477701/18075088/ce13d8ec-6eac-11e6-8425-6263a29e8081.png;> - Click Spark project https://cloud.githubusercontent.com/assets/6477701/18075089/cf8ffa7a-6eac-11e6-84dd-0516c8ebe77a.png;> ### How to re-build/stop/check the builds 1. Click PROJECTS and then Spark project - Click PROJECTS menu https://cloud.githubusercontent.com/assets/6477701/18075088/ce13d8ec-6eac-11e6-8425-6263a29e8081.png;> - Click Spark project https://cloud.githubusercontent.com/assets/6477701/18075089/cf8ffa7a-6eac-11e6-84dd-0516c8ebe77a.png;> 2. Check build status https://cloud.githubusercontent.com/assets/6477701/18075147/4853579a-6ead-11e6-98b3-cf2a797da970.png;> 3. Re-build https://cloud.githubusercontent.com/assets/6477701/18075336/de618b52-6eae-11e6-8f01-e4ce48963087.png;> 4. Stop build https://cloud.githubusercontent.com/assets/6477701/18075308/90657de6-6eae-11e6-9c8a-f0bf6d8293ca.png;> ### How this build works Identically with Travis CI. Per-commit. ### Options to enable/disable the build for branch, PR and commits. 1. Master branch only - Click settings https://cloud.githubusercontent.com/assets/6477701/18075402/6d02ae7c-6eaf-11e6-953c-d683cc18c1c9.png;> - Set the branch to build as below: https://cloud.githubusercontent.com/assets/6477701/18075416/8fac36c8-6eaf-11e6-9262-797a2a66fec4.png;> https://cloud.githubusercontent.com/assets/6477701/18075421/97b17734-6eaf-11e6-8b19-bc1dca840c96.png;> 2. Disable builds for Pull Reqeusts (no PRs) - Click Spark's Github Project settings  - Clieck **Webhooks & services** https://cloud.githubusercontent.com/assets/6477701/18075583/fee3094e-6eb0-11e6-9341-8e9e529a0586.png;> - Clieck https://ci.appveyor.com/api/github/webhook (pull_request and push) https://cloud.githubusercontent.com/assets/6477701/18075585/030712d6-6eb1-11e6-9d4b-6a5dac26af02.png;> - Disable webhook for each pull requests https://cloud.githubusercontent.com/assets/6477701/18075587/064f41a2-6eb1-11e6-810c-6c8f1386e682.png;> 3. Run the build periodically https://cloud.githubusercontent.com/assets/6477701/18075450/d4ef256a-6eaf-11e6-8e41-74e38dac8ca0.png;> 4. Select commits to build (Some PRs) - It seems we can filter some commits to run a build as described [here](https://www.appveyor.com/docs/how-to/filtering-commits/) - If we enable [Rolling build](https://www.appveyor.com/docs/build-configuration/#rolling-builds) and filter the commits, I think it'd be okay maybe. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64616/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13599 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #64616 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64616/consoleFull)** for PR 13599 at commit [`5c1a183`](https://github.com/apache/spark/commit/5c1a183c903f16eb11997fa66b12f1ef6c643100). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/14712 @gatorsmile Thank you for the information! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14712 **[Test build #64626 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64626/consoleFull)** for PR 14712 at commit [`7e39a86`](https://github.com/apache/spark/commit/7e39a86030e45f10ae0c171a475c054b7c208d20). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14712 Since you are added to the white list, you can trigger the test by yourself. Below is the command you can use: - "ok to test" to accept this pull request for testing - "test this please" for a one time test run - If the build fails for other various reasons you can rebuild. "retest this please" to start a new build --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/14712 @cloud-fan @hvanhovell Oh, sorry, it's already been lauched. There's latency for about 5 mins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14712 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64625/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14712 **[Test build #64625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64625/consoleFull)** for PR 14712 at commit [`9c27071`](https://github.com/apache/spark/commit/9c27071c05da5f285726381dff7eff3dfab7eda9). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14712 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14712 **[Test build #64625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64625/consoleFull)** for PR 14712 at commit [`9c27071`](https://github.com/apache/spark/commit/9c27071c05da5f285726381dff7eff3dfab7eda9). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14712: [SPARK-17072] [SQL] support table-level statistics gener...
Github user wzhfy commented on the issue: https://github.com/apache/spark/pull/14712 @cloud-fan @hvanhovell Could you launch a test for this pr? Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14872: [SPARK-3162][MLlib][WIP] Add local tree training for dec...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14872 **[Test build #64624 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64624/consoleFull)** for PR 14872 at commit [`aa4fcc8`](https://github.com/apache/spark/commit/aa4fcc8d401385f38fe0cdfdb9fe39062c3a9f96). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64610/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14871 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14871 **[Test build #64610 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64610/consoleFull)** for PR 14871 at commit [`321d0c6`](https://github.com/apache/spark/commit/321d0c685225f81a8415fb0796e907271d1995ab). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14872: [SPARK-3162][MLlib][WIP] Add local tree training for dec...
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/14872 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14868 **[Test build #64623 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64623/consoleFull)** for PR 14868 at commit [`9d7fb85`](https://github.com/apache/spark/commit/9d7fb85f6f2741f050344b06f2efac3254da77bf). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64623/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14868 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14868 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64621/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14868 **[Test build #64621 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64621/consoleFull)** for PR 14868 at commit [`dc09d8c`](https://github.com/apache/spark/commit/dc09d8c2f6191c98fefe4b9dfe7bdd817907560c). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14868 **[Test build #64623 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64623/consoleFull)** for PR 14868 at commit [`9d7fb85`](https://github.com/apache/spark/commit/9d7fb85f6f2741f050344b06f2efac3254da77bf). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14855: [SPARK-17284] [SQL] Remove Statistics-related Tab...
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/14855#discussion_r76725105 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -791,11 +791,22 @@ case class ShowCreateTableCommand(table: TableIdentifier) extends RunnableComman } } + // These table properties should not be included in the output statement of SHOW CREATE TABLE + val excludedTableProperties = Set( +// The following are hive-generated statistics fields +"COLUMN_STATS_ACCURATE", +"numFiles", +"numPartitions", +"numRows", +"rawDataSize", +"totalSize" + ) --- End diff -- This PR is for fixing a bug. We might need to backport to 2.0. When we implementing the translation layer, we can do that, just like what we did for the property names of the Data Source Table schema --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14868: Implements percentile_approx aggregation function which ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14868 **[Test build #64621 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64621/consoleFull)** for PR 14868 at commit [`dc09d8c`](https://github.com/apache/spark/commit/dc09d8c2f6191c98fefe4b9dfe7bdd817907560c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14531 **[Test build #64622 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64622/consoleFull)** for PR 14531 at commit [`1f70f9a`](https://github.com/apache/spark/commit/1f70f9ae2fdf2651889c574c0d29a99564685443). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14531 Update: `unsupportedFeatures` should not be copied from the source table. The created table does not have these unsupported features. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14855: [SPARK-17284] [SQL] Remove Statistics-related Tab...
Github user wzhfy commented on a diff in the pull request: https://github.com/apache/spark/pull/14855#discussion_r76724760 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/command/tables.scala --- @@ -791,11 +791,22 @@ case class ShowCreateTableCommand(table: TableIdentifier) extends RunnableComman } } + // These table properties should not be included in the output statement of SHOW CREATE TABLE + val excludedTableProperties = Set( +// The following are hive-generated statistics fields +"COLUMN_STATS_ACCURATE", +"numFiles", +"numPartitions", +"numRows", +"rawDataSize", +"totalSize" + ) --- End diff -- Shouldn't we set each of these property names as a constant so that we can use them in the translation layer? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14866: [SPARK-17298][SQL] Require explicit CROSS join for carte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14866 Build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14866: [SPARK-17298][SQL] Require explicit CROSS join for carte...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14866 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64612/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bugs in C...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14531 **[Test build #64620 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64620/consoleFull)** for PR 14531 at commit [`cf747f8`](https://github.com/apache/spark/commit/cf747f80d0756591e1ed801b89a6ef18257ed1df). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14866: [SPARK-17298][SQL] Require explicit CROSS join for carte...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14866 **[Test build #64612 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64612/consoleFull)** for PR 14866 at commit [`256ad85`](https://github.com/apache/spark/commit/256ad8583da6e97f5bdfc35bab0fb2734d0f9b18). * This patch **fails Spark unit tests**. * This patch **does not merge cleanly**. * This patch adds the following public classes _(experimental)_: * `case class CheckCartesianProducts(conf: CatalystConf)` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14801 **[Test build #64619 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64619/consoleFull)** for PR 14801 at commit [`439db0b`](https://github.com/apache/spark/commit/439db0bb86179d376874c83e89f7cf3981fb1b6a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14801 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14801 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14801 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64618/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14801 **[Test build #64618 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64618/consoleFull)** for PR 14801 at commit [`439db0b`](https://github.com/apache/spark/commit/439db0bb86179d376874c83e89f7cf3981fb1b6a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14871 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14871 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/64609/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14871: [SPARK-17304] Fix perf. issue caused by TaskSetManager.a...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14871 **[Test build #64609 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64609/consoleFull)** for PR 14871 at commit [`5d20b44`](https://github.com/apache/spark/commit/5d20b445200ab23283dd9456f7bd3c765dd11d2a). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14859: [SPARK-17200][PROJECT INFRA][BUILD][SparkR] Automate bui...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14859 Thanks all. Then, let me try to write some documentation for.. - How this build works - Where we can check its status - What we can do to restart a build - Options to enable/disable the build for branch, PR and commits. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14649: [SPARK-17059][SQL] Allow FileFormat to specify partition...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14649 (As I am already here), I also think this should be helpful, in particular, for S3 with Parquet. However, I am wondering if this might be only Parquet-specific optimization. I mean, we don't have metafile for other file based data sources. So, my personal opinion is, to put this within Parquet without adding another interface. If we can implement this for other data sources in the future, I think we can add this interface in the future but not now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/14801 LGTM, pending jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14801: [SPARK-17234] [SQL] Table Existence Checking when Index ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14801 **[Test build #64618 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64618/consoleFull)** for PR 14801 at commit [`439db0b`](https://github.com/apache/spark/commit/439db0bb86179d376874c83e89f7cf3981fb1b6a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11956: [SPARK-14098][SQL] Generate Java code that gets a float/...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/11956 @davies, could you please rerun Jenkins for PR [#13899](https://github.com/apache/spark/pull/13899)? I would like to know which tests were failed by the current implementation. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14671: [SPARK-17091][SQL] ParquetFilters rewrite IN to OR of Eq
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14671 @ash211 I am happy to do so but I would like to make sure if there is a offline benchmark performed already and if we can disable this if the performance is better. I don't want to duplicate some efforts so I just, at least, want a stamp of approval from one of committers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #11956: [SPARK-14098][SQL] Generate Java code that gets a float/...
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/11956 Thank you for your clarification. [Here](https://trongkhoanguyenblog.wordpress.com/2014/12/14/understand-the-storage-module-in-spark-core/) is a good summary for me. We may add property like `spark.sql.cache.compression.codec` to specify a compression method. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14678: [MINOR][SQL] Add missing functions for some options in S...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14678 ping @rxin --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14660: [SPARK-17071][SQL] Add an option to support for reading ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/14660 ping @rxin and @liancheng --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14298: [SPARK-16283][SQL] Implement `percentile_approx` ...
Github user lw-lin closed the pull request at: https://github.com/apache/spark/pull/14298 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76721400
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
+Option(classOf[LazySimpleSerDe].getCanonicalName))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE a data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
Some("default")))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+ // The table type of the source table should be a Hive-managed data
source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+val path = dir.getCanonicalPath
+spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH
'$path')")
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+// The source table should be an external data source table
+val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+// The table type of the source table should be an external data
source table
+assert(DDLUtils.isDatasourceTable(sourceTable))
+assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+checkCreateTableLike(sourceTable, targetTable)
+ }
+}
+ }
+
+ test("CREATE TABLE LIKE a managed Hive serde table") {
+val catalog = spark.sessionState.catalog
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS
SELECT 1 key, 'a'")
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable =
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external Hive serde table") {
+val catalog = spark.sessionState.catalog
+withTempDir { tmpDir =>
+ val basePath = tmpDir.getCanonicalPath
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+assert(tmpDir.listFiles.isEmpty)
+sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment
'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
+ """.stripMargin)
+for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12"))
{
+ sql(
+
[GitHub] spark issue #14298: [SPARK-16283][SQL] Implement `percentile_approx` SQL fun...
Github user lw-lin commented on the issue: https://github.com/apache/spark/pull/14298 @clockfly Your PR's improvements overall look reasonable to me, so I'm closing this one. Good job! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13704: [SPARK-15985][SQL] Eliminate redundant cast from an arra...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/13704 left some comment, let's go ahead and merge it after that :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #13704: [SPARK-15985][SQL] Eliminate redundant cast from ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r76721258
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyCastsSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl._
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types._
+
+class SimplifyCastsSuite extends PlanTest {
+
+ object Optimize extends RuleExecutor[LogicalPlan] {
+val batches = Batch("SimplifyCasts", FixedPoint(50), SimplifyCasts) ::
Nil
+ }
+
+ test("non-nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, false)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
false)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, false)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
true)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, true)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
false)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+comparePlans(optimized, plan)
+ }
+
+ test("nullable to nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, true)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
true)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to non-nullable map cast") {
+val input = LocalRelation('m.map(MapType(StringType, StringType,
false)))
+val plan = input.select('m.cast(MapType(StringType, StringType, false))
+ .as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('m.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to nullable map cast") {
+val input = LocalRelation('m.map(MapType(StringType, StringType,
false)))
+val plan = input.select('m.cast(MapType(StringType, StringType, true))
+ .as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('m.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("nullable to non-nullable map cast") {
+val input = LocalRelation('m.map(MapType(StringType, StringType,
true)))
+val plan = input.select('m.cast(MapType(StringType, StringType, false))
+ .as("casted")).analyze
+val optimized = Optimize.execute(plan)
+comparePlans(optimized, plan)
+ }
+
+ test("nullable to nullable map cast") {
+val input = LocalRelation('m.map(MapType(StringType, StringType,
true)))
+val plan = input.select('m.cast(MapType(StringType, StringType, true))
+ .as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('m.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+}
--- End diff --
[GitHub] spark pull request #13704: [SPARK-15985][SQL] Eliminate redundant cast from ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r76721180
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyCastsSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl._
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types._
+
+class SimplifyCastsSuite extends PlanTest {
+
+ object Optimize extends RuleExecutor[LogicalPlan] {
+val batches = Batch("SimplifyCasts", FixedPoint(50), SimplifyCasts) ::
Nil
+ }
+
+ test("non-nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, false)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
false)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, false)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
true)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, true)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
false)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+comparePlans(optimized, plan)
+ }
+
+ test("nullable to nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, true)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
true)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to non-nullable map cast") {
+val input = LocalRelation('m.map(MapType(StringType, StringType,
false)))
+val plan = input.select('m.cast(MapType(StringType, StringType, false))
+ .as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('m.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to nullable map cast") {
--- End diff --
non-nullable value map to nullable value map cast
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76721164
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
--- End diff --
Sure, will do it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76721108
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
+Option(classOf[LazySimpleSerDe].getCanonicalName))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE a data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
Some("default")))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+ // The table type of the source table should be a Hive-managed data
source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+val path = dir.getCanonicalPath
+spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH
'$path')")
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+// The source table should be an external data source table
+val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+// The table type of the source table should be an external data
source table
+assert(DDLUtils.isDatasourceTable(sourceTable))
+assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+checkCreateTableLike(sourceTable, targetTable)
+ }
+}
+ }
+
+ test("CREATE TABLE LIKE a managed Hive serde table") {
+val catalog = spark.sessionState.catalog
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS
SELECT 1 key, 'a'")
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable =
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external Hive serde table") {
+val catalog = spark.sessionState.catalog
+withTempDir { tmpDir =>
+ val basePath = tmpDir.getCanonicalPath
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+assert(tmpDir.listFiles.isEmpty)
+sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment
'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
+ """.stripMargin)
+for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12"))
{
+ sql(
+
[GitHub] spark pull request #13704: [SPARK-15985][SQL] Eliminate redundant cast from ...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/13704#discussion_r76721154
--- Diff:
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/SimplifyCastsSuite.scala
---
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.catalyst.optimizer
+
+import org.apache.spark.sql.catalyst.dsl._
+import org.apache.spark.sql.catalyst.dsl.expressions._
+import org.apache.spark.sql.catalyst.dsl.plans._
+import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.plans.PlanTest
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.RuleExecutor
+import org.apache.spark.sql.types._
+
+class SimplifyCastsSuite extends PlanTest {
+
+ object Optimize extends RuleExecutor[LogicalPlan] {
+val batches = Batch("SimplifyCasts", FixedPoint(50), SimplifyCasts) ::
Nil
+ }
+
+ test("non-nullable to non-nullable array cast") {
+val input = LocalRelation('a.array(ArrayType(IntegerType, false)))
+val plan = input.select('a.cast(ArrayType(IntegerType,
false)).as("casted")).analyze
+val optimized = Optimize.execute(plan)
+val expected = input.select('a.as("casted")).analyze
+comparePlans(optimized, expected)
+ }
+
+ test("non-nullable to nullable array cast") {
--- End diff --
`non-nullable element array to nullable element array cast`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76720950
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
+Option(classOf[LazySimpleSerDe].getCanonicalName))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE a data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
Some("default")))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+ // The table type of the source table should be a Hive-managed data
source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+val path = dir.getCanonicalPath
+spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH
'$path')")
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+// The source table should be an external data source table
+val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+// The table type of the source table should be an external data
source table
+assert(DDLUtils.isDatasourceTable(sourceTable))
+assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+checkCreateTableLike(sourceTable, targetTable)
+ }
+}
+ }
+
+ test("CREATE TABLE LIKE a managed Hive serde table") {
+val catalog = spark.sessionState.catalog
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS
SELECT 1 key, 'a'")
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable =
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external Hive serde table") {
+val catalog = spark.sessionState.catalog
+withTempDir { tmpDir =>
+ val basePath = tmpDir.getCanonicalPath
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+assert(tmpDir.listFiles.isEmpty)
+sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment
'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
+ """.stripMargin)
+for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12"))
{
+ sql(
+
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76720856
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
+Option(classOf[LazySimpleSerDe].getCanonicalName))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE a data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
Some("default")))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+ // The table type of the source table should be a Hive-managed data
source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+val path = dir.getCanonicalPath
+spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH
'$path')")
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+// The source table should be an external data source table
+val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+// The table type of the source table should be an external data
source table
+assert(DDLUtils.isDatasourceTable(sourceTable))
+assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+checkCreateTableLike(sourceTable, targetTable)
+ }
+}
+ }
+
+ test("CREATE TABLE LIKE a managed Hive serde table") {
+val catalog = spark.sessionState.catalog
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS
SELECT 1 key, 'a'")
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable =
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external Hive serde table") {
+val catalog = spark.sessionState.catalog
+withTempDir { tmpDir =>
+ val basePath = tmpDir.getCanonicalPath
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+assert(tmpDir.listFiles.isEmpty)
+sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment
'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
+ """.stripMargin)
+for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12"))
{
+ sql(
+
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76720777
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
+Option(classOf[LazySimpleSerDe].getCanonicalName))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE a data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.write.format("json").saveAsTable(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
Some("default")))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+ // The table type of the source table should be a Hive-managed data
source table
+ assert(DDLUtils.isDatasourceTable(sourceTable))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external data source table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ withTempPath { dir =>
+val path = dir.getCanonicalPath
+spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+ .write.format("parquet").save(path)
+sql(s"CREATE TABLE $sourceTabName USING parquet OPTIONS (PATH
'$path')")
+sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+// The source table should be an external data source table
+val sourceTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(sourceTabName, Some("default")))
+val targetTable = spark.sessionState.catalog.getTableMetadata(
+ TableIdentifier(targetTabName, Some("default")))
+// The table type of the source table should be an external data
source table
+assert(DDLUtils.isDatasourceTable(sourceTable))
+assert(sourceTable.tableType == CatalogTableType.EXTERNAL)
+
+checkCreateTableLike(sourceTable, targetTable)
+ }
+}
+ }
+
+ test("CREATE TABLE LIKE a managed Hive serde table") {
+val catalog = spark.sessionState.catalog
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ sql(s"CREATE TABLE $sourceTabName TBLPROPERTIES('prop1'='value1') AS
SELECT 1 key, 'a'")
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
catalog.getTableMetadata(TableIdentifier(sourceTabName, Some("default")))
+ assert(sourceTable.tableType == CatalogTableType.MANAGED)
+ assert(sourceTable.properties.get("prop1").nonEmpty)
+ val targetTable =
catalog.getTableMetadata(TableIdentifier(targetTabName, Some("default")))
+
+ checkCreateTableLike(sourceTable, targetTable)
+}
+ }
+
+ test("CREATE TABLE LIKE an external Hive serde table") {
+val catalog = spark.sessionState.catalog
+withTempDir { tmpDir =>
+ val basePath = tmpDir.getCanonicalPath
+ val sourceTabName = "tab1"
+ val targetTabName = "tab2"
+ withTable(sourceTabName, targetTabName) {
+assert(tmpDir.listFiles.isEmpty)
+sql(
+ s"""
+ |CREATE EXTERNAL TABLE $sourceTabName (key INT comment
'test', value STRING)
+ |COMMENT 'Apache Spark'
+ |PARTITIONED BY (ds STRING, hr STRING)
+ |LOCATION '$basePath'
+ """.stripMargin)
+for (ds <- Seq("2008-04-08", "2008-04-09"); hr <- Seq("11", "12"))
{
+ sql(
+
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76720758
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
--- End diff --
i see, we should leave a comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request #14531: [SPARK-16943] [SPARK-16942] [SQL] Fix multiple bu...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/14531#discussion_r76720688
--- Diff:
sql/hive/src/test/scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala
---
@@ -620,6 +623,219 @@ class HiveDDLSuite
}
}
+ test("CREATE TABLE LIKE a temporary table") {
+val sourceTabName = "tab1"
+val targetTabName = "tab2"
+withTable(sourceTabName, targetTabName) {
+ spark.range(10).select('id as 'a, 'id as 'b, 'id as 'c, 'id as 'd)
+.createTempView(sourceTabName)
+ sql(s"CREATE TABLE $targetTabName LIKE $sourceTabName")
+
+ val sourceTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(sourceTabName,
None))
+ val targetTable =
+
spark.sessionState.catalog.getTableMetadata(TableIdentifier(targetTabName,
Some("default")))
+
+ assert(targetTable.storage.serde ==
--- End diff --
Since the source table is temporary view, we are unable to get the serde
from the source table and thus we choose the default one. Here, it is for
confirming that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #13599: [SPARK-13587] [PYSPARK] Support virtualenv in pyspark
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13599 **[Test build #64617 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/64617/consoleFull)** for PR 13599 at commit [`31cb42c`](https://github.com/apache/spark/commit/31cb42cfceb937941373caa76fe602abdfca3c9a). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
