[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14498 Merging in master/branch-2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/68268/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #68268 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68268/consoleFull)** for PR 14498 at commit [`05390ad`](https://github.com/apache/spark/commit/05390ade4f9af7ecc66d365683657d2689e56157). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14498 I'm ok with not having them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 Based on the previous discussions in the other PRs, it sounds like these Hive-specific test cases are not very useful. Do we still need to port them back? ``` "auto_join19", "auto_join22", "auto_join25", "auto_join26", "auto_join27", "auto_join28", "auto_join30", "auto_join31", "auto_join_nulls", "auto_join_reordering_values", "correlationoptimizer1", "correlationoptimizer2", "correlationoptimizer3", "correlationoptimizer4", "multiMapJoin1", "orc_dictionary_threshold", "udf_hash" ``` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14498 If this is no longer WIP, please update the title. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #68268 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/68268/consoleFull)** for PR 14498 at commit [`05390ad`](https://github.com/apache/spark/commit/05390ade4f9af7ecc66d365683657d2689e56157). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 @rxin Sure, will do it this weekend. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user rxin commented on the issue: https://github.com/apache/spark/pull/14498 @gatorsmile can we take this to the finish line? This actually does not impact hash at all since it's only here for testing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 @tejasapatil Thanks for the inputs! I can understand your usage cases. It looks very reasonable. Actually, in Spark 2.0, we already removed it. You are unable to load/invoke the Hive's hash function. This PR is just to clean the usage of `hash` function in our testing package. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user tejasapatil commented on the issue: https://github.com/apache/spark/pull/14498 Is Spark's hashing function semantically equivalent to Hive's ? AFAIK, its not. I think it would be better to have a mode to be able to use Hive's hash method. eg. case when this would be needed: Users running a query in Hive want to switch to Spark. As this happens, you want to verify if the data produced is same or not. Also, for a brief time the pipeline would run in both the engines. Upstream consumers of the data generated should not see differences due to running in the different engines --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 #14472 has been merged. Will work on it soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 Yeah, we should check all the disabled test cases and see if we can move it. I am waiting for https://github.com/apache/spark/pull/14472. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/14498 This seems fair. Shouldn't we do something about all the disabled test cases though? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63247/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #63247 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63247/consoleFull)** for PR 14498 at commit [`3b3f3c8`](https://github.com/apache/spark/commit/3b3f3c8543a33d87bf53aa4f2945a34f6b7d0786). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #63247 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63247/consoleFull)** for PR 14498 at commit [`3b3f3c8`](https://github.com/apache/spark/commit/3b3f3c8543a33d87bf53aa4f2945a34f6b7d0786). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/14498 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/63240/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #63240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63240/consoleFull)** for PR 14498 at commit [`3b3f3c8`](https://github.com/apache/spark/commit/3b3f3c8543a33d87bf53aa4f2945a34f6b7d0786). * This patch **fails to build**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/14498 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #14498: [SPARK-16904] [SQL] Removal of Hive Built-in Hash Functi...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/14498 **[Test build #63240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/63240/consoleFull)** for PR 14498 at commit [`3b3f3c8`](https://github.com/apache/spark/commit/3b3f3c8543a33d87bf53aa4f2945a34f6b7d0786). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
