[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 Oh, thank you, @cloud-fan ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17266 thanks, merging to master/2.1! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 For non-error message cases, incorrect result is also a problem in this issue. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 The following is the error message. Since we are not escaping in the spark master, the behavior (incorrect filtering or the error message) is the same from the master branch Spark. ``` java.lang.RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive. You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem, however this will result in degraded performance. Please report a bug: https://issues.apache.org/jira/browse/SPARK at org.apache.spark.sql.hive.client.Shim_v0_13.getPartitionsByFilter(HiveShim.scala:612) ... Caused by: java.lang.reflect.InvocationTargetException: org.apache.hadoop.hive.metastore.api.MetaException: Error parsing partition filter : line 1:8 mismatched character '' expecting '"' at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ... at org.apache.spark.sql.hive.client.Shim_v0_13.getPartitionsByFilter(HiveShim.scala:599) ... 103 more Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Error parsing partition filter : line 1:8 mismatched character '' expecting '"' at org.apache.hadoop.hive.metastore.ObjectStore.getFilterParser(ObjectStore.java:2569) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilterInternal(ObjectStore.java:2512) at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsByFilter(ObjectStore.java:2335) ... org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_partitions_by_filter(HiveMetaStore.java:4442) ... org.apache.hadoop.hive.metastore.HiveMetaStoreClient.listPartitionsByFilter(HiveMetaStoreClient.java:1103) ... at org.apache.hadoop.hive.ql.metadata.Hive.getPartitionsByFilter(Hive.java:2254) ... 108 more ``` HIVE-11723 seems to resove that in SemanticAnalyzer. So, I need to try that soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17266 can we say something more in the error message? We should explain that it's a hive bug. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 For HIVE-11723, it resolved it in [SemanticAnalyzer](https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L1002-L1003). I think it's possible to bring that into our @JoshRosen's [repo](https://github.com/JoshRosen/hive/blob/release-1.2.1-spark2/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java#L858-L859). Let's me backport that to see this is enough. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 Sure! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17266 Could you do more investigation about the impact of the following two Hive JIRAs? https://issues.apache.org/jira/browse/HIVE-11723 https://issues.apache.org/jira/browse/HIVE-2943 Thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 #17271 failed as expected. Hive API does not handle the filters with escaped string, e.g. two escaped chars like `\"\"`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 Yep. I see. Thank for the guide, @gatorsmile ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17266 Ok, how about submitting a separate PR by escaping the string? You can show the reviewers the failure cases there. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 If you want, I will make some other failure test cases in this PR. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 It's not bug of Hive CLI, it seems a limitation of that API. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17266 Could you try Hive to double check it? Is this a bug in Hive? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 Yes. What I meant was that is not supported correctly from Hive documentation. For the mixed case, all combination of the escaping fails. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17266 Have you tried C-style escaping? `"'` -> `\"\'` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 BTW, I forgot to thank you. :) Thank you for review. For the non-mixed cases, I think we don't need to escape. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue:
https://github.com/apache/spark/pull/17266
Based on Hive' doc
(https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types):
> String literals can be expressed with either single quotes (') or double
quotes ("). Hive uses C-style escaping within the strings.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17266 Yep. I tried escaping first, but it doesn't work inside Hive side. Also, I tried `'"'"'"` for `"'` because it works in Hive. But, for filter parsing, it fails at parsing Lexer layer. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17266 We should add the escape, instead of adding quotes. Right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17266 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17266 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/74400/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17266 **[Test build #74400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74400/testReport)** for PR 17266 at commit [`37bff5d`](https://github.com/apache/spark/commit/37bff5dfbe7dd8c2b3f650915107bdb84a4b7fac). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #17266: [SPARK-19912][SQL] String literals should be escaped for...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17266 **[Test build #74400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/74400/testReport)** for PR 17266 at commit [`37bff5d`](https://github.com/apache/spark/commit/37bff5dfbe7dd8c2b3f650915107bdb84a4b7fac). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
