[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14536905#comment-14536905 ] Lefty Leverenz commented on HIVE-10454: --- Doc note: I revised the wiki's description of *hive.mapred.mode* to point to this issue for an example. * [Configuration Properties -- hive.mapred.mode | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.mapred.mode] Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14528436#comment-14528436 ] Aihua Xu commented on HIVE-10454: - It makes sense. Seems before I misunderstood Xuefu's point. I will resolve as won't fix then. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14526982#comment-14526982 ] Aihua Xu commented on HIVE-10454: - [~ashutoshc] Can you help review this patch as well? Currently we only check TableScanOperator if it has any ColumnExpr. If the predicates are not pushed down, then you can't run the query in strict mode. With the patch, we will check TableScanOperator and the children to see if they have any column match any partition key. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527047#comment-14527047 ] Ashutosh Chauhan commented on HIVE-10454: - Can you create a RB entry for it, that makes review little easier. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14527076#comment-14527076 ] Ashutosh Chauhan commented on HIVE-10454: - I see point Xuefu is making and I agree with him. For query you pointed : {code:sql} select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {code} Your point is in query user has given us has a predicate on partitioning column, so exception should not be thrown in strict mode. But the spirit of strict mode is not whether predicate is given or not, it is to stop user from doing full table scans (reading all rows from disk IO) which will happen in this case, since partition pruner cant handle this expression (reason being udf being underministic). So, I think we need not to check for filter any where else other than on TS. When exception is thrown in such cases, its telling user that Hive is undertaking full table scan, which is disallowed under strict mode. He has two options at this point. He need to rewrite his filter to make it more strict (by making it deterministic) or get out of strict mode. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523796#comment-14523796 ] Aihua Xu commented on HIVE-10454: - I reverted the alias change in PrunerOperatorFactory.java since seems it will cause a different issue. Will file separate jira to fix it. {noformat} select * from (select count(1) from t1 join t2 on t1.key=t2.key where t1.ds='1') t3; FAILED: SemanticException [Error 10041]: No partition predicate found for Alias t3:t1 Table t2 {noformat} The alias should be t3:t2. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523625#comment-14523625 ] Aihua Xu commented on HIVE-10454: - Submitted new patch to save the partition filter status for reuse. Also fixed the problem of the alias for the table (we were always using the first table alias). Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.2.patch, HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523214#comment-14523214 ] Aihua Xu commented on HIVE-10454: - Before the change, we only check predicates on TableScan operator. PPD will optimize to push the predicates down to TableScan in the most cases. If ppd optimization is turned off or the predicates are nondeterministic, then predicates are not pushed down to TableScan. See the following example, {noformat} TableScan alias: srcpart Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats: NONE Filter Operator predicate: (time to_date(date_add(from_unixtime(unix_timestamp()), 1))) (type: boolean) {noformat} Before the change, since we only check TableScan operator, then it will always throw No partition predicate found exception. After the change, we will also check the child Filter Operator and then it won't throw exceptions. This predicate check tells the user if the query contains partition predicates only and the actual partition pruning logic doesn't change. I'm still checking input39.q unit test. It could be related to my change. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522169#comment-14522169 ] Aihua Xu commented on HIVE-10454: - The failures seems unrelated to the patch. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14522383#comment-14522383 ] Xuefu Zhang commented on HIVE-10454: Patch looks clear in logic. Since we are now going thru the whole operator tree to determine if a partition column is every referenced in an expression, I'm not sure if such expression will ALWAYS end up being used for partition pruning. [~aihuaxu], it would be great if you can explain. I'm not an expert on this, so others please chime in if you know more. Thanks. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520166#comment-14520166 ] Aihua Xu commented on HIVE-10454: - Two unit tests failure are related. Fixed in the new patch. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518433#comment-14518433 ] Hive QA commented on HIVE-10454: {color:red}Overall{color}: -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12728785/HIVE-10454.patch {color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 8825 tests executed *Failed tests:* {noformat} TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more - did not produce a TEST-*.xml file TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a TEST-*.xml file TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_alter_view_failure6 org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_input_part0_neg org.apache.hive.jdbc.TestSSL.testSSLVersion {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3633/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/3633/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-3633/ Messages: {noformat} Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 17 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12728785 - PreCommit-HIVE-TRUNK-Build Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu Attachments: HIVE-10454.patch The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509254#comment-14509254 ] Aihua Xu commented on HIVE-10454: - That condition doesn't check how many partitions will be involved. It's just reminding you that you need to provide predicates. We will always have such issue with nondeterministic UDF like unix_timstamp(), even with the query like: select * from t1 where t1.c2 = to_date(date_add(from_unixtime( unix_timestamp() ),1)); For predicate with nondeterministic UDF, the predicate won't be pushed down to TableScanOperator, but currently we only check if TableScanOperator has predicate. So we need not only check if TableScanOperator has predicates but also the child ops (e.g., FilterOperator) to determine if the table has predicate. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509545#comment-14509545 ] Aihua Xu commented on HIVE-10454: - In this query, I'm not filtering the rows but filtering the partitions, so we won't scan all the partitions. Strict mode by the definition should allow such query. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509403#comment-14509403 ] Xuefu Zhang commented on HIVE-10454: I think the point of strict mode is to prevent full scan all partitions of a table. In your case, while rows are filtered, the scanner will have to scan all partitions, which should be prevented by the virtue of the strict mode. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10454) Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified.
[ https://issues.apache.org/jira/browse/HIVE-10454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14509084#comment-14509084 ] Xuefu Zhang commented on HIVE-10454: I'm not sure if in strict mode an equal condition on partition column is expected. Otherwise, the query can still span to all or a large number of partitions. Query against partitioned table in strict mode failed with No partition predicate found even if partition predicate is specified. --- Key: HIVE-10454 URL: https://issues.apache.org/jira/browse/HIVE-10454 Project: Hive Issue Type: Bug Reporter: Aihua Xu Assignee: Aihua Xu The following queries fail: {noformat} create table t1 (c1 int) PARTITIONED BY (c2 string); set hive.mapred.mode=strict; select * from t1 where t1.c2 to_date(date_add(from_unixtime( unix_timestamp() ),1)); {noformat} The query failed with No partition predicate found for alias t1. -- This message was sent by Atlassian JIRA (v6.3.4#6332)