[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14485298#comment-14485298
]
Hive QA commented on HIVE-10122:
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12723749/HIVE-10122.patch
{color:red}ERROR:{color} -1 due to 171 failed/errored test(s), 8663 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver-bucketmapjoin6.q-constprog_partitioner.q-infer_bucket_sort_dyn_part.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-external_table_with_space_in_location_path.q-infer_bucket_sort_merge.q-auto_sortmerge_join_16.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-groupby2.q-import_exported_table.q-bucketizedhiveinputformat.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-index_bitmap3.q-stats_counter_partitioned.q-temp_table_external.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_map_operators.q-join1.q-bucketmapjoin7.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_num_buckets.q-disable_merge_for_bucketing.q-uber_reduce.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-infer_bucket_sort_reducers_power_two.q-scriptfile1.q-scriptfile1_win.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-leftsemijoin_mr.q-load_hdfs_file_with_space_in_the_name.q-root_dir_external_table.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-list_bucket_dml_10.q-bucket_num_reducers.q-bucket6.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-load_fs2.q-file_with_header_footer.q-ql_rewrite_gbtoidx_cbo_1.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-parallel_orderby.q-reduce_deduplicate.q-ql_rewrite_gbtoidx_cbo_2.q-and-1-more
- did not produce a TEST-*.xml file
TestMinimrCliDriver-ql_rewrite_gbtoidx.q-smb_mapjoin_8.q - did not produce a
TEST-*.xml file
TestMinimrCliDriver-schemeAuthority2.q-bucket4.q-input16_cc.q-and-1-more - did
not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_11
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_date
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_avro_timestamp
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketcontext_8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketizedhiveinputformat_auto
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucketmapjoin_negative2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_views
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_auto_mult_tables
[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484218#comment-14484218
]
Sergey Shelukhin commented on HIVE-10122:
-
https://reviews.apache.org/r/32941/
Hive metastore filter-by-expression is broken for non-partition expressions
---
Key: HIVE-10122
URL: https://issues.apache.org/jira/browse/HIVE-10122
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10122.patch
See
https://issues.apache.org/jira/browse/HIVE-10091?focusedCommentId=14382413page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14382413
These two lines of code
{noformat}
// Replace virtual columns with nulls. See javadoc for details.
prunerExpr = removeNonPartCols(prunerExpr, extractPartColNames(tab),
partColsUsedInFilter);
// Remove all parts that are not partition columns. See javadoc for
details.
ExprNodeDesc compactExpr = compactExpr(prunerExpr.clone());
{noformat}
are supposed to take care of this; I see there were bunch of changes to this
code over some time, and now it appears to be broken.
Thanks to [~thejas] for info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14484280#comment-14484280
]
Thejas M Nair commented on HIVE-10122:
--
yeah, it would be really good to have unit tests for these. That would have
detected this regression in the first place.
If logs need to be looked at, that is also possible. See my changes to TestHive
in HIVE-10214, its easy to capture log output from specific classes.
Hive metastore filter-by-expression is broken for non-partition expressions
---
Key: HIVE-10122
URL: https://issues.apache.org/jira/browse/HIVE-10122
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin
Attachments: HIVE-10122.patch
See
https://issues.apache.org/jira/browse/HIVE-10091?focusedCommentId=14382413page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14382413
These two lines of code
{noformat}
// Replace virtual columns with nulls. See javadoc for details.
prunerExpr = removeNonPartCols(prunerExpr, extractPartColNames(tab),
partColsUsedInFilter);
// Remove all parts that are not partition columns. See javadoc for
details.
ExprNodeDesc compactExpr = compactExpr(prunerExpr.clone());
{noformat}
are supposed to take care of this; I see there were bunch of changes to this
code over some time, and now it appears to be broken.
Thanks to [~thejas] for info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384874#comment-14384874
]
Mostafa Mokhtar commented on HIVE-10122:
[~sershe] [~thejas] [~hagleitn]
I ran explain for this query
{code}
select
ss_item_sk rowcount
from
store_sales
where
ss_sold_date_sk between 2450816 and 2450817
and ss_ticket_number 1
and ss_item_sk 50;
{code}
And the query that gets issues to MySQL looks correct to me as only the
qualified partitions are queried.
What am I missing?
{code}
select
COLUMN_NAME,
COLUMN_TYPE,
min(LONG_LOW_VALUE),
max(LONG_HIGH_VALUE),
min(DOUBLE_LOW_VALUE),
max(DOUBLE_HIGH_VALUE),
min(BIG_DECIMAL_LOW_VALUE),
max(BIG_DECIMAL_HIGH_VALUE),
sum(NUM_NULLS),
max(NUM_DISTINCTS),
max(AVG_COL_LEN),
max(MAX_COL_LEN),
sum(NUM_TRUES),
sum(NUM_FALSES)
from
PART_COL_STATS
where
DB_NAME = 'tpcds_bin_partitioned_orc_3'
and TABLE_NAME = 'store_sales'
and COLUMN_NAME in ('ss_item_sk' , 'ss_ticket_number')
and PARTITION_NAME in ('ss_sold_date_sk=2450816' ,
'ss_sold_date_sk=2450817')
group by COLUMN_NAME , COLUMN_TYPE
{code}
Hive metastore filter-by-expression is broken for non-partition expressions
---
Key: HIVE-10122
URL: https://issues.apache.org/jira/browse/HIVE-10122
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Sergey Shelukhin
See
https://issues.apache.org/jira/browse/HIVE-10091?focusedCommentId=14382413page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14382413
These two lines of code
{noformat}
// Replace virtual columns with nulls. See javadoc for details.
prunerExpr = removeNonPartCols(prunerExpr, extractPartColNames(tab),
partColsUsedInFilter);
// Remove all parts that are not partition columns. See javadoc for
details.
ExprNodeDesc compactExpr = compactExpr(prunerExpr.clone());
{noformat}
are supposed to take care of this; I see there were bunch of changes to this
code over some time, and now it appears to be broken.
Thanks to [~thejas] for info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10122) Hive metastore filter-by-expression is broken for non-partition expressions
[
https://issues.apache.org/jira/browse/HIVE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14384880#comment-14384880
]
Sergey Shelukhin commented on HIVE-10122:
-
That is stats; do you see MySQL queries to PARTITIONS table?
Hive metastore filter-by-expression is broken for non-partition expressions
---
Key: HIVE-10122
URL: https://issues.apache.org/jira/browse/HIVE-10122
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.14.0, 1.0.0, 1.1.0
Reporter: Sergey Shelukhin
See
https://issues.apache.org/jira/browse/HIVE-10091?focusedCommentId=14382413page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14382413
These two lines of code
{noformat}
// Replace virtual columns with nulls. See javadoc for details.
prunerExpr = removeNonPartCols(prunerExpr, extractPartColNames(tab),
partColsUsedInFilter);
// Remove all parts that are not partition columns. See javadoc for
details.
ExprNodeDesc compactExpr = compactExpr(prunerExpr.clone());
{noformat}
are supposed to take care of this; I see there were bunch of changes to this
code over some time, and now it appears to be broken.
Thanks to [~thejas] for info.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
