cyanzheng2926 opened a new pull request, #6576:
URL: https://github.com/apache/hive/pull/6576
/<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA:
https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g.,
'[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a
faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section
is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster
reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class
hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other
DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
When stats collection is skipped for some columns with
hive.stats.reliable=false, no stats are written to the metastore, but
COLUMN_STATS_ACCURATE still remains "true" for these columns.
Updated ColStatsProcessor to track for columns that failed stats gathering
due to such errors, and remove the stale accurate markers for these columns.
Note on limitations:
- Non-native partitioned tables (e.g. Iceberg) only table-level cleanup is
done, since currently there is no storage-handler API to clear partition-level
COLUMN_STATS_ACCURATE.
- Marker removal is using alterPartitions and alterTable with
transactional=false, since these are parameter-only metadata updates, and the
pre-existing setOrRemoveColumnStatsAccurateProperty success path is currently
implemented in the same way.
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
Column stats collection can fail for individual columns (e.g.
UnsupportedDoubleException with Infinity/NaN). With hive.stats.reliable=false
the failure is logged and skipped, but still leaves COLUMN_STATS_ACCURATE
claiming those columns are accurate even though no stats were stored. The
optimizer may use non-existent stats and produce suboptimal or incorrect plans
as a result.
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as
the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes
- provide the console output, description, screenshot and/or a reproducable
example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to
the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
Yes.
Before: After a partial or total column-stats failure, DESCRIBE FORMATTED
<table> <column> could show COLUMN_STATS_ACCURATE with "column":"true" even
when COLUMN_STATS was missing for that column.
After: Failed columns are removed from COLUMN_STATS_ACCURATE. Only columns
with successfully collected stats remain marked accurate.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some
test cases that check the changes thoroughly including negative and positive
cases if possible.
If it was tested in a way different from regular unit tests, please clarify
how you tested step by step, ideally copy and paste-able, so that other
reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why
it was difficult to add.
-->
Added qtest: stats_col_stats_inaccurate.q and corresponding golden, test for
multiple cases when certain columns of partitioned/non-partitioned tables fail
at stats gathering with Infinity/NaN, and desc formatted output should not
return COLUMN_STATS_ACCURATE markers for these columns
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
