Hi All,

I am looking for a committer to review
PR: https://github.com/apache/hive/pull/6505
JIRA: https://issues.apache.org/jira/browse/HIVE-29625

ColStatistics.countDistinct currently overloads 0: a genuinely
verified zero (all-NULL column, empty table) is indistinguishable
from "NDV was never computed", which leaks through as the Thrift
default. Consumers therefore apply the same fallback heuristics
(numRows/2, factor *= 0.5, ...) to both cases - wrong for verified
zero, and masking absent information for unknown. The PR establishes
that a negative countDistinct means "unknown", parallel to the
historical "negative = unknown" convention for
numNulls/numTrues/numFalses, which HIVE-29438 reinforced. It follows
the direction suggested during review of PR 6359:
https://github.com/apache/hive/pull/6359#discussion_r2994697977
as "In fact, everything would be simpler if we could use -1 for NDV to
declare unknown as it happens for the other stats.
This is probably a bigger change to
digest so let's not go into this direction for now." since it ultimately
appears as the most pragmatic approach to solve a large class of NDV
problems.

The PR is not a draft, precommit and SonarCloud are green on the
current head, and it is 8 production files plus 13 test files with
no open review threads. It generalizes and supersedes my earlier
point fix in PR 6418 (HIVE-29556), which I will close when/if this
direction is confirmed.

Happy to address any feedback.

Thanks,
Konstantin

Reply via email to