Yes ColStatistics is in Hive and it holds all statistics about the columns.
On 2018/05/03 16:26:02, Julian Hyde <[email protected]> wrote: > It depends on the statistic. Most of them are approximate. > > It’s the "garbage in, garbage out" principle. An exact statistic may be of a > bit more (or a lot more) use to the consumer of the statistic, but is more > effort for the producer of the statistic. > > RelMdMaxRowCount is one of the few exact ones. If RelMdMaxRowCount says 10, > the relation might return 0 rows or 9 rows or 10 rows but never 11 rows. > > RelMdPredicates and is also exact (albeit not numeric). RelMdUniqueKeys is > exact (which is to say, it returns a key, it is definitely unique; there may > be some unique keys that it does not know about). > > I don’t know what ColStatistics is. Is it a Hive thing? I surmise that is it > based on RelMdRowCount, which is approximate. > > Julian > > > > On May 3, 2018, at 5:41 AM, Valli Annamalai <[email protected]> wrote: > > > > In Hive, column statistics like countDistinct, isPrimaryKey, etc.are need > > to be set. While doing so, in Hive, the following function sets primary key > > to true based on a assumption. > > > > > > public static void inferAndSetPrimaryKey(long numRows, > > List<ColStatistics> colStats) { > > if (colStats != null) { > > for (ColStatistics cs : colStats) { > > if (cs != null && cs.getCountDistint() >= numRows) { > > cs.setPrimaryKey(true); > > } > > else if (cs != null && cs.getRange() != null && > > cs.getRange().minValue != null && > > cs.getRange().maxValue != null) { > > if (numRows == > > ((cs.getRange().maxValue.longValue() - > > cs.getRange().minValue.longValue()) + 1)) { > > cs.setPrimaryKey(true); > > } > > } > > } > > } > > } > > > > If this is the case, considering I have only 2 values filled over the > > entire column, which are 1 and 1000, and 1000 is the numRows, then having > > primary key as true would be wrong. While planning, if suppose aggregation > > is the upcoming node, then that node need not be proceeded, considering > > primary key column will have only unique values. > > > > If we are assuming as above function to set primary key and if calcite also > > proceed with these assumptions, then the result will also be wrong. So how > > this could be solved? > > > > Similarly for count distinct also, is it okay to give approximate values to > > calcite? > >
