[jira] [Created] (HIVE-24764) insert overwrite on a partition resets row count stats in other partitions

[Rajesh Balamohan (Jira)]

Rajesh Balamohan created HIVE-24764:
---------------------------------------

             Summary: insert overwrite on a partition resets row count stats in 
other partitions
                 Key: HIVE-24764
                 URL: https://issues.apache.org/jira/browse/HIVE-24764
             Project: Hive
          Issue Type: Bug
            Reporter: Rajesh Balamohan


After insert overwrite on a partition, stats on other partitions are messed up. 
Subsequent queries end up with plans with PARTIAL stats. In certain cases, this 
leads to suboptimal query plans.

{noformat}

drop table if exists test_stats;
drop table if exists test_stats_2;

create table test_stats(i int, j bigint);
create table test_stats_2(i int) partitioned by (j bigint);

insert into test_stats values (1, 1), (2, 2), (3, 3), (4, 4), (5, NULL);

-- select * from test_stats;
1   1
2   2
3   3
4   4
5   <null>


insert overwrite table test_stats_2 partition(j)  select i, j from test_stats 
where j is not null;


-- After executing this statement, stat gets messed up.
insert overwrite table test_stats_2 partition(j)  select i, j from test_stats 
where j is null;

-- select * from test_stats_2;
1   1
2   2
3   3
4   4
5   <null>


-- This would return "PARTIAL" stats instead of "COMPLETE"
explain select i, count(*) as c from test_stats_2 group by i order by c desc 
limit 10;


STAGE DEPENDENCIES:
  Stage-1 is a root stage
  Stage-0 depends on stages: Stage-1

STAGE PLANS:
  Stage: Stage-1
    Tez
      DagId: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
      Edges:
        Reducer 2 <- Map 1 (SIMPLE_EDGE)
        Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
      DagName: hive_20210208093110_62ced99e-f068-42d4-9ba8-d45fccd6c0a2:68
      Vertices:
        Map 1 
            Map Operator Tree:
                TableScan
                  alias: test_stats_2
                  Statistics: Num rows: 125 Data size: 500 Basic stats: PARTIAL 
Column stats: COMPLETE
                  Select Operator
                    expressions: i (type: int)
                    outputColumnNames: i
                    Statistics: Num rows: 125 Data size: 500 Basic stats: 
PARTIAL Column stats: COMPLETE
                    Group By Operator
                      aggregations: count()
                      keys: i (type: int)
                      minReductionHashAggr: 0.99
                      mode: hash
                      outputColumnNames: _col0, _col1
                      Statistics: Num rows: 125 Data size: 1500 Basic stats: 
PARTIAL Column stats: COMPLETE
                      Reduce Output Operator
                        key expressions: _col0 (type: int)
                        null sort order: a
                        sort order: +
                        Map-reduce partition columns: _col0 (type: int)
                        Statistics: Num rows: 125 Data size: 1500 Basic stats: 
PARTIAL Column stats: COMPLETE
                        value expressions: _col1 (type: bigint)
            Execution mode: vectorized, llap
            LLAP IO: may be used (ACID table)
        Reducer 2 
            Execution mode: vectorized, llap
            Reduce Operator Tree:
              Group By Operator
                aggregations: count(VALUE._col0)
                keys: KEY._col0 (type: int)
                mode: mergepartial
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
                Top N Key Operator
                  sort order: -
                  keys: _col1 (type: bigint)
                  null sort order: a
                  Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
                  top n: 10
                  Reduce Output Operator
                    key expressions: _col1 (type: bigint)
                    null sort order: a
                    sort order: -
                    Statistics: Num rows: 62 Data size: 744 Basic stats: 
PARTIAL Column stats: COMPLETE
                    TopN Hash Memory Usage: 0.04
                    value expressions: _col0 (type: int)
        Reducer 3 
            Execution mode: vectorized, llap
            Reduce Operator Tree:
              Select Operator
                expressions: VALUE._col0 (type: int), KEY.reducesinkkey0 (type: 
bigint)
                outputColumnNames: _col0, _col1
                Statistics: Num rows: 62 Data size: 744 Basic stats: PARTIAL 
Column stats: COMPLETE
                Limit
                  Number of rows: 10
                  Statistics: Num rows: 10 Data size: 120 Basic stats: PARTIAL 
Column stats: COMPLETE
                  File Output Operator
                    compressed: false
                    Statistics: Num rows: 10 Data size: 120 Basic stats: 
PARTIAL Column stats: COMPLETE
                    table:
                        input format: 
org.apache.hadoop.mapred.SequenceFileInputFormat
                        output format: 
org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat
                        serde: 
org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe

  Stage: Stage-0
    Fetch Operator
      limit: 10
      Processor Tree:
        ListSink
{noformat}

https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/stats/fs/FSStatsAggregator.java#L138

FSStats should return "null" or "" if partitions are not present in its list.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)