[
https://issues.apache.org/jira/browse/HIVE-9560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14302891#comment-14302891
]
Xin Hao commented on HIVE-9560:
-------------------------------
For example, we have an ORC table named 'item'.
(a) Before running 'analyze table item compute statistics;',
the 'rawDataSize' was '884720592'.
The result of 'describe extended item':
Detailed Table Information Table(tableName:item, dbName:bigbenchorc,
owner:root, createTime:1421984899, lastAccessTime:0, retention:0,
sd:StorageDescriptor(cols:[FieldSchema(name:i_item_sk, type:bigint,
comment:null), FieldSchema(name:i_item_id, type:string, comment:null),
FieldSchema(name:i_rec_start_date, type:string, comment:null),
FieldSchema(name:i_rec_end_date, type:string, comment:null),
FieldSchema(name:i_item_desc, type:string, comment:null),
FieldSchema(name:i_current_price, type:double, comment:null),
FieldSchema(name:i_wholesale_cost, type:double, comment:null),
FieldSchema(name:i_brand_id, type:int, comment:null), FieldSchema(name:i_brand,
type:string, comment:null), FieldSchema(name:i_class_id, type:int,
comment:null), FieldSchema(name:i_class, type:string, comment:null),
FieldSchema(name:i_category_id, type:int, comment:null),
FieldSchema(name:i_category, type:string, comment:null),
FieldSchema(name:i_manufact_id, type:int, comment:null),
FieldSchema(name:i_manufact, type:string, comment:null),
FieldSchema(name:i_size, type:string, comment:null),
FieldSchema(name:i_formulation, type:string, comment:null),
FieldSchema(name:i_color, type:string, comment:null), FieldSchema(name:i_units,
type:string, comment:null), FieldSchema(name:i_container, type:string,
comment:null), FieldSchema(name:i_manager_id, type:int, comment:null),
FieldSchema(name:i_product_name, type:string, comment:null)],
location:hdfs://bhx1:8020/user/hive/warehouse/bigbenchorc.db/item,
inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde,
parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
partitionKeys:[], parameters:{numFiles=4, transient_lastDdlTime=1421984899,
COLUMN_STATS_ACCURATE=true, totalSize=83267548, numRows=563518,
rawDataSize=884720592}, viewOriginalText:null, viewExpandedText:null,
tableType:MANAGED_TABLE)
Time taken: 0.527 seconds, Fetched: 24 row(s)
(b)After running 'analyze table TABLE_NAME compute statistics;'
the 'rawDataSize' will be changed to '0',
The result of 'describe extended item':
Detailed Table Information Table(tableName:item, dbName:bigbenchorc,
owner:root, createTime:1421984899, lastAccessTime:0, retention:0,
sd:StorageDescriptor(cols:[FieldSchema(name:i_item_sk, type:bigint,
comment:null), FieldSchema(name:i_item_id, type:string, comment:null),
FieldSchema(name:i_rec_start_date, type:string, comment:null),
FieldSchema(name:i_rec_end_date, type:string, comment:null),
FieldSchema(name:i_item_desc, type:string, comment:null),
FieldSchema(name:i_current_price, type:double, comment:null),
FieldSchema(name:i_wholesale_cost, type:double, comment:null),
FieldSchema(name:i_brand_id, type:int, comment:null), FieldSchema(name:i_brand,
type:string, comment:null), FieldSchema(name:i_class_id, type:int,
comment:null), FieldSchema(name:i_class, type:string, comment:null),
FieldSchema(name:i_category_id, type:int, comment:null),
FieldSchema(name:i_category, type:string, comment:null),
FieldSchema(name:i_manufact_id, type:int, comment:null),
FieldSchema(name:i_manufact, type:string, comment:null),
FieldSchema(name:i_size, type:string, comment:null),
FieldSchema(name:i_formulation, type:string, comment:null),
FieldSchema(name:i_color, type:string, comment:null), FieldSchema(name:i_units,
type:string, comment:null), FieldSchema(name:i_container, type:string,
comment:null), FieldSchema(name:i_manager_id, type:int, comment:null),
FieldSchema(name:i_product_name, type:string, comment:null)],
location:hdfs://bhx1:8020/user/hive/warehouse/bigbenchorc.db/item,
inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat,
compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde,
parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),
partitionKeys:[], parameters:{numFiles=4, transient_lastDdlTime=1421984899,
COLUMN_STATS_ACCURATE=true, totalSize=83267548, numRows=563518,
rawDataSize=884720592}, viewOriginalText:null, viewExpandedText:null,
tableType:MANAGED_TABLE)
Time taken: 0.527 seconds, Fetched: 24 row(s)
> When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will
> result in value '0' after running 'analyze table TABLE_NAME compute
> statistics;'
> --------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HIVE-9560
> URL: https://issues.apache.org/jira/browse/HIVE-9560
> Project: Hive
> Issue Type: Bug
> Reporter: Xin Hao
>
> When hive.stats.collect.rawdatasize=true, 'rawDataSize' for an ORC table will
> result in value '0' after running 'analyze table TABLE_NAME compute
> statistics;'
> Reproduce step:
> (1) set hive.stats.collect.rawdatasize=true；
> (2) Generate an ORC table in hive, and the value of its 'rawDataSize' is NOT
> zero.
> You can find the value of 'rawDataSize' (NOT zero) by executing 'describe
> extended TABLE_NAME;'
> (4) Execute 'analyze table TABLE_NAME compute statistics;'
> (5) Execute 'describe extended TABLE_NAME;' again, and you will find that
> the value of 'rawDataSize' will be changed to '0'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
