[
https://issues.apache.org/jira/browse/HIVE-1076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Paul Yang updated HIVE-1076:
----------------------------
Attachment: HIVE-1076.1.patch
The reason why this is occurring is due to a custom pre-execute hook (which is
in use by facebook) that was calling alterTable to the metastore to set a flag.
The pre-execute hook would get information about what values to set for the
partition through the WriteEntity object generated by
SemanticAnalyzer:genFileSinkPlan(). The partition information (stored in the
Partition object) for the WriteEntity object is populated by the tableSpec
class in BaseSemantic analyzer.
When creating the Partition object, tableSpec does not query the metastore and
instead just creates the Partition object using the constructor. Consequently,
create time is initialized to the default value of 0. For cases where the
partition did not exist before, there is no problem as there is nothing to
alter when the pre-execute hook runs. However, for partitions that already
exist, the alter partition call in the pre-execute hook would overwrite the
partition info with creation time = 0.
To fix this, the patch queries the metastore in tableSpec to get the correct
partition information.
Currently, it's difficult to create a test for this issue because DESCRIBE
EXTENDED will print out a single line for the partition attributes. Because the
line probably contains 'file:/', diff will ignore the whole line.
Zheng - how quickly do you need the fix? Fixing that, plus adding a test case
will take some additional time. A quick solution would be to stop using that
pre-execution hook.
> CreateTime is reset to 0 when a partition is overwritten
> --------------------------------------------------------
>
> Key: HIVE-1076
> URL: https://issues.apache.org/jira/browse/HIVE-1076
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Zheng Shao
> Assignee: Paul Yang
> Attachments: HIVE-1076.1.patch
>
>
> Hive should keep "CreateTime" when a partition is overwritten. The
> "CreateTime" should be the first time the partition is created.
> {code}
> hive> describe extended zshao_ttp;
> OK
> d string
> ds string
> Detailed Table Information Table(tableName:zshao_ttp, dbName:default,
> owner:zshao, createTime:1264027720, lastAccessTime:0, retention:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:d, type:string, comment:null)],
> location:hdfs://hdfs:9000/user/hive/zshao_ttp,
> inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}), partitionKeys:[FieldSchema(name:ds, type:string,
> comment:null)], parameters:{transient_lastDdlTime=1264027720})
> Time taken: 3.062 seconds
> hive> describe extended zshao_ttp partition(ds='2010-01-01');
> OK
> d string
> ds string
> Detailed Partition Information Partition(values:[2010-01-01],
> dbName:default, tableName:zshao_ttp, createTime:1264027788, lastAccessTime:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:d, type:string, comment:null)],
> location:hdfs://hdfs:9000/user/hive/zshao_ttp/ds=2010-01-01,
> inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}), parameters:{transient_lastDdlTime=1264027788})
> Time taken: 0.436 seconds
> hive> insert overwrite table zshao_ttp partition (ds='2010-01-01') select d
> from zshao_ttp where ds = '2010-01-01';
> Total MapReduce jobs = 2
> Launching Job 1 out of 2
> Number of reduce tasks is set to 0 since there's no reduce operator
> Starting Job = job_200912262300_1111, Tracking URL =
> http://jobtracker:50030/jobdetails.jsp?jobid=job_200912262300_1111
> Kill Command = hadoop/bin/../bin/hadoop job
> -Dmapred.job.tracker=jobtracker:50029 -kill job_200912262300_1111
> 2010-01-20 15:04:15,272 Stage-1 map = 0%, reduce = 0%
> 2010-01-20 15:05:16,895 Stage-1 map = 0%, reduce = 0%
> 2010-01-20 15:06:16,768 Stage-1 map = 100%, reduce = 0%
> 2010-01-20 15:06:43,929 Stage-1 map = 100%, reduce = 100%
> Ended Job = job_200912262300_1111
> Launching Job 2 out of 2
> Moving data to: hdfs://hdfs:9000/tmp/hive-zshao/262641680/10000
> Loading data to table zshao_ttp partition {ds=2010-01-01}
> Moved to trash: /user/hive/zshao_ttp/ds=2010-01-01
> 2 Rows loaded to zshao_ttp
> OK
> Time taken: 187.049 seconds
> hive> describe extended zshao_ttp partition(ds='2010-01-01');
> OK
> d string
> ds string
> Detailed Partition Information Partition(values:[2010-01-01],
> dbName:default, tableName:zshao_ttp, createTime:0, lastAccessTime:0,
> sd:StorageDescriptor(cols:[FieldSchema(name:d, type:string, comment:null)],
> location:hdfs://hdfs:9000/user/hive/zshao_ttp/ds=2010-01-01,
> inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
> outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
> compressed:false, numBuckets:-1, serdeInfo:SerDeInfo(name:null,
> serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe,
> parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
> parameters:{}),
> parameters:{lastQueryTime=1264028626290,archiveFlag=false,transient_lastDdlTime=1264028626})
> Time taken: 0.283 seconds
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.