GitHub user debugger87 reopened a pull request:
https://github.com/apache/spark/pull/18900
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
## What changes were proposed in this pull request?
Set createTime for every hive partition created in Spark SQL, which could
be used to manage data lifecycle in Hive warehouse. We found that almost every
partition created by spark sql has not been set createTime.
```
mysql> select * from partitions where create_time=0 limit 1\G;
*************************** 1. row ***************************
PART_ID: 1028584
CREATE_TIME: 0
LAST_ACCESS_TIME: 1502203611
PART_NAME: date=20170130
SD_ID: 1543605
TBL_ID: 211605
LINK_TARGET_ID: NULL
1 row in set (0.27 sec)
```
## How was this patch tested?
N/A
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/debugger87/spark
fix/set-create-time-for-hive-partition
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18900.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18900
----
commit 71a660ac8dad869d9ba3b4e206b74f5c44660ee6
Author: debugger87 <yangchaozhong.2009@...>
Date: 2017-08-10T04:17:00Z
[SPARK-21687][SQL] Spark SQL should set createTime for Hive partition
commit f668ce8837ee553c61687bd03d04cddd32e5f36f
Author: debugger87 <yangchaozhong.2009@...>
Date: 2017-08-11T07:50:26Z
added createTime and lastAccessTime into CatalogTablePartition
commit 2fb1ddabdb2ab8f7b585ee7aea93280f96a23467
Author: debugger87 <yangchaozhong.2009@...>
Date: 2017-08-11T07:54:26Z
minor tweak
commit c833ce7aa5f2ba0b684494fd1b24b7995f1c09c9
Author: debugger87 <yangchaozhong.2009@...>
Date: 2017-08-11T08:07:57Z
fix type missmatch
commit bf2a1052f807a7ae36004c819e66fff5c4b45820
Author: debugger87 <yangchaozhong.2009@...>
Date: 2017-08-11T23:26:29Z
added createTime and lastAccessTime into partition map for display
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
