[
https://issues.apache.org/jira/browse/HUDI-206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pratyaksh Sharma closed HUDI-206.
---------------------------------
Assignee: Pratyaksh Sharma
Resolution: Won't Fix
This functionality is available through TimestampBasedKeyGenerator class
> Modification of partition path format for writing to hudi datasets to support
> simplified queries
> ------------------------------------------------------------------------------------------------
>
> Key: HUDI-206
> URL: https://issues.apache.org/jira/browse/HUDI-206
> Project: Apache Hudi (incubating)
> Issue Type: Improvement
> Components: deltastreamer
> Reporter: Pratyaksh Sharma
> Assignee: Pratyaksh Sharma
> Priority: Major
> Labels: patch
>
> When we build a CDC pipeline, generally the field used for partitioning is
> date (created_at), and the general format of created_at is yyyy-MM-dd
> HH:mm:ss.S.
> If we have this field formatted to yyyy/MM/dd, then your hive queries for
> fetching data between any two dates become much complex, which is the usual
> case. For example -
> # if the partitions are in format yyyy/MM/dd -
> *_All Days Between 2015-01-01 and 2015-03-01_*
> _SELECT * FROM TableA WHERE YEAR=2015 and ((MONTH=01 OR MONTH=02) OR (MONTH
> =03 and DAY =01))_
> 2. Instead if partitions are in the format yyyy-MM-dd or yyyymmdd it supports
> direct queries on the data
> *_All Days Between 2015-01-01 and 2015-03-01_*
> _SELECT * FROM TableA WHERE DateStamp BETWEEN ‘2015-01-01’ AND ‘2015-03-01’_
> _For more details:_
> _Reference -
> [https://community.hortonworks.com/questions/29031/best-pratices-for-hive-partitioning-especially-by.html]_
>
> The proposal is to make the default partitioning to yyyy-mm-dd OR at least
> provide a provision to change the format
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
