[
https://issues.apache.org/jira/browse/HUDI-512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17277302#comment-17277302
]
Vinoth Chandar commented on HUDI-512:
-------------------------------------
[~afilipchik] here are my thoughts. the physical partitioning is just an
optimization technique to filter files. and as long as we can on the query
side, easily filter out files that don't match a predicate, I think we can just
recommend keeping the table nonpartitioned in Hudi (we need not remove
partitioning altogether IMO, since it may be useful for a lot of users who have
gotten accustomed to it).
Now, to solving the problem of file pruning. the physical partitioning is
brittle and has its issues with objects stores (which we try to workaround in
side Hudi writer by parallelizing the prefix listing), but its a good way to
completely prune out tons of files in bulk, at the folder level without even
having to evaluate the files individually.
If we simply track say column ranges for each file (similar to iceberg), with
millions of files in a table. then for each query we need to compare the
predicate say `datestr = 2021-01-01` against all files. RFC-15's next stage is
to hopefully build a more performant version of this, mapping column values
against files that fall in that range. There are tradeoffs. but one of these
should solve this issue, correct?
> Decouple logical partitioning from physical one.
> -------------------------------------------------
>
> Key: HUDI-512
> URL: https://issues.apache.org/jira/browse/HUDI-512
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core
> Reporter: Alexander Filipchik
> Priority: Major
> Labels: features
> Fix For: 0.8.0
>
>
> This one is more inspirational, but, I believe, will be very useful.
> Currently hudi is following Hive table format, which means that data is
> logically and physically partitioned into folder structure like:
> table_name
> 2019
> 01
> 02
> bla.parquet
>
> This has several issues:
> 1) Modern object stores (AWS S3, GCP) are more performant when each file
> name starts with some kind of a random value. By definition Hive layout is
> not perfect
> 2) Hive Metastore stores partitions in the text field in the single table (2
> tables with very similar information) and doesn't support proper filtering.
> Data partitioned by day will be stored like:
> 2019/01/10
> 2019/01/11
> so only regexp queries are suported (at least in Hive 2.X.X)
> 3) Having a single POF which relies on non distributed DB is dangerous and
> creates bottlenecks.
>
> The idea is to get rid of logical partitioning all together (and hive
> metastore as well). If dataset has a time columns, user should be able to
> query it without understanding what is the physical layout of the table (by
> specifying those partitions explicitly or ending up with a full table scan
> accidentally).
> It will require some kind of mapping of time to file locations (similar to
> Iceberg). I'm also leaning towards the idea that storing table metadata with
> the table is a good thing as it can be read by the engine in one shot and
> will be faster that taxing a standalone metastore.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
