[
https://issues.apache.org/jira/browse/HUDI-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed HUDI-7466.
----------------------------
Fix Version/s: 0.15.0
1.0.0
Resolution: Fixed
Fixed via master branch: e726306cf0905512e3afe4b875cb15dfbef40e52
> AWS Glue sync
> -------------
>
> Key: HUDI-7466
> URL: https://issues.apache.org/jira/browse/HUDI-7466
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Vitali Makarevich
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.15.0, 1.0.0
>
>
> Ticket to track
> [https://github.com/apache/hudi/pull/10460]
> Currently, AWS Glue sync works and provides 2 interfaces - one is
> {{HoodieHiveSyncClient}} using Hive, then Glue -> Hive implementation(hidden
> by AWS), and another is {{{}AWSGlueCatalogSyncClient{}}}.
> Both of them have limitations - although Hive has improved a bit to use
> pushdown on a big scale still fails and fallback may not work for certain
> partitioning schemes.
> For syncing to Glue using {{{}HoodieHiveSyncClient{}}}, there is a set of
> limitations:
> # Create/update is not parallelized under the hood, meaning for big sets
> it's very slow - empirically it's about 40 partitions/sec MAX, which
> translates to minutes for bigger scale.
> # The pushdown filter is not really effective since for 1st case(specifying
> exact partitions) - it works unpredictably, since the longer the partition
> value you have, the fewer partitions you can specify, in our case we cannot
> specify > 100 partitions, therefore it falls back to min-max predicate.
> # Min-max predicate does not work if the number of partitions is growing
> with nesting, e.g. on level 1 there are 10, on level 2 there are 100, on
> level 3 there are 1000. In this case, min-max will cut down high-level ones,
> but load all levels down, therefore not really making optimization.
> # When there is e.g. a schema change, Hive-Glue calls {{{}cascade{}}}, and
> for big tables it's impossible to sync in meaningful time - although for Glue
> -> Hudi does not specify schema on partition level, so this is wasted effort.
> This is why {{AWSGlueCatalogSyncClient}} is preferable. But there are other
> problems with it.
> Particular list of problems:
> # Create/Update/Delete were not optimized before - now optimized to be
> async, but without a meaningful high border, it will simply reach the request
> limit and stay there. *This solution adds a parameter for such parallelism
> and creates parallelization logic.*
> # Listing all partitions is used always for {{AWSGlueCatalogSyncClient}} -
> this is way suboptimal since the goal of this is to distinguish which of
> changed-since-last-sync are created and which are deletes, therefore more
> optimal API can be used -
> [{{BatchGetPartition}}|https://docs.aws.amazon.com/glue/latest/webapi/API_BatchGetPartition.html].
> Also, it can be parallelized easily. *I added a new method to sync client
> classes and moved Hive-pushdown into a Hive-specific class and implemented
> this method for the AWSGlue client class. Also, parameter controlling
> parallelism is added.*
> # Listing all partitions is suboptimal - it is still needed for initial
> sync/resync, but it's done in a straightforward way and is suboptimal. In
> particular - it uses basic {{nextToken}} which makes it sequential and works
> slowly in heavily partitioned tables. AWS has an improvement for this
> particular
> [method|https://docs.aws.amazon.com/glue/latest/webapi/API_GetPartitions.html],
> called {{{}segment{}}}. This allows us to basically create 1 to 10 start
> positions and use standard({{{}nextToken{}}}) API to list partitions. Also -
> [last public version of Hive-> Glue interface implementation uses
> it|https://docs.aws.amazon.com/glue/latest/webapi/API_GetPartitions.html].
> When we switched from the Hive sync class to AWS Glue specific - first what
> we faced is performance degradation with the listing. *I added {{segment}}
> API parameter usage and added parameter controlling parallelism.*
> All this has been tested for a partitioned table with >200k partitions.
> I managed to get speed improvement from 2-3 minutes to 3 seconds. Let me know
> if you are interested in numbers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
