[
https://issues.apache.org/jira/browse/HUDI-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kazdy updated HUDI-6232:
------------------------
Description:
Glue has soft limit of 100k table versions. With every table update new version
is created. In streaming scenarios we can hit 100k table versions quickly.
Users should be able to skip table archival when using glue meta sync.
It's been asked many times in Hudi Slack, how to reduce no of glue table
versions created. There is conditional hive sync, but it does not solve the
problem fully. With .skipArchive users can reduce no. of table versions created
even more.
was:
Glue has soft limit of 100k table versions. With every table update new version
is created. In streaming scenarios we can hit 100k table versions quickly.
Users should be able to skip table archival when using glue meta sync.
> Support skip archive for glue sync
> ----------------------------------
>
> Key: HUDI-6232
> URL: https://issues.apache.org/jira/browse/HUDI-6232
> Project: Apache Hudi
> Issue Type: Improvement
> Components: meta-sync
> Reporter: kazdy
> Assignee: kazdy
> Priority: Major
> Fix For: 0.14.0
>
>
> Glue has soft limit of 100k table versions. With every table update new
> version is created. In streaming scenarios we can hit 100k table versions
> quickly.
> Users should be able to skip table archival when using glue meta sync.
> It's been asked many times in Hudi Slack, how to reduce no of glue table
> versions created. There is conditional hive sync, but it does not solve the
> problem fully. With .skipArchive users can reduce no. of table versions
> created even more.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
