[
https://issues.apache.org/jira/browse/HUDI-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo updated HUDI-7332:
------------------------------
Fix Version/s: 1.0.2
> The best way to force cleaning hoodie metadata
> ----------------------------------------------
>
> Key: HUDI-7332
> URL: https://issues.apache.org/jira/browse/HUDI-7332
> Project: Apache Hudi
> Issue Type: Bug
> Components: cleaning, hudi-utilities, metadata, table-service
> Environment: Environment Description
> Hudi version : 0.11.0
> Spark version : 3.2.1
> Amazon EMR : emr-6.11.1
> Hadoop version : 3.2.1
> Hive : 3.1.3
> Storage (HDFS/S3/GCS..) : S3
> Running on Docker? (yes/no) : No, yarn.
> Reporter: Haitham Eltaweel
> Priority: Major
> Fix For: 1.0.2
>
>
> We have spark structured streaming job writing data to hudi tables. After an
> upgrade to hudi 0.11, we found that we have thousands of files under hoodie
> metadata which were not cleaned or archived. This impacts the overall
> processing of the streaming job. I found similar issue in
> [https://github.com/apache/hudi/issues/7472] and they mentioned this was
> fixed in 0.13. Since we have the issue in Prod, we will not be able to
> upgrade to 0.13 for now. I found that I can run sperate spark submit job to
> execute HoodieCleaner. I also found that deleting hudie metadata from
> hudi-cli could be an option but I am not sure if its safe to use that
> approach as we are using upsert hudi operation in the streaming job.
> Please advise what is the best way to force cleaning and archiving the
> metadata files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
