[
https://issues.apache.org/jira/browse/HUDI-6007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zouxxyy updated HUDI-6007:
--------------------------
Description:
Currently hudi's savepoint only saves the base file, and filter the files in it
when clean.
But when using the MOR table with flink, hudi's savepoint may be invalid,
because there may no base file in a FG.
file:
{code:java}
.4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.1_0-1-0
.4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.2_0-1-0
.hoodie_partition_metadata {code}
20230330235205548.savepoint:
{code:java}
{
"savepointedBy": "",
"savepointedAt": 1680191759153,
"comments": "",
"partitionMetadata": {
"dt=2021-12-09/hh=10": {
"partitionPath": "dt=2021-12-09/hh=10",
"savepointDataFile": []
}
},
"version": {
"int": 1
}
} {code}
was:
Currently hudi's savepoint only saves the base file, and filter the files in it
when clean.
But when using the MOR table with flink, hudi's savepoint is may invalid,
because there may no base file in a FG.
file:
{code:java}
.4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.1_0-1-0
.4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.2_0-1-0
.hoodie_partition_metadata {code}
20230330235205548.savepoint:
{code:java}
{
"savepointedBy": "",
"savepointedAt": 1680191759153,
"comments": "",
"partitionMetadata": {
"dt=2021-12-09/hh=10": {
"partitionPath": "dt=2021-12-09/hh=10",
"savepointDataFile": []
}
},
"version": {
"int": 1
}
} {code}
> When using the MOR table with flink, hudi savepoint may be invalid which lead
> to consistency issue
> --------------------------------------------------------------------------------------------------
>
> Key: HUDI-6007
> URL: https://issues.apache.org/jira/browse/HUDI-6007
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: zouxxyy
> Priority: Major
>
> Currently hudi's savepoint only saves the base file, and filter the files in
> it when clean.
> But when using the MOR table with flink, hudi's savepoint may be invalid,
> because there may no base file in a FG.
> file:
> {code:java}
> .4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.1_0-1-0
> .4e164104-da91-4c39-89b8-a81890cf2c8c_20230330235205548.log.2_0-1-0
> .hoodie_partition_metadata {code}
> 20230330235205548.savepoint:
> {code:java}
> {
> "savepointedBy": "",
> "savepointedAt": 1680191759153,
> "comments": "",
> "partitionMetadata": {
> "dt=2021-12-09/hh=10": {
> "partitionPath": "dt=2021-12-09/hh=10",
> "savepointDataFile": []
> }
> },
> "version": {
> "int": 1
> }
> } {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
