[
https://issues.apache.org/jira/browse/HUDI-6051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangkang updated HUDI-6051:
---------------------------
Description:
We plan to use offline compation mechanism in Hudi 0.12.0 (compaction plan is
generated by Flink task). I meet a question, a new compaction plan will been
generated after five deltacommits by default, but there will not generate any
new compaction plan if the old compaction plan isn't executed whatever how many
deltacommits submit.
{panel:title=My title}
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3419 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 1560 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164722114.compaction.requested
{-}rw-rw{-}---+ 3 kwang kwang 3281 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.requested
{-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.inflight
{-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.requested
{panel}
was:
We plan to use offline compation mechanism in Hudi (compaction plan is
generated by Flink task). I meet a question, a new compaction plan will been
generated after five deltacommits by default, but there will not generate any
new compaction plan if the old compaction plan isn't executed whatever how many
deltacommits submit.
{panel:title=My title}
-rw-rw----+ 3 kwang kwang 3283 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-03-29 11:31
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-03-30 17:07
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-03-30 17:50
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-03 10:22
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3419 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.requested
-rw-rw----+ 3 kwang kwang 1560 2023-04-03 16:47
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164722114.compaction.requested
-rw-rw----+ 3 kwang kwang 3281 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-03 19:54
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-07 10:26
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:14
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:15
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-07 11:16
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.requested
-rw-rw----+ 3 kwang kwang 3283 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit
-rw-rw----+ 3 kwang kwang 0 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.inflight
-rw-rw----+ 3 kwang kwang 0 2023-04-10 14:46
hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.requested
{panel}
> Flink task generate only one compaction plan if the old compaction plan not
> executed
> ------------------------------------------------------------------------------------
>
> Key: HUDI-6051
> URL: https://issues.apache.org/jira/browse/HUDI-6051
> Project: Apache Hudi
> Issue Type: Bug
> Reporter: wangkang
> Priority: Major
>
> We plan to use offline compation mechanism in Hudi 0.12.0 (compaction plan is
> generated by Flink task). I meet a question, a new compaction plan will been
> generated after five deltacommits by default, but there will not generate any
> new compaction plan if the old compaction plan isn't executed whatever how
> many deltacommits submit.
> {panel:title=My title}
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-29 11:31
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-29 11:31
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-29 11:31
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230329113123218.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-30 17:07
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:07
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:07
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330170752044.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-03-30 17:50
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:50
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-03-30 17:50
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230330175043697.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-03 10:22
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 10:22
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 10:22
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403102202737.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3419 2023-04-03 16:47
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 16:47
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 16:47
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164717103.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 1560 2023-04-03 16:47
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403164722114.compaction.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3281 2023-04-03 19:54
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 19:54
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-03 19:54
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230403195454000.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 10:26
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 10:26
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 10:26
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407102630796.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:14
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:14
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:14
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111431169.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111511524.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:15
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111554300.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-07 11:16
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:16
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-07 11:16
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230407111642037.deltacommit.requested
> {-}rw-rw{-}---+ 3 kwang kwang 3283 2023-04-10 14:46
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-10 14:46
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.inflight
> {-}rw-rw{-}---+ 3 kwang kwang 0 2023-04-10 14:46
> hdfs://nameservice/user/kwang/hudi/t1/.hoodie/20230410144637366.deltacommit.requested
> {panel}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
