[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-948247852 @bgt-cdedels : sorry, don't think there is any easier way. I can only think of a naive solution. you can try to recreate a new table w/ data from this table and delete old one.

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-946363380 @codope : Can you create a ticket for adding ability via hudi-cli to clean up dangling data files. -- This is an automated message from the Apache Git Service. To respond to the

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-937434037 I am not aware of any easier option or not hudi-cli has any option for this. @vinothchandar @bhasudha @bvaradar @n3nash : any suggestions here. Here is the question: if

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-937434037 I am not aware of any easier option or not hudi-cli has any option for this. @vinothchandar @bhasudha @bvaradar @n3nash : any suggestions here. Here is the question: if

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-936736174 or is your questions, due to mis-configuration, if archival trimmed some commits from timeline which cleaner did not get a chance to clean, is there a way to go about cleaning them

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-936731947 to clarify, archival touches only the timeline and cleaner touches only the data files. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-936729727 let me illustrate w/ an example. archival works with timeline, where as cleaner deals with data files. this difference is important to understand the interplays here.

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-933549686 Let me know if you need anything. else we can close this issue out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-933549405 You can set it to 10. should work out. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-932199950 awesome, good to know. So here is the thing. Hudi has something called active timeline and archived timeline. Archival will kick in for every commit and move some older commits

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931743495 sure. can we try hoodie.keep.max.commits: 20 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931544474 likely its the archival thats coming into play. let me know how it goes if you set higher value for hoodie.keep.max.commits. -- This is an automated message from the Apache Git

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931540799 also, can you try setting this to higher no hoodie.keep.max.commits: may be 10. -- This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931537607 may be a dumb thought. can you try setting the config values as string. 'hoodie.compact.inline': 'true' 'hoodie.cleaner.commits.retained':'4' -- This is an automated

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931533803 sorry, I see that you do have 5 compactions. Can you try enabling inline cleaning? hoodie.compact.inline=true -- This is an automated message from the Apache Git Service. To

[GitHub] [hudi] nsivabalan commented on issue #3739: Hoodie clean is not deleting old files

nsivabalan commented on issue #3739: URL: https://github.com/apache/hudi/issues/3739#issuecomment-931525996 thanks. Here is what is possibly happening. If you can tigger more updates, eventually you will see cleaning kicking in. In short, this has something to do w/ MOR table. cleaner