parisni commented on issue #8925: URL: https://github.com/apache/hudi/issues/8925#issuecomment-1585528986
Then I confirm u use timeline server. Also from your stats I am not surprised writing 30k partition and updating so much files takes 8min (the doing partition and writing files job). The tagging and building profile also looks correct). What is weird is the first web ui view. I wonder why you have those listings happening before starting the upsert. Since you have MDT enabled this cannot be listing partition for getting table file. Could you also share the first 40 jobs to understand what's going on with // file listing ? On June 9, 2023 9:12:58 PM UTC, Samarth Raval ***@***.***> wrote: >Also I have large number of partitions, last few commits look like this >~Approximate number of partition could be (2000*10*12 ~24,000 - 30,000) [ Assuming this as I don't know how to count number of partitions] > > > > >-- >Reply to this email directly or view it on GitHub: >https://github.com/apache/hudi/issues/8925#issuecomment-1585134643 >You are receiving this because you were mentioned. > >Message ID: ***@***.***> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
