the-other-tim-brown commented on PR #12164: URL: https://github.com/apache/hudi/pull/12164#issuecomment-2481873107
> I'm wondering how much gains we can have for this patch, it looks like the code become messy and more error prone, another direction is we resolve the problem of the active timeline listing issue throughly(and internally in the timeline listing itself, for e.g, another menifest file to keep the instant main properties: instant time, action name, state) so that all the codes can be beneficiary. @danny0405 it is significant at scale on cloud storage with large timelines (thousands of commits retained). Each listing costs money and processing time. Each write will list a timeline close to 10 times with Metadata table enabled based on my reading of the code. This can be reduced even further and I will post more PRs if there is no other solution in the works. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
