HEPBO3AH commented on PR #6581: URL: https://github.com/apache/hudi/pull/6581#issuecomment-1328350889
A general question: **Will this cluster other small files that might be in the partition that was touched that are not part of the current group?** Think of the following example: Timestamped event data arrives from devices once every hour, up to 24 times per day. Data is ingested in a single batch every time the processing is ran. 1% of the devices is offline for up to 100 days. The storage has daily partitions. Over the course of 100 days, the 1% of devices create up to `100 * 24 = 2400` file groups in the partition that is 100 days before `today`. With this PR merged, will all of those files be clustered? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
