dataproblems commented on issue #12116: URL: https://github.com/apache/hudi/issues/12116#issuecomment-2468949623
@ad1happy2go - I increased the min file count to `10000` for the 37 TB dataset that I have, however, on two trials to create the table using insert mode, I see the job failing during the `Committing Stats: collect at SparkRDDWriteClient.java:103` stage - I can see that the data is in S3 but I think it's failing during the creation of the commit file + index files. Any recommendations on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
