maheshguptags commented on issue #10456:
URL: https://github.com/apache/hudi/issues/10456#issuecomment-1893110358
Hi @xicm and @danny0405,
I tried to increase the parallelism as @xicm suggested but it is trying to
consume the data in a single commit
i.e. it accumulates the data into a single commit which causes a Heap OOM
issue.
<img width="1123" alt="image"
src="https://github.com/apache/hudi/assets/115445723/0b02127c-e14c-47b4-9033-db96d5e45a51";>
**Commit size from .hoodie folder**
Second commit is trying to consume the entire data in one commit i.e. it is
creating 41MB .commit file.
<img width="1006" alt="image"
src="https://github.com/apache/hudi/assets/115445723/ea1fad68-7a9d-4eaf-970c-63ec9adbe479";>
Can we reduce/control the commit filesize?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
