n3nash commented on issue #2110: URL: https://github.com/apache/hudi/issues/2110#issuecomment-699025467
@tooptoop4 If you're talking about executor memory, in this particular scenario, you can start with 2GB. In general, the upsert code performs a hash merge for which it uses a spillable map which has the following default setting -> https://github.com/apache/hudi/blob/master/hudi-client/src/main/java/org/apache/hudi/config/HoodieMemoryConfig.java#L89. Ideally, if your target file is not very large (<256 MB), you should be able to use 2GB executor memory. The driver memory depends on how many records you're ingesting and what is the size of them, in this case that you described, I think you can even set the driver memory to 2GB and see if it OOMs. For better analysis of memory usage, you can use some open source tools such as https://github.com/uber-common/jvm-profiler or https://github.com/linkedin/dr-elephant ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
