garyli1019 commented on issue #1890: URL: https://github.com/apache/hudi/issues/1890#issuecomment-668207368
@bvaradar I tried `HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_MERGE_PROP=0.01` and `HoodieMemoryConfig.MAX_MEMORY_FRACTION_FOR_COMPACTION_PROP=0.01`, but this issue still persist when the log file group is larger than 2GB. https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/util/collection/DiskBasedMap.java#L353 ``` // Size (numberOfBytes) of the value written to disk private Integer sizeOfValue; ``` The max value of Integer to represent byte is 2GB, so I believe this issue is related to this. For the log file group that was 1.8GB, I don't see this issue anymore. Tried with both low and high memory fraction. Would you try in your environment to have a log file group larger than 2GB and see if you can reproduce? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
