llincc commented on code in PR #9778:
URL: https://github.com/apache/hudi/pull/9778#discussion_r1335807331
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java:
##########
@@ -213,6 +213,8 @@ public R put(T key, R value) {
if (this.inMemoryMap.containsKey(key)) {
this.inMemoryMap.put(key, value);
+ } else if (inDiskContainsKey(key)) {
+ getDiskBasedMap().put(key, value);
} else if (this.currentInMemoryMapSize < this.maxInMemorySizeInBytes) {
this.currentInMemoryMapSize += this.estimatedPayloadSize;
this.inMemoryMap.put(key, value);
Review Comment:
simplify it:
1、 fullfill in-memory map first (currentInMemoryMapSize >=
maxInMemorySizeInBytes now )
2、 insert key1 record into ExternalSpillableMap so it will be spilled to
disk
3、trigger size esitimate make currentInMemoryMapSize <
maxInMemorySizeInBytes again
4、 upsert key1 record, it will be put into in-memory map because
currentInMemoryMapSize < maxInMemorySizeInBytes
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java:
##########
@@ -213,6 +213,8 @@ public R put(T key, R value) {
if (this.inMemoryMap.containsKey(key)) {
this.inMemoryMap.put(key, value);
+ } else if (inDiskContainsKey(key)) {
+ getDiskBasedMap().put(key, value);
} else if (this.currentInMemoryMapSize < this.maxInMemorySizeInBytes) {
this.currentInMemoryMapSize += this.estimatedPayloadSize;
this.inMemoryMap.put(key, value);
Review Comment:
simplify it:
1、 fullfill in-memory map first (currentInMemoryMapSize >=
maxInMemorySizeInBytes now )
2、 insert key1 record into ExternalSpillableMap so it will be spilled to
disk
3、trigger size esitimate make currentInMemoryMapSize <
maxInMemorySizeInBytes again
4、 upsert key1 record, it will be put into in-memory map because
currentInMemoryMapSize < maxInMemorySizeInBytes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
