llincc commented on code in PR #9778:
URL: https://github.com/apache/hudi/pull/9778#discussion_r1335807331
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java:
##########
@@ -213,6 +213,8 @@ public R put(T key, R value) {
if (this.inMemoryMap.containsKey(key)) {
this.inMemoryMap.put(key, value);
+ } else if (inDiskContainsKey(key)) {
+ getDiskBasedMap().put(key, value);
} else if (this.currentInMemoryMapSize < this.maxInMemorySizeInBytes) {
this.currentInMemoryMapSize += this.estimatedPayloadSize;
this.inMemoryMap.put(key, value);
Review Comment:
simplify it:
1、 fullfill in-memory map first (currentInMemoryMapSize >=
maxInMemorySizeInBytes now )
2、 insert key1 record into ExternalSpillableMap so it will be spilled to
disk
3、trigger size esitimate make currentInMemoryMapSize <
maxInMemorySizeInBytes again
4、 upsert key1 record, it will be put into in-memory map because
currentInMemoryMapSize < maxInMemorySizeInBytes
5、one record in memory and one on disk ， they has the same key1
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java:
##########
@@ -213,6 +213,8 @@ public R put(T key, R value) {
if (this.inMemoryMap.containsKey(key)) {
this.inMemoryMap.put(key, value);
+ } else if (inDiskContainsKey(key)) {
+ getDiskBasedMap().put(key, value);
} else if (this.currentInMemoryMapSize < this.maxInMemorySizeInBytes) {
this.currentInMemoryMapSize += this.estimatedPayloadSize;
this.inMemoryMap.put(key, value);
Review Comment:
simplify it:
1、 fullfill in-memory map first (currentInMemoryMapSize >=
maxInMemorySizeInBytes now )
2、 insert key1 record into ExternalSpillableMap so it will be spilled to
disk
3、trigger size esitimate make currentInMemoryMapSize <
maxInMemorySizeInBytes again
4、 upsert key1 record, it will be put into in-memory map because
currentInMemoryMapSize < maxInMemorySizeInBytes
5、one record in memory and one on disk finally， they has the same key1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
