llincc commented on code in PR #9778:
URL: https://github.com/apache/hudi/pull/9778#discussion_r1336679563
##########
hudi-common/src/main/java/org/apache/hudi/common/util/collection/ExternalSpillableMap.java:
##########
@@ -213,6 +213,8 @@ public R put(T key, R value) {
if (this.inMemoryMap.containsKey(key)) {
this.inMemoryMap.put(key, value);
+ } else if (inDiskContainsKey(key)) {
Review Comment:
@bvaradar
thanks for your suggestions. this obviously has better performance when
calling get() for option two. There are two ways to implement this:
1. Remove the old version record from disk first and then put the new
record to in-memory map. but there is a performance problem to remove record.
2. keep the behavior of putting data and deduplicate data when iterator.
But it is hard to implement hasNext() function of IteratorWrapper , do you have
any good ideas about it ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
