dombizita commented on code in PR #4696:
URL: https://github.com/apache/ozone/pull/4696#discussion_r1195294213
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -76,13 +87,12 @@ public ContainerKeyMapperTask(ReconContainerMetadataManager
@Override
public Pair<String, Boolean> reprocess(OMMetadataManager omMetadataManager) {
long omKeyCount = 0;
- // Maps the (container, key) -> count
+
+ // In-memory maps for fast look up and batch write
+ // key -> containerId
Review Comment:
here the `(container, key) -> count` comment would be the correct, right?
the `ContainerKeyPrefix` is storing information about the containerID and the
key (+ key version).
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -145,8 +196,17 @@ public Pair<String, Boolean> process(OMUpdateEventBatch
events) {
Iterator<OMDBUpdateEvent> eventIterator = events.getIterator();
int eventCount = 0;
final Collection<String> taskTables = getTaskTables();
+
+ // In-memory maps for fast look up and batch write
+ // (HDDS-8580) containerKeyMap map is allowed to be used
+ // in "process" without batching since the maximum number of keys
+ // is bounded by delta limit configurations
+
+ // key -> containerId
Review Comment:
same here as mentioned above
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +443,64 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count to the Global Stats table.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @param containerKeyMap we keep the added containerKeys in this map
+ * to allow incremental batching to containerKeyTable
+ * @param containerKeyCountMap we keep the containerKey counts in this map
+ * to allow batching to containerKeyCountTable
+ * after reprocessing is done
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void handleKeyReprocess(String key,
Review Comment:
I just want to make sure I understand this method correctly: the only
difference between `handlePutOMKeyEvent` and this is that we don't ask from the
`reconContainerMetadataManager` if the container exists, we rather ask if the
`containerKeyCountMap` contains the container (as a key) and if it does not
have it we just easily set the key count to zero (instead of asking it from the
`reconContainerMetadataManager`), right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
