ivandika3 commented on code in PR #4696:
URL: https://github.com/apache/ozone/pull/4696#discussion_r1202338364
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +398,60 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count to the Global Stats table.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @param containerKeyCountMap we keep the containerKey counts in this map
+ * to allow batching to containerKeyCountTable
+ * after reprocessing is done
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void handleKeyReprocess(String key,
+ OmKeyInfo omKeyInfo,
+ Map<Long, Long> containerKeyCountMap)
+ throws IOException {
+ long containerCountToIncrement = 0;
+ for (OmKeyLocationInfoGroup omKeyLocationInfoGroup : omKeyInfo
+ .getKeyLocationVersions()) {
+ long keyVersion = omKeyLocationInfoGroup.getVersion();
+ for (OmKeyLocationInfo omKeyLocationInfo : omKeyLocationInfoGroup
+ .getLocationList()) {
+ long containerId = omKeyLocationInfo.getContainerID();
+ ContainerKeyPrefix containerKeyPrefix = new ContainerKeyPrefix(
+ containerId, key, keyVersion);
+ if (reconContainerMetadataManager.getCountForContainerKeyPrefix(
+ containerKeyPrefix) == 0) {
+ // Save on writes. No need to save same container-key prefix
+ // mapping again.
+ reconContainerMetadataManager.storeContainerKeyMapping(
Review Comment:
@devmadhuu @dombizita I finished testing for 150,000 and the memory
consumption is seems to be normal. There doesn't seem to be significant
performance improvement compared to 1,000. I have updated the value to 150,000.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +398,60 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count to the Global Stats table.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @param containerKeyCountMap we keep the containerKey counts in this map
+ * to allow batching to containerKeyCountTable
+ * after reprocessing is done
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void handleKeyReprocess(String key,
+ OmKeyInfo omKeyInfo,
+ Map<Long, Long> containerKeyCountMap)
+ throws IOException {
+ long containerCountToIncrement = 0;
+ for (OmKeyLocationInfoGroup omKeyLocationInfoGroup : omKeyInfo
+ .getKeyLocationVersions()) {
+ long keyVersion = omKeyLocationInfoGroup.getVersion();
+ for (OmKeyLocationInfo omKeyLocationInfo : omKeyLocationInfoGroup
+ .getLocationList()) {
+ long containerId = omKeyLocationInfo.getContainerID();
+ ContainerKeyPrefix containerKeyPrefix = new ContainerKeyPrefix(
+ containerId, key, keyVersion);
+ if (reconContainerMetadataManager.getCountForContainerKeyPrefix(
+ containerKeyPrefix) == 0) {
+ // Save on writes. No need to save same container-key prefix
+ // mapping again.
+ reconContainerMetadataManager.storeContainerKeyMapping(
Review Comment:
@devmadhuu @dombizita I finished testing for 150,000 and the memory
consumption is seems to be normal. There doesn't seem to be significant
performance improvement compared to 1,000. I have updated the defaul value to
150,000.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
