ashishkumar50 commented on code in PR #4696:
URL: https://github.com/apache/ozone/pull/4696#discussion_r1190722906
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +369,55 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void writeOMKeyToContainerDB(String key, OmKeyInfo omKeyInfo)
+ throws IOException {
+ long containerCountToIncrement = 0;
+ for (OmKeyLocationInfoGroup omKeyLocationInfoGroup : omKeyInfo
+ .getKeyLocationVersions()) {
+ long keyVersion = omKeyLocationInfoGroup.getVersion();
+ for (OmKeyLocationInfo omKeyLocationInfo : omKeyLocationInfoGroup
+ .getLocationList()) {
+ long containerId = omKeyLocationInfo.getContainerID();
+ ContainerKeyPrefix containerKeyPrefix = new ContainerKeyPrefix(
+ containerId, key, keyVersion);
+ if (reconContainerMetadataManager.getCountForContainerKeyPrefix(
+ containerKeyPrefix) == 0) {
+ // Save on writes. No need to save same container-key prefix
+ // mapping again.
+ reconContainerMetadataManager.storeContainerKeyMapping(
+ containerKeyPrefix, 1);
+
+ // check if container already exists and
+ // increment the count of containers if it does not exist
+ if (!reconContainerMetadataManager.doesContainerExists(containerId))
{
+ containerCountToIncrement++;
+ }
+
+ // update the count of keys for the given containerID
+ long keyCount = reconContainerMetadataManager
+ .getKeyCountForContainer(containerId);
Review Comment:
getKeyCountForContainer also we can avoid querying from rocksdb always,
Initialize keyCount as 0 and if container doesn't exist we can avoid calling
this.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +369,55 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void writeOMKeyToContainerDB(String key, OmKeyInfo omKeyInfo)
+ throws IOException {
+ long containerCountToIncrement = 0;
+ for (OmKeyLocationInfoGroup omKeyLocationInfoGroup : omKeyInfo
+ .getKeyLocationVersions()) {
+ long keyVersion = omKeyLocationInfoGroup.getVersion();
+ for (OmKeyLocationInfo omKeyLocationInfo : omKeyLocationInfoGroup
+ .getLocationList()) {
+ long containerId = omKeyLocationInfo.getContainerID();
+ ContainerKeyPrefix containerKeyPrefix = new ContainerKeyPrefix(
+ containerId, key, keyVersion);
+ if (reconContainerMetadataManager.getCountForContainerKeyPrefix(
+ containerKeyPrefix) == 0) {
+ // Save on writes. No need to save same container-key prefix
+ // mapping again.
+ reconContainerMetadataManager.storeContainerKeyMapping(
+ containerKeyPrefix, 1);
+
+ // check if container already exists and
+ // increment the count of containers if it does not exist
+ if (!reconContainerMetadataManager.doesContainerExists(containerId))
{
+ containerCountToIncrement++;
Review Comment:
I think in earlier case containerKeyMap keep appending and increasing memory
during reprocess for each key.
After this change we are querying from rocksdb always, I think better we can
avoid that.
May be we can keep map in this function locally to keep containerId so that
we can avoid querying rocksdb again for same container. There won't be many
containers per key and hence memory will not increase much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
