devmadhuu commented on code in PR #4696:
URL: https://github.com/apache/ozone/pull/4696#discussion_r1201936317
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java:
##########
@@ -383,4 +398,60 @@ private void handlePutOMKeyEvent(String key, OmKeyInfo
omKeyInfo,
}
}
+ /**
+ * Write an OM key to container DB and update containerID -> no. of keys
+ * count to the Global Stats table.
+ *
+ * @param key key String
+ * @param omKeyInfo omKeyInfo value
+ * @param containerKeyCountMap we keep the containerKey counts in this map
+ * to allow batching to containerKeyCountTable
+ * after reprocessing is done
+ * @throws IOException if unable to write to recon DB.
+ */
+ private void handleKeyReprocess(String key,
+ OmKeyInfo omKeyInfo,
+ Map<Long, Long> containerKeyCountMap)
+ throws IOException {
+ long containerCountToIncrement = 0;
+ for (OmKeyLocationInfoGroup omKeyLocationInfoGroup : omKeyInfo
+ .getKeyLocationVersions()) {
+ long keyVersion = omKeyLocationInfoGroup.getVersion();
+ for (OmKeyLocationInfo omKeyLocationInfo : omKeyLocationInfoGroup
+ .getLocationList()) {
+ long containerId = omKeyLocationInfo.getContainerID();
+ ContainerKeyPrefix containerKeyPrefix = new ContainerKeyPrefix(
+ containerId, key, keyVersion);
+ if (reconContainerMetadataManager.getCountForContainerKeyPrefix(
+ containerKeyPrefix) == 0) {
+ // Save on writes. No need to save same container-key prefix
+ // mapping again.
+ reconContainerMetadataManager.storeContainerKeyMapping(
Review Comment:
@dombizita Since this threshold is for storing ContainerKeyPrefix objects
in a map, and this class has 2 long variables and 1 String (size depends on
length as it uses 2 bytes per UTF-16 code as java internally uses UTF-16), And
if we consider just for example keyPrefix maximum length as 200 chars so
ideally one object size -> 8*2 + (200*2) ~ 416 bytes, based on that assumption,
we can test with 1 billion keys if possible or can extrapolate on testing data
of 1 million keys and then try to optimise this threshold value. IMO, we can
keep above suggested value, but let @ivandika3 test and come up with more near
to optimum.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
