dombizita commented on code in PR #5037:
URL: https://github.com/apache/ozone/pull/5037#discussion_r1446214166
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableInsightTask.java:
##########
@@ -124,69 +139,26 @@ public Pair<String, Boolean> reprocess(OMMetadataManager
omMetadataManager) {
if (!objectCountMap.isEmpty()) {
writeDataToDB(objectCountMap);
}
- if (!unReplicatedSizeCountMap.isEmpty()) {
- writeDataToDB(unReplicatedSizeCountMap);
+ if (!unReplicatedSizeMap.isEmpty()) {
+ writeDataToDB(unReplicatedSizeMap);
}
- if (!replicatedSizeCountMap.isEmpty()) {
- writeDataToDB(replicatedSizeCountMap);
+ if (!replicatedSizeMap.isEmpty()) {
+ writeDataToDB(replicatedSizeMap);
}
LOG.info("Completed a 'reprocess' run of OmTableInsightTask.");
return new ImmutablePair<>(getTaskName(), true);
}
- /**
- * Returns a triple with the total count of records (left), total
unreplicated
- * size (middle), and total replicated size (right) in the given iterator.
- * Increments count for each record and adds the dataSize if a record's value
- * is an instance of OmKeyInfo. If the iterator is null, returns (0, 0, 0).
- *
- * @param iterator The iterator over the table to be iterated.
- * @return A Triple with three Long values representing the count,
- * unreplicated size and replicated size.
- * @throws IOException If an I/O error occurs during the iterator traversal.
- */
- private Triple<Long, Long, Long> getTableSizeAndCount(
- TableIterator<String, ? extends Table.KeyValue<String, ?>> iterator)
- throws IOException {
- long count = 0;
- long unReplicatedSize = 0;
- long replicatedSize = 0;
-
- if (iterator != null) {
- while (iterator.hasNext()) {
- Table.KeyValue<String, ?> kv = iterator.next();
- if (kv != null && kv.getValue() != null) {
- if (kv.getValue() instanceof OmKeyInfo) {
- OmKeyInfo omKeyInfo = (OmKeyInfo) kv.getValue();
- unReplicatedSize += omKeyInfo.getDataSize();
- replicatedSize += omKeyInfo.getReplicatedSize();
- count++;
- }
- if (kv.getValue() instanceof RepeatedOmKeyInfo) {
- RepeatedOmKeyInfo repeatedOmKeyInfo = (RepeatedOmKeyInfo) kv
- .getValue();
- Pair<Long, Long> result = repeatedOmKeyInfo.getTotalSize();
- unReplicatedSize += result.getRight();
- replicatedSize += result.getLeft();
- // Since we can have multiple deleted keys of same name
- count += repeatedOmKeyInfo.getOmKeyInfoList().size();
- }
- }
- }
- }
-
- return Triple.of(count, unReplicatedSize, replicatedSize);
- }
-
/**
* Returns a collection of table names that require data size calculation.
*/
public Collection<String> getTablesToCalculateSize() {
- List<String> taskTables = new ArrayList<>();
+ java.util.List<String> taskTables = new ArrayList<>();
Review Comment:
Is the package needed here?
```suggestion
List<String> taskTables = new ArrayList<>();
```
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/OMMetadataManagerTestUtils.java:
##########
@@ -397,23 +397,31 @@ public static void writeDirToOm(OMMetadataManager
omMetadataManager,
.build());
}
+ @SuppressWarnings("parameternumber")
public static void writeDeletedDirToOm(OMMetadataManager omMetadataManager,
String bucketName,
String volumeName,
String dirName,
long parentObjectId,
long bucketObjectId,
- long volumeObjectId)
+ long volumeObjectId,
+ long objectId)
throws IOException {
- // DB key in DeletedDirectoryTable => "volumeID/bucketID/parentId/dirName"
- String omKey = omMetadataManager.getOzonePathKey(volumeObjectId,
- bucketObjectId, parentObjectId, dirName);
+ // DB key in DeletedDirectoryTable =>
+ // "volumeID/bucketID/parentId/dirName/dirObjectId"
+
+ String ozoneDbKey = omMetadataManager.getOzonePathKey(volumeObjectId,
+ bucketObjectId, parentObjectId, dirName);
+ String ozoneDeleteKey = omMetadataManager.getOzoneDeletePathKey(
+ objectId, ozoneDbKey);
+
Review Comment:
What caused this change? Why did the DB key change? Or this was incorrect
before?
##########
hadoop-ozone/recon/src/test/java/org/apache/hadoop/ozone/recon/tasks/TestOmTableInsightTask.java:
##########
@@ -1,40 +1,31 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- * <p>
- * http://www.apache.org/licenses/LICENSE-2.0
- * <p>
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
Review Comment:
This shouldn't be removed.
##########
hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/OmTableHandler.java:
##########
@@ -0,0 +1,87 @@
+package org.apache.hadoop.ozone.recon.tasks;
+
+import org.apache.commons.lang3.tuple.Triple;
+import org.apache.hadoop.hdds.utils.db.Table;
+import org.apache.hadoop.hdds.utils.db.TableIterator;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.HashMap;
+
+/**
+ * Interface for handling PUT, DELETE and UPDATE events for size-related
+ * tables for OM Insights.
+ */
+public interface OmTableHandler {
+
+ /**
+ * Handles a PUT event for size-related tables by updating both the data
+ * sizes and their corresponding record counts in the tables.
+ *
+ * @param event The PUT event to be processed.
+ * @param tableName Table name associated with the event.
+ * @param sizeRelatedTables Tables Requiring Size Calculation.
Review Comment:
Maybe I missed something, but I don't see any usages why we should pass the
`sizeRelatedTables` to the `handlePutEvent()`, `handleDeleteEvent()` and to the
`handleUpdateEvent()`. Can you check this out? I didn't see usages in any of
the `OmTableHandler` implementations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
