arp7 commented on a change in pull request #832: HDDS-1535. Space tracking for
Open Containers : Handle Node Startup. Contributed by Supratim Deka
URL: https://github.com/apache/hadoop/pull/832#discussion_r286181220
##########
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java
##########
@@ -215,4 +224,27 @@ public void verifyContainerData(ContainerData
containerData)
ContainerProtos.Result.UNKNOWN_CONTAINER_TYPE);
}
}
+
+ private void initializeUsedBytes(KeyValueContainer container)
+ throws IOException {
+ KeyValueBlockIterator blockIter = new KeyValueBlockIterator(
+ container.getContainerData().getContainerID(),
+ new File(container.getContainerData().getContainerPath()));
+ long usedBytes = 0;
+
+ while (blockIter.hasNext()) {
+ BlockData block = blockIter.nextBlock();
+ long blockLen = 0;
+
+ List<ContainerProtos.ChunkInfo> chunkInfoList = block.getChunks();
Review comment:
Hi @supratimdeka , it looks like we are initializing the counts using the
metadata in RocksDB. I wonder if it is better to initialize this value using
bytes on disk i.e. count usage of all the chunk files.
Both should match usually. However if there are chunk files that were
written but never committed successfully then there could be a discrepancy.
i.e. failed commit.
I don't think we are actively deleting such unreferenced chunk files
currently, so if there are lot of failed writes the container could go far
beyond its maximum capacity. This is a failure that could never be seen by
DataNode so short of deleting the chunk files via scanner there is not much we
can do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
