[
https://issues.apache.org/jira/browse/HDDS-1535?focusedWorklogId=246317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-246317
]
ASF GitHub Bot logged work on HDDS-1535:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/May/19 19:09
Start Date: 21/May/19 19:09
Worklog Time Spent: 10m
Work Description: arp7 commented on pull request #832: HDDS-1535. Space
tracking for Open Containers : Handle Node Startup. Contributed by Supratim Deka
URL: https://github.com/apache/hadoop/pull/832#discussion_r286181220
##########
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/ContainerReader.java
##########
@@ -215,4 +224,27 @@ public void verifyContainerData(ContainerData
containerData)
ContainerProtos.Result.UNKNOWN_CONTAINER_TYPE);
}
}
+
+ private void initializeUsedBytes(KeyValueContainer container)
+ throws IOException {
+ KeyValueBlockIterator blockIter = new KeyValueBlockIterator(
+ container.getContainerData().getContainerID(),
+ new File(container.getContainerData().getContainerPath()));
+ long usedBytes = 0;
+
+ while (blockIter.hasNext()) {
+ BlockData block = blockIter.nextBlock();
+ long blockLen = 0;
+
+ List<ContainerProtos.ChunkInfo> chunkInfoList = block.getChunks();
Review comment:
Hi @supratimdeka , it looks like we are initializing using the metadata in
RocksDB. I wonder if it is better to initialize this value using bytes on disk
i.e. count usage of all the chunk files.
Both should match usually. However if there are chunk files that were
written but never committed successfully then there could be a discrepancy.
i.e. failed commit.
I don't think we are actively deleting such unreferenced chunk files
currently, so if there are lot of failed writes the container could go far
beyond its maximum capacity. This is a failure that could never be seen by
DataNode so short of deleting the chunk files via scanner there is not much we
can do.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 246317)
Time Spent: 0.5h (was: 20m)
> Space tracking for Open Containers : Handle Node Startup
> --------------------------------------------------------
>
> Key: HDDS-1535
> URL: https://issues.apache.org/jira/browse/HDDS-1535
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Supratim Deka
> Assignee: Supratim Deka
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This is related to HDDS-1511
> Space tracking for Open Containers (committed space in the volume) relies on
> usedBytes in the Container state. usedBytes is not persisted for every update
> (chunkWrite). So on a node restart the value is stale.
> The proposal is to:
> iterate the block DB for each open container during startup and compute the
> used space.
> The block DB process will be accelerated by spawning executors for each
> container.
> This process will be carried out as part of building the container set during
> startup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
