Siddhant Sangwan created HDDS-12691:
---------------------------------------
Summary: Calculation of committed space in Datanode seems incorrect
Key: HDDS-12691
URL: https://issues.apache.org/jira/browse/HDDS-12691
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Datanode
Reporter: Siddhant Sangwan
The following method is called on every writeChunk request, after writing bytes
to the disk to update stats:
{code}
public void incrWriteBytes(long bytes) {
long unused = getMaxSize() - getBytesUsed();
this.writeBytes.addAndGet(bytes);
/*
Increase the cached Used Space in VolumeInfo as it
maybe not updated, DU or DedicatedDiskSpaceUsage runs
periodically to update the Used Space in VolumeInfo.
*/
this.getVolume().incrementUsedSpace(bytes);
// only if container size < max size
if (committedSpace && unused > 0) {
//with this write, container size might breach max size
long decrement = Math.min(bytes, unused);
this.getVolume().incCommittedBytes(0 - decrement);
}
}
{code}
For a container, 5GB (default) is committed when the container is created. Then
every writeChunk is supposed to increment usedSpace and decrement
committedSpace. Here, the logic of long decrement = Math.min(bytes, unused);
seems incorrect to me.
Consider that the latest writeChunk wrote 256MB into a close-to-full container.
Assume that now, unused = getMaxSize() - getBytesUsed() = 100MB. The current
logic would decrement the committedSpace by 100MB, whereas it should have been
decremented by 256MB.
Moreover, what about the case where the because of the latest chunk, the used
size of the container exceeds the max size (used > max, for example I've seen
13GB containers in actual clusters since the max is just a soft limit),
shouldn't we still decrement the committed space by getMaxSize() -
(getBytesUsed() - bytes)? That's the remaining amount of committed space that
this container was taking up in the volume before the "bytes" sized chunk was
written to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
