[
https://issues.apache.org/jira/browse/HDDS-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597588#comment-17597588
]
Kaijie Chen commented on HDDS-7177:
-----------------------------------
Seems it is cause by PutBlock failure when writing the first stripe.
Here's logs of writing block 109611004723200568 on each DN (index not
corresponding to replica index).
{noformat}
DN1:
2022-08-30 11:41:07,944 [ChunkReader-1] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
2022-08-30 11:41:15,092 [ChunkReader-7] DEBUG
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher:
[service=DatanodeClient] [type=PutBlock] request is processed. Response:
<json>cmdType: PutBlock\ntraceID: ""\nresult: SUCCESS\nputBlock {\n
committedBlockLength {\n blockID {\n containerID: 23\n localID:
109611004723200568\n blockCommitSequenceId: 0\n }\n blockLength:
1048576\n }\n}\n</json>
DN2:
2022-08-30 11:41:07,945 [ChunkReader-5] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN3:
2022-08-30 11:41:07,941 [ChunkReader-5] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
2022-08-30 11:41:15,092 [ChunkReader-5] DEBUG
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher:
[service=DatanodeClient] [type=PutBlock] request is processed. Response:
<json>cmdType: PutBlock\ntraceID: ""\nresult: SUCCESS\nputBlock {\n
committedBlockLength {\n blockID {\n containerID: 23\n localID:
109611004723200568\n blockCommitSequenceId: 0\n }\n blockLength:
1048576\n }\n}\n</json>
DN4:
2022-08-30 11:41:07,943 [ChunkReader-8] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
2022-08-30 11:41:15,092 [ChunkReader-9] DEBUG
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher:
[service=DatanodeClient] [type=PutBlock] request is processed. Response:
<json>cmdType: PutBlock\ntraceID: ""\nresult: SUCCESS\nputBlock {\n
committedBlockLength {\n blockID {\n containerID: 23\n localID:
109611004723200568\n blockCommitSequenceId: 0\n }\n blockLength:
1048576\n }\n}\n</json>
DN5:
2022-08-30 11:41:07,942 [ChunkReader-5] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN6:
2022-08-30 11:41:07,943 [ChunkReader-3] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
2022-08-30 11:41:15,092 [ChunkReader-6] DEBUG
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher:
[service=DatanodeClient] [type=PutBlock] request is processed. Response:
<json>cmdType: PutBlock\ntraceID: ""\nresult: SUCCESS\nputBlock {\n
committedBlockLength {\n blockID {\n containerID: 23\n localID:
109611004723200568\n blockCommitSequenceId: 0\n }\n blockLength:
1048576\n }\n}\n</json>
DN7:
2022-08-30 11:41:07,946 [ChunkReader-8] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/ozone-ec/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8DN1urrent/containerDir0/23/chunks/109611004723200568.block
DN8:
2022-08-30 11:41:08,038 [ChunkReader-5] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/ozone-ec/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
2022-08-30 11:41:15,107 [ChunkReader-3] DEBUG
org.apache.hadoop.ozone.container.common.impl.HddsDispatcher:
[service=DatanodeClient] [type=PutBlock] request is processed. Response:
<json>cmdType: PutBlock\ntraceID: ""\nresult: SUCCESS\nputBlock {\n
committedBlockLength {\n blockID {\n containerID: 23\n localID:
109611004723200568\n blockCommitSequenceId: 0\n }\n blockLength:
1048576\n }\n}\n</json>
DN9:
2022-08-30 11:41:08,687 [ChunkReader-9] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN10:
2022-08-30 11:41:08,494 [ChunkReader-4] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN11:
2022-08-30 11:41:15,093 [ChunkReader-6] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN12:
2022-08-30 11:41:08,243 [ChunkReader-0] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN13:
2022-08-30 11:41:08,474 [ChunkReader-4] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
DN14:
2022-08-30 11:41:09,090 [ChunkReader-2] DEBUG
org.apache.hadoop.ozone.container.keyvalue.impl.FilePerBlockStrategy: Writing
chunk ChunkInfo{chunkName='109611004723200568_chunk_1, offset=0, len=1048576}
(overwrite: false) in stage COMBINED to file
/data/storage/hdds/CID-161ac683-0352-4be0-bdf6-ca762bf6ae8f/current/containerDir0/23/chunks/109611004723200568.block
{noformat}
> EC produces some unknown 1MB blocks without the control of deleting service
> ---------------------------------------------------------------------------
>
> Key: HDDS-7177
> URL: https://issues.apache.org/jira/browse/HDDS-7177
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Xu Shao Hong
> Priority: Major
> Attachments: image-2022-08-26-16-28-26-251.png
>
>
> We found that even deleted all the EC keys in the cluster and flushed the
> transactionbuffer, there are still some blocks left. They are all of 1M size.
> The config is rs-10-4-1024k
> This could be replayed.
> !image-2022-08-26-16-28-26-251.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
