[
https://issues.apache.org/jira/browse/HDDS-7177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17597611#comment-17597611
]
Kaijie Chen commented on HDDS-7177:
-----------------------------------
Another problem I found is, stripe rewriting is very common.
Which results to many small block files on disk.
It's better to retry a few times before writing to a new block group.
{noformat}
39.90% |████████████████████████████████████████
| 399/1000 Time: 0:07:102022-08-30 15:26:24,575
[pool-2-thread-10] WARN io.KeyOutput
Stream: Rewriting stripe to new block group
2022-08-30 15:26:24,596 [pool-2-thread-85] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,604 [pool-2-thread-53] WARN io.KeyOutputStream: Rewriting
stripe to new block group
40.20% |█████████████████████████████████████████
| 402/1000 Time: 0:07:112022-08-30 15:26:24,786
[pool-2-thread-16] WARN io.KeyOutput
Stream: Rewriting stripe to new block group
2022-08-30 15:26:24,849 [pool-2-thread-79] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-2] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-28] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-9] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-82] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-21] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-33] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,856 [pool-2-thread-39] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,903 [pool-2-thread-15] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:24,920 [pool-2-thread-77] WARN io.KeyOutputStream: Rewriting
stripe to new block group
41.10% |██████████████████████████████████████████
| 411/1000 Time: 0:07:182022-08-30 15:26:31,684
[pool-2-thread-11] WARN io.KeyOutputStream: Rewriting stripe to new block group
2022-08-30 15:26:31,684 [pool-2-thread-90] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,187 [pool-2-thread-45] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,187 [pool-2-thread-92] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,197 [pool-2-thread-6] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,254 [pool-2-thread-24] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,254 [pool-2-thread-13] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:32,255 [pool-2-thread-99] WARN io.KeyOutputStream: Rewriting
stripe to new block group
41.50% |██████████████████████████████████████████
| 415/1000 Time: 0:07:232022-08-30 15:26:37,176
[pool-2-thread-58] WARN io.KeyOutputStream: Rewriting stripe to new block group
2022-08-30 15:26:37,419 [pool-2-thread-99] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,427 [pool-2-thread-13] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,535 [pool-2-thread-64] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,536 [pool-2-thread-8] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,537 [pool-2-thread-61] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,554 [pool-2-thread-3] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,571 [pool-2-thread-85] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:37,576 [pool-2-thread-46] WARN io.KeyOutputStream: Rewriting
stripe to new block group
41.50% |██████████████████████████████████████████
| 415/1000 Time: 0:07:242022-08-30 15:26:38,262
[pool-2-thread-32] WARN io.KeyOutputStream: Rewriting stripe to new block group
2022-08-30 15:26:38,262 [pool-2-thread-48] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,559 [pool-2-thread-47] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,559 [pool-2-thread-7] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,560 [pool-2-thread-98] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,560 [pool-2-thread-57] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,574 [pool-2-thread-43] WARN io.KeyOutputStream: Rewriting
stripe to new block group
2022-08-30 15:26:38,613 [pool-2-thread-25] WARN io.KeyOutputStream: Rewriting
stripe to new block group
44.00% |█████████████████████████████████████████████
{noformat}
> EC produces some unknown 1MB blocks without the control of deleting service
> ---------------------------------------------------------------------------
>
> Key: HDDS-7177
> URL: https://issues.apache.org/jira/browse/HDDS-7177
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Xu Shao Hong
> Priority: Major
> Attachments: image-2022-08-26-16-28-26-251.png
>
>
> We found that even deleted all the EC keys in the cluster and flushed the
> transactionbuffer, there are still some blocks left. They are all of 1M size.
> The config is rs-10-4-1024k
> This could be replayed.
> !image-2022-08-26-16-28-26-251.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
