[
https://issues.apache.org/jira/browse/HDDS-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16783142#comment-16783142
]
Yiqun Lin commented on HDDS-1163:
---------------------------------
Thanks for the response, [~sdeka].
{quote}3. Regarding separate executors for full-check and fast-check. Actually,
fast-check is a subset of the full-check. And the idea is to restrict to
fast-check for Containers which are still Open. Because this is simple. For
everything else always do full-check. Are you suggesting we have an executor
which keeps running fast-check on all Containers independently(and at a higher
frequency than full-check)?
{quote}
The main point from me is that the rate for full-check and fast-check should be
different. We can make fask-check for open state in a more higher frequency.
Currently, we put all check in same loop, sometimes the fast-check will be
blocked by a long time full-check. So I suggest we can have two executors: One
for fast-check and the other one for full-check. By default, we can make these
two executor both use 1 thread number size but different execution frequency.
How do you think of this?
Others make sense to me.
> Basic framework for Ozone Data Scrubber
> ---------------------------------------
>
> Key: HDDS-1163
> URL: https://issues.apache.org/jira/browse/HDDS-1163
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Supratim Deka
> Assignee: Supratim Deka
> Priority: Major
> Attachments: HDDS-1163.000.patch
>
>
> Included in the scope:
> 1. Background scanner thread to iterate over container set and dispatch check
> tasks for individual containers
> 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for
> example 1 container/s)
> 3. Check disk layout of Container - basic check for integrity of the
> directory hierarchy inside the container, include chunk directory and
> metadata directories
> 4. Check container file - basic sanity checks for the container metafile
> 5. Check Block Database - iterate over entries in the container block
> database and check for the existence and accessibility of the chunks for each
> block.
> Not in scope (will be done as separate subtasks):
> 1. Dynamic scheduling/pacing of background scan based on system load and
> available resources.
> 2. Detection and handling of orphan chunks
> 3. Checksum verification for Chunks
> 4. Corruption handling - reporting (to SCM) and subsequent handling of any
> corruption detected by the scanner. The current subtask will simply log any
> corruption which is detected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
