[
https://issues.apache.org/jira/browse/HDDS-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785123#comment-16785123
]
Bharat Viswanadham commented on HDDS-1163:
------------------------------------------
Hi [~sdeka]
Thank You for the patch.
# I don't think we need to store SCMID inside each container metadata file. As
for a particular SCM created containers, they all will be inside a SCMID
directory on a datanode. As if you see when we create Container, we pass scmID.
(Ex:getContainerMetaDataPath). Not sure if there is any other reason for having
this.
# Minor Nit: Avoid * in imports.
# Can we use enum name as CamelCase, instead of All caps.
# Do we need this check in KeyValueContainerCheck.java Line 424-427.
# In checkContainerFile, We need to check chunksPath, RocksDBType,
layOutversion
# Can we add Javadocs for methods, and also for Enums of error codes could you
add the explanation, it will be easy to understand the code.
# Can we change the log level to trace/debug for in KeyValueContainerCheck().
# During OzoneContainer() stop, we need to stop the ContainerCheck background
running thread.
Few questions:
# When a background delete is happening on the thread, we tried for 3 times
even if we are getting the error we break the loop, so that this container will
be scanned in next iteration. Can we note these containers into a list, and
during next scan they get preferred first.
# And also ContainerCheckScanner can be split in two, fast scan for all open
containers, and full scan for all closed containers. So, we can have 2 kinds of
Scanners in the system.
> Basic framework for Ozone Data Scrubber
> ---------------------------------------
>
> Key: HDDS-1163
> URL: https://issues.apache.org/jira/browse/HDDS-1163
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
> Components: Ozone Datanode
> Reporter: Supratim Deka
> Assignee: Supratim Deka
> Priority: Major
> Attachments: HDDS-1163.000.patch, HDDS-1163.001.patch,
> HDDS-1163.002.patch
>
>
> Included in the scope:
> 1. Background scanner thread to iterate over container set and dispatch check
> tasks for individual containers
> 2. Fixed rate scheduling - dispatch tasks at a pre-determined rate (for
> example 1 container/s)
> 3. Check disk layout of Container - basic check for integrity of the
> directory hierarchy inside the container, include chunk directory and
> metadata directories
> 4. Check container file - basic sanity checks for the container metafile
> 5. Check Block Database - iterate over entries in the container block
> database and check for the existence and accessibility of the chunks for each
> block.
> Not in scope (will be done as separate subtasks):
> 1. Dynamic scheduling/pacing of background scan based on system load and
> available resources.
> 2. Detection and handling of orphan chunks
> 3. Checksum verification for Chunks
> 4. Corruption handling - reporting (to SCM) and subsequent handling of any
> corruption detected by the scanner. The current subtask will simply log any
> corruption which is detected.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
