[
https://issues.apache.org/jira/browse/HDFS-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14254474#comment-14254474
]
Colin Patrick McCabe edited comment on HDFS-7430 at 12/20/14 2:53 AM:
----------------------------------------------------------------------
So, I changed the rate calculation a bit. Now it simply scans at the
configured bytes per second in the case in which the average bytes per second
in the last hour is lower than the target. If the average bytes per second is
too high, it stops scanning. I was having some trouble getting the rate
calculation to be exactly what I wanted (It's kind of a PID control problem),
so I think a fixed rate is easier to understand. Added a unit test for this as
well.
was (Author: cmccabe):
So, I changed the rate calculation a bit. Now it simply scans at the
configured bytes per second the average bytes per second in the last hour is
lower than the target, and scans nothing if not. I was having some trouble
getting the rate calculation to be exactly what I wanted (It's kind of a PID
control problem), so I think a fixed rate is easier to understand. Added a
unit test for this as well.
> Refactor the BlockScanner to use O(1) memory and use multiple threads
> ---------------------------------------------------------------------
>
> Key: HDFS-7430
> URL: https://issues.apache.org/jira/browse/HDFS-7430
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 2.7.0
> Reporter: Colin Patrick McCabe
> Assignee: Colin Patrick McCabe
> Attachments: HDFS-7430.002.patch, HDFS-7430.003.patch,
> HDFS-7430.004.patch, HDFS-7430.005.patch, HDFS-7430.006.patch,
> HDFS-7430.007.patch, memory.png
>
>
> We should update the BlockScanner to use a constant amount of memory by
> keeping track of what block was scanned last, rather than by tracking the
> scan status of all blocks in memory. Also, instead of having just one
> thread, we should have a verification thread per hard disk (or other volume),
> scanning at a configurable rate of bytes per second.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
