Hey Sanjeev,
While figuring out the UI ports and statistics, would like to add
another question.
So let's assume our scenario once more. (Data nodes *dn01* to *dn10*
exist. Each data node has 10 x 10TB drives. ) Also remembering that the
check takes 3 weeks (504 hours) and runs at 1MB/s.
In order to check the CRC 32C checksum, assuming maximum speed of
300MB/s, drives at 75% capacity, and that disks are checked in parallel,
these are the timings I'm getting:
1) 300MB /s : 7.5TB (7864320MB) . 7864320/300 * 60 * 60 = 7.28 Hours.
2) 1MB /s : 7.5TB (7864320MB) . 7864320/1 * 60 * 60 = 2184
Hours. This equals 89.5 days. This would overlap with the next 3 week
cycle (21 days). Therefore it can't effectively complete.
Given this, I'm thinking in order for HDFS to properly check larger
clusters, the default values of
*dfs.block.scanner.volume.bytes.per.second* and
**dfs.datanode.scan.period.hours**would need to be adjusted. Otherwise,
there is a potential that HDFS block (volume) scanner isn't checking
things well.
Am I correct in the above?
****
Thx,
TK
Did these tests to get a rough estimate of the time it takes for the
various checksums to complete:
root / home time gsutil hash BigFile17GiBinRAM;
Hashes [base64] for BigFile17GiBinRAM:7 MiB/s
Hash (crc32c): lRucsQ==
Hash (md5): v3yZMSuBlJ/q1N8Xavshgg==
Operation completed over 1 objects/16.0 GiB.
real 4m7.993s
user 3m44.568s
sys 0m22.338s
root / home time sha512sum BigFile17GiBinRAM
ac2c90c6acfeed591530ad9db639734308ef1bf494a3608c10f6334b933149d35c8098791a460fcdc302dcc921a4e16e2821712d8bbbb5d51ea8f5dc329de0ad
BigFile17GiBinRAM
real 2m21.008s
user 1m51.815s
sys 0m15.877s
root / home time sha1sum BigFile17GiBinRAM
69d37b2c75a13fc02e99bf8e0205ce2a09d98b24 BigFile17GiBinRAM
real 1m18.117s
user 1m1.428s
sys 0m15.505s
root / home time gsutil hash BigFile17GiBinRAM;
Hashes [base64] for BigFile17GiBinRAM:3 MiB/s
Hash (crc32c): lRucsQ==
Hash (md5): v3yZMSuBlJ/q1N8Xavshgg==
Operation completed over 1 objects/16.0 GiB.
real 4m9.437s
user 3m43.473s
sys 0m23.597s
root / home
On 10/22/2020 9:48 PM, TomK wrote:
Hey Austin, Sanjeev,
The ports defined are as follows in hdfs-site.xml:
cm-r01wn01.mws.mds.xyz root … run cloudera-scm-agent process grep
-Ei "dfs.datanode.http.address|dfs.datanode.https.address" -A 2
./3370-hdfs-DATANODE/hdfs-site.xml
<name>dfs.datanode.http.address</name>
<value>cm-r01wn01.mws.mds.xyz:1006</value>
</property>
--
<name>dfs.datanode.https.address</name>
<value>cm-r01wn01.mws.mds.xyz:9865</value>
</property>
cm-r01wn01.mws.mds.xyz root … run cloudera-scm-agent process
Checking the ports used:
cm-r01wn01.mws.mds.xyz root ~ netstat -pnltu|grep -Ei
"9866|1004|9864|9865|1006|9867"
tcp 0 0 10.3.0.160:9867 0.0.0.0:*
LISTEN 30096/jsvc.exec
tcp 0 0 10.3.0.160:1004 0.0.0.0:*
LISTEN 30096/jsvc.exec
tcp 0 0 10.3.0.160:1006 0.0.0.0:*
LISTEN 30096/jsvc.exec
cm-r01wn01.mws.mds.xyz root ~
cm-r01wn01.mws.mds.xyz root ~
cm-r01wn01.mws.mds.xyz root ~ hds getconf -confKey
dfs.datanode.address
-bash: hds: command not found
cm-r01wn01.mws.mds.xyz root ~ hdfs getconf -confKey
dfs.datanode.address
0.0.0.0:9866
cm-r01wn01.mws.mds.xyz root ~ hdfs getconf -confKey
dfs.datanode.http.address
0.0.0.0:9864
cm-r01wn01.mws.mds.xyz root ~ hdfs getconf -confKey
dfs.datanode.https.address
0.0.0.0:9865
cm-r01wn01.mws.mds.xyz root ~ hdfs getconf -confKey
dfs.datanode.ipc.address
0.0.0.0:9867
cm-r01wn01.mws.mds.xyz root ~
The scanner looks to be initialized:
cm-r01wn01.mws.mds.xyz root / var log hadoop-hdfs grep
-EiR "Periodic block scanner is not running" *
cm-r01wn01.mws.mds.xyz root / var log hadoop-hdfs
cm-r01wn01.mws.mds.xyz root / var log hadoop-hdfs grep
-EiR "Initialized block scanner with targetBytesPerSec" *|wc -l;
32
cm-r01wn01.mws.mds.xyz root / var log hadoop-hdfs
And yes, indeed it is started up. It kicked off around the time when
I restarted the DataNode service.
cm-r01wn01.mws.mds.xyz root / var log hadoop-hdfs vi
hadoop-cmf-hdfs-DATANODE-cm-r01wn01.mws.mds.xyz.log.out
STARTUP_MSG: build = http://github.com/cloudera/hadoop -r
7f07ef8e6df428a8eb53009dc8d9a249dbbb50ad; compiled by 'jenkins' on
2019-07-18T17:09Z
STARTUP_MSG: java = 1.8.0_181
************************************************************/
2020-10-22 20:54:58,488 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: registered UNIX
signal handlers for [TERM, HUP, INT]
2020-10-22 20:54:59,762 INFO
org.apache.hadoop.security.UserGroupInformation: Login successful for
user hdfs/cm-r01wn01.mws.mds....@mws.mds.xyz using keytab file
hdfs.keytab. Keytab auto renewal enabled : false
2020-10-22 20:55:00,265 INFO
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker:
Scheduling a check for [DISK]file:/hdfs/1/dfs/dn
2020-10-22 20:55:00,295 INFO
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker:
Scheduling a check for [DISK]file:/hdfs/2/dfs/dn
2020-10-22 20:55:00,296 INFO
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker:
Scheduling a check for [DISK]file:/hdfs/3/dfs/dn
2020-10-22 20:55:00,297 INFO
org.apache.hadoop.hdfs.server.datanode.checker.ThrottledAsyncChecker:
Scheduling a check for [DISK]file:/hdfs/4/dfs/dn
2020-10-22 20:55:00,521 INFO
org.apache.hadoop.metrics2.impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2020-10-22 20:55:00,723 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2020-10-22 20:55:00,723 INFO
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: DataNode metrics
system started
2020-10-22 20:55:00,947 INFO
org.apache.hadoop.hdfs.server.common.Util:
dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling
file IO profiling
2020-10-22 20:55:00,953 INFO
org.apache.hadoop.hdfs.server.datanode.BlockScanner: *Initialized
block scanner with targetBytesPerSec 1048576*
2020-10-22 20:55:00,961 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: File descriptor
passing is enabled.
2020-10-22 20:55:00,963 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Configured hostname
is cm-r01wn01.mws.mds.xyz
2020-10-22 20:55:00,965 INFO
org.apache.hadoop.hdfs.server.common.Util:
dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling
file IO profiling
2020-10-22 20:55:00,995 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Starting DataNode
with maxLockedMemory = 299892736
2020-10-22 20:55:01,018 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Opened streaming
server at /10.3.0.160:1004
2020-10-22 20:55:01,023 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth
is 10485760 bytes/s
2020-10-22 20:55:01,024 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for
balancing is 50
2020-10-22 20:55:01,029 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Balancing bandwidth
is 10485760 bytes/s
2020-10-22 20:55:01,029 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Number threads for
balancing is 50
2020-10-22 20:55:01,029 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Listening on UNIX
domain socket: /var/run/hdfs-sockets/dn
2020-10-22 20:55:01,304 INFO org.eclipse.jetty.util.log: Logging
initialized @8929ms
2020-10-22 20:55:01,559 INFO org.apache.hadoop.http.HttpRequestLog:
Http request log for http.requests.datanode is not defined
2020-10-22 20:55:01,585 INFO org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2020-10-22 20:55:01,589 INFO org.apache.hadoop.http.HttpServer2: Added
filter authentication
(class=org.apache.hadoop.security.authentication.server.AuthenticationFilter)
to context datanode
This answers another question I had: under what conditions does the
block / volume checker kick off. When removing a datanode and adding
it back in, it appears the checker will get kicked off on the worker
at that time.
Only the Secure DataNode port returns a login, as to be expected. (
http://cm-r01wn01.mws.mds.xyz:1006/ )
Thx,
TK
On 10/22/2020 11:56 AM, संजीव (Sanjeev Tripurari) wrote:
Hi Tom,
Can you start your datanode service, and share the datanode logs,
check if it is started properly or not.
Regards
-Sanjeev
On Thu, 22 Oct 2020 at 20:33, Austin Hackett <hacketta...@me.com
<mailto:hacketta...@me.com>> wrote:
Hi Tom
It might be worth restarting the DataNode process? I didn’t think
you could disable the DataNode Web UI as such, but I could be
wrong on this point. Out of interest, what does hdfs-site.xml say
with regards to dfs.datanode.http.address/dfs.datanode.https.address?
Regarding the logs, a quick look on GitHub suggests there may be
a couple of useful log messages:
https://github.com/apache/hadoop/blob/88a9f42f320e7c16cf0b0b424283f8e4486ef286/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java
<https://github.com/apache/hadoop/blob/88a9f42f320e7c16cf0b0b424283f8e4486ef286/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockScanner.java>
For example, LOG.warn(“Periodic block scanner is not running”) or
LOG.info <http://LOG.info>(“Initialized block scanner with
targetBytesPerSec {}”).
Of course, you’d need make sure those LOG statements are present
in the Hadoop version included with CDH 6.3. Git “blame” suggests
the LOG statements were added 6 years, so chance are you have them...
Thanks
Austin
On 22 Oct 2020, at 14:44, TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Thanks Austin. However none of these are open on a standard
Cloudera 6.3 build.
# netstat -pnltu|grep -Ei "9866|1004|9864|9865|1006|9867"
#
Would there be anything in the logs to indicate whether or not
the block / volume scanner is running?
Thx,
TK
On 10/22/2020 3:09 AM, Austin Hackett wrote:
Hi Tom
I not too familiar with the CDH distribution, but this page has
the default ports used by DataNode:
https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_ports.html
<https://docs.cloudera.com/documentation/enterprise/latest/topics/cdh_ports.html>
I believe it’s the settings for
dfs.datanode.http.address/dfs.datanode.https.address that
you’re interested in (9864/9865)
Since the data block scanner related config parameters are not
set, the defaults of 3 weeks and 1MB should be applied.
Thanks
Austin
On 22 Oct 2020, at 06:35, TomK <tomk...@mdevsys.com>
<mailto:tomk...@mdevsys.com> wrote:
Hey Austin, Sanjeev,
Thanks once more! Took some time to review the pages. That
was certainly very helpful. Appreciated!
However, I tried to access https://dn01/blockScannerReport
<https://dn01/blockScannerReport> on a test Cloudera 6.3
cluster. Didn't work Tried the following as well:
http://dn01:50075/blockscannerreport?listblocks
<http://dn01:50075/blockscannerreport?listblocks>
https://dn01:50075/blockscannerreport
<https://dn01:50075/blockscannerreport>
https://dn01:10006/blockscannerreport
<https://dn01:10006/blockscannerreport>
Checked that port 50075 is up ( netstat -pnltu ). There's no
service on that port on the workers. Checked the pages:
https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_ports_cdh5.html
<https://docs.cloudera.com/documentation/enterprise/5-14-x/topics/cdh_ig_ports_cdh5.html>
It is defined on the pages. Checked if the following is set:
The following 2 configurations in/hdfs-site.xml/are the most
used for block scanners.
*
*
* *dfs.block.scanner.volume.bytes.per.second* to throttle
the scan bandwidth to configurable bytes per second.
*Default value is 1M*. Setting this to 0 will disable the
block scanner.
* *dfs.datanode.scan.period.hours*to configure the scan
period, which defines how often a whole scan is performed.
This should be set to a long enough interval to really
take effect, for the reasons explained above. *Default
value is 3 weeks (504 hours)*. Setting this to 0 will use
the default value. Setting this to a negative value will
disable the block scanner.
These are NOT explicitly set. Checked hdfs-site.xml. Nothing
defined there. Checked the Configuration tab in the cluster.
It's not defined either.
Does this mean that the defaults are applied OR does it mean
that the block / volume scanner is disabled? I see the pages
detail what values for these settings mean but I didn't see
any notes pertaining to the situation where both values are
not explicitly set.
Thx,
TK
On 10/21/2020 1:34 PM, संजीव (Sanjeev Tripurari) wrote:
Yes Austin,
you are right every datanode will do its block verification,
which is send as health check report to the namenode
Regards
-Sanjeev
On Wed, 21 Oct 2020 at 21:53, Austin Hackett
<hacketta...@me.com <mailto:hacketta...@me.com>> wrote:
Hi Tom
It is my understanding that in addition to block
verification on client reads, each data node runs a
DataBlockScanner in a background thread that periodically
verifies all the blocks stored on the data node. The
dfs.datanode.scan.period.hours property controls how
often this verification occurs.
I think the reports are available via the data node
/blockScannerReport HTTP endpoint, although I’m not sure
I ever actually looked at one. (add ?listblocks to get
the verification status of each block).
More info here:
https://blog.cloudera.com/hdfs-datanode-scanners-and-disk-checker-explained/
<https://blog.cloudera.com/hdfs-datanode-scanners-and-disk-checker-explained/>
Thanks
Austin
On 21 Oct 2020, at 16:47, TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Hey Sanjeev,
Allright. Thank you once more. This is clear.
However, this poses an issue then. If during the two
years, disk drives develop bad blocks but do not
necessarily fail to the point that they cannot be
mounted, that checksum would have changed since those
filesystem blocks can no longer be read. However, from
an HDFS perspective, since no checks are done regularly,
that is not known. So HDFS still reports that the file
is fine, in other words, no missing blocks. For
example, if a disk is going bad, but those files are not
read for two years, the system won't know that there is
a problem. Even when removing a data node temporarily
and re-adding the datanode, HDFS isn't checking because
that HDFS file isn't read.
So let's assume this scenario. Data nodes *dn01* to
*dn10* exist. Each data node has 10 x 10TB drives.
And let's assume that there is one large file on those
drives and it's replicated to factor of X3.
If during the two years the file isn't read, and 10 of
those drives develop bad blocks or other underlying
hardware issues, then it is possible that HDFS will
still report everything fine, even with a replication
factor of 3. Because with 10 disks failing, it's
possible a block or sector has failed under each of the
3 copies of the data. But HDFS would NOT know since
nothing triggered a read of that HDFS file. Based on
everything below, then corruption is very much possible
even with a replication of factor X3. A this point the
file is unreadable but HDFS still reports no missing
blocks.
Similarly, if once I take a data node out, I adjust one
of the files on the data disks, HDFS will not know and
still report everything fine. That is until someone
read's the file.
Sounds like this is a very real possibility.
Thx,
TK
On 10/21/2020 10:26 AM, संजीव (Sanjeev Tripurari) wrote:
Hi Tom
Therefore, if I write a file to HDFS but access it two
years later, then the checksum will be computed only
twice, at the beginning of the two years and again at
the end when a client connects? Correct? As long as no
process ever accesses the file between now and two
years from now, the checksum is never redone and
compared to the two year old checksum in the fsimage?
yes, Exactly unless data is read checksum is not
verified. (when data is written and when the data is
read),
if checksum is mismatched, there is no way to correct
it, you will have to re-write that file.
When datanode is added back in, there is no real read
operation on the files themselves. The datanode just
reports the blocks but doesn't really read the blocks
that are there to re-verify the files and ensure
consistency?
yes, Exactly, datanode maintains list of files and
their blocks, which it reports, along with total disk
size and used size.
Namenode only has list of blocks, unless datanodes is
connected it wont know where the blocks are stored.
Regards
-Sanjeev
On Wed, 21 Oct 2020 at 18:31, TomK <tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Hey Sanjeev,
Thank you very much again. This confirms my suspision.
Therefore, if I write a file to HDFS but access it
two years later, then the checksum will be computed
only twice, at the beginning of the two years and
again at the end when a client connects? Correct?
As long as no process ever accesses the file
between now and two years from now, the checksum is
never redone and compared to the two year old
checksum in the fsimage?
When datanode is added back in, there is no real
read operation on the files themselves. The
datanode just reports the blocks but doesn't really
read the blocks that are there to re-verify the
files and ensure consistency?
Thx,
TK
On 10/21/2020 12:38 AM, संजीव (Sanjeev Tripurari) wrote:
Hi Tom,
Every datanode sends heartbeat to namenode, on its
list of blocks it has.
When a datanode which is disconnected for a while,
after connecting will send heartbeat to namenode,
with list of blocks it has (till then namenode
will have under-replicated blocks).
As soon as the datanode is connected to namenode,
it will clear under-replicatred blocks.
*When a client connects to read or write a file,
it will run checksum to validate the file.*
There is no independent process running to do
checksum, as it will be heavy process on each node.
Regards
-Sanjeev
On Wed, 21 Oct 2020 at 00:18, Tom <t...@mdevsys.com
<mailto:t...@mdevsys.com>> wrote:
Thank you. That part I understand and am Ok
with it.
What I would like to know next is when again
the CRC32C checksum is ran and checked against
the fsimage that the block file has not
changed or become corrupted?
For example, if I take a datanode out, and
within 15 minutes, plug it back in, does HDF
rerun the CRC 32C on all data disks on that
node to make sure blocks are ok?
Cheers,
TK
Sent from my iPhone
On Oct 20, 2020, at 1:39 PM, संजीव (Sanjeev
Tripurari) <sanjeevtripur...@gmail.com
<mailto:sanjeevtripur...@gmail.com>> wrote:
its done as sson as a file is stored on disk..
Sanjeev
On Tuesday, 20 October 2020, TomK
<tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Thanks again.
At what points is the checksum validated
(checked) after that? For example, is it
done on a daily basis or is it done only
when the file is accessed?
Thx,
TK
On 10/20/2020 10:18 AM, संजीव (Sanjeev
Tripurari) wrote:
As soon as the file is written first
time checksum is calculated and updated
in fsimage (first in edit logs), and
same is replicated other replicas.
On Tue, 20 Oct 2020 at 19:15, TomK
<tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Hi Sanjeev,
Thank you. It does help.
At what points is the checksum
calculated?
Thx,
TK
On 10/20/2020 3:03 AM, संजीव
(Sanjeev Tripurari) wrote:
For Missing blocks and corrupted
blocks, do check if all the
datanode services are up, non of
the disks where hdfs data is stored
is accessible and have no issues,
hosts are reachable from namenode,
If you are able to re-generate the
data and write its great, otherwise
hadoop cannot correct itself.
Could you please elaborate on this?
Does it mean I have to continuously
access a file for HDFS to be able to
detect corrupt blocks and correct
itself?
*"Does HDFS check that the data
node is up, data disk is mounted,
path to
the file exists and file can be read?"*
-- yes, only after it fails it will
say missing blocks.
*Or does it also do a filesystem
check on that data disk as well as
perhaps a checksum to ensure block
integrity?*
-- yes, every file cheksum is
maintained and cross checked, if it
fails it will say corrupted blocks.
hope this helps.
-Sanjeev
*
*
On Tue, 20 Oct 2020 at 09:52, TomK
<tomk...@mdevsys.com
<mailto:tomk...@mdevsys.com>> wrote:
Hello,
HDFS Missing Blocks / Corrupt
Blocks Logic: What are the
specific
checks done to determine a
block is bad and needs to be
replicated?
Does HDFS check that the data
node is up, data disk is
mounted, path to
the file exists and file can be
read?
Or does it also do a filesystem
check on that data disk as well as
perhaps a checksum to ensure
block integrity?
I've googled on this quite a
bit. I don't see the exact
answer I'm
looking for. I would like to
know exactly what happens
during file
integrity verification that
then constitutes missing blocks
or corrupt
blocks in the reports.
--
Thank You,
TK.
---------------------------------------------------------------------
To unsubscribe, e-mail:
user-unsubscr...@hadoop.apache.org
<mailto:user-unsubscr...@hadoop.apache.org>
For additional commands,
e-mail:
user-h...@hadoop.apache.org
<mailto:user-h...@hadoop.apache.org>
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
--
Thx,
TK.
