mismatch corrupt blocks from fsck and dfadmin report.

[권병창]

Hi. hadooper!
 
I use hadoop-2.7.1 and  my cluster has 130 nodes.
 
recently I am facing a problem.
 
I have found corrupt block by nagios.
 
nagios request 
http://namenode01:50070/jmx?qry=Hadoop:service=NameNode,name=FSNamesystem. 
 
below is result.  notice CorruptBlocks is 1. 

 {
  "beans" : [ {
"name" : "Hadoop:service=NameNode,name=FSNamesystem",
"modelerType" : "FSNamesystem",
"tag.Context" : "dfs",
"tag.HAState" : "active",
"tag.Hostname" : "css0700.nhnsystem.com",
"MissingBlocks" : 0,
"MissingReplOneBlocks" : 0,
"ExpiredHeartbeats" : 10,
"TransactionsSinceLastCheckpoint" : 820630,
"TransactionsSinceLastLogRoll" : 1916,
"LastWrittenTransactionId" : 376685578,
"LastCheckpointTime" : 1452583950883,
"CapacityTotal" : 1650893130075660,
"CapacityTotalGB" : 1537514.0,
"CapacityUsed" : 1237079990848257,
"CapacityUsedGB" : 1152121.0,
"CapacityRemaining" : 410189981364473,
"CapacityRemainingGB" : 382019.0,
"CapacityUsedNonDFS" : 3623157862930,
"TotalLoad" : 6717,
"SnapshottableDirectories" : 0,
"Snapshots" : 0,
"BlocksTotal" : 4034155,
"FilesTotal" : 2866690,
"PendingReplicationBlocks" : 0,
"UnderReplicatedBlocks" : 0,
"CorruptBlocks" : 1,
"ScheduledReplicationBlocks" : 0,
"PendingDeletionBlocks" : 0,
"ExcessBlocks" : 87,
"PostponedMisreplicatedBlocks" : 0,
"PendingDataNodeMessageCount" : 0,
"MillisSinceLastLoadedEdits" : 0,
"BlockCapacity" : 67108864,
"StaleDataNodes" : 0,
"TotalFiles" : 2866690
  } ]
}


 
 
however  'hdfs fsck / -list-corruptfileblocks' does not found.
 
below is fsck result.

 The filesystem under path '/' has 0 CORRUPT files



 
'hdfs dfsadmin -report'  is similar first result.

 Configured Capacity: 1650892318461452 (1.47 PB)
Present Capacity: 1647353181258422 (1.46 PB)
DFS Remaining: 408711410865856 (371.72 TB)
DFS Used: 1238641770392566 (1.10 PB)
DFS Used%: 75.19%
Under replicated blocks: 0
Blocks with corrupt replicas: 1
Missing blocks: 0
Missing blocks (with replication factor 1): 0



 
My question is
 
1. why different result?
2. How do I find corrupt filename? I wonder which file is corrupt.
 
Thank you.
 

Kerberos authentication using username/pswd

[Frank Luo]

Everywhere I searched, there are plenty samples of using keytab file for 
Kerberos authentication, but haven’t yet find anything samples to do so using 
username/pswd. I thought it was impossible until “Toad For Hadoop” started 
doing that.

So doesn’t anyone know how it is implemented? In particular, I’d like to run a 
java program in Windows, authenticate against Kerberos using username/pswd, and 
interact with Hadoop.

The how-to for Toad For Hadoop can be found here. BTW, it is a pretty neat tool 
for interacting with Hadoop world.

http://www.toadworld.com/products/toad-for-hadoop/b/weblog/archive/2015/08/10/connecting-to-your-kerberized-hadoop-environment-with-toad-for-hadoop

[”MerkleONE”]

This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 
communication may be confidential and may be subject to the attorney-client 
privilege.