Ke Han created HDFS-17219:
-----------------------------
Summary: Inconsistent count results when upgrading hdfs clusetr
from 2.10.2 to 3.3.6
Key: HDFS-17219
URL: https://issues.apache.org/jira/browse/HDFS-17219
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 3.3.6, 2.10.2
Reporter: Ke Han
Attachments: hpLjvJVW.tar.gz
When upgrading hdfs cluster from 2.10.2 to 3.3.6, the results returned from
*dfs count* command is inconsistent.
h1. Reproduce
Start up 2.10.2 hdfs cluster (1 NN, 2 DN, 1 SNN), execute the following commands
{code:java}
dfs -mkdir /GscWZRxS
dfs -put -f -d /tmp/hpLjvJVW/cl /GscWZRxS/
dfs -put -f -d /tmp/hpLjvJVW/Zjpk /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR
dfsadmin -clrQuota /GscWZRxS/cl
dfs -expunge -immediate
dfsadmin -refreshSuperUserGroupsConfiguration
dfs -mkdir /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf/mGpVA
dfsadmin -refreshCallQueue
dfsadmin -saveNamespace -beforeShutdown
dfsadmin -refreshServiceAcl
dfsadmin -saveNamespace
dfsadmin -clrQuota /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -setSpaceQuota 2 -storageType DISK
/GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
dfsadmin -refreshNodes
dfsadmin -setSpaceQuota 2 -storageType DISK /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd
dfsadmin -clrSpaceQuota -storageType ARCHIVE /GscWZRxS/cl
dfsadmin -restoreFailedStorage true
dfs -chmod 22 2 /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/vGVnspAl/ih
dfs -chmod 22 2 /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/pqd
dfs -createSnapshot /GscWZRxS/cl mGpVA {code}
Now, before upgrade, I check the quota results
{code:java}
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
none inf none inf
/GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf {code}
Then prepare the upgrade: enter safemode, create image, shutdown the cluster
and start up the new cluster
{code:java}
bin/hdfs dfs -count -q -h -u /GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf
8.0 E 8.0 E none inf
/GscWZRxS/cl/lBsmFBlyBd/pozIeNFjzd/PsLbgpR/Zjpk/Cf {code}
The values of the first two columns are inconsistent with the quota I set
before.
I have attached the file used by the command. I am digging out the root cause,
I'll try to submit a patch once I can fix it. Any help is appreciated!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
