[
https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16534859#comment-16534859
]
Hari Sekhon edited comment on HDFS-13724 at 7/6/18 2:26 PM:
------------------------------------------------------------
I tried a workaround of dupming the for now is to do the following as hdfs
superuser - dump the fsimage, convert to XML and then grep the tiers path info:
{code:java}
su - hdfs
kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
hdfs dfsadmin -fetchImage .
# this step might take a long time on big clusters (eg. 20 mins for 12GB
fsimage.xml result file from a moderate 600TB cluster)
hadoop oiv -i $(ls -tr fsimage_* | tail -n1) -p XML -o fsimage.xml
grep ... fsimage.xml{code}
but I can't find anything relating to 'policy' or the name of our storage
policy or the directory I know it's applied to.
was (Author: harisekhon):
I tried a workaround of dupming the for now is to do the following as hdfs
superuser - dump the fsimage, convert to XML and then grep the tiers path info:
{code:java}
su - hdfs
kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
hdfs dfsadmin -fetchImage .
# this step might take a long time on big clusters (eg. 20 mins for 12GB
fsimage.xml result file from a moderate 600TB cluster)
hadoop oiv -i $(ls -tr fsimage_* | tail -n1) -p XML -o fsimage.xml
grep ...{code}
but I can't find anything relating to 'policy' or the name of our storage
policy or the directory I know it's applied to.
> Storage Tiering Show Paths with Policies applied
> ------------------------------------------------
>
> Key: HDFS-13724
> URL: https://issues.apache.org/jira/browse/HDFS-13724
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Hari Sekhon
> Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for
> which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to
> determine if one has been applied, but if another hadoop admin has configured
> policies on anything but trivially obvious paths such as /archive then there
> is no way to find which paths have policies applied to them other than by
> querying every single directory and subdirectory one by one which might
> potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah
> which doesn't show when trying
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory
> in hdfs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
