[jira] [Commented] (HDFS-13724) Storage Tiering Show Paths with Policies applied
[
https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536726#comment-16536726
]
Hari Sekhon commented on HDFS-13724:
I would guess that something along the lines of this would be more intuitive to
users:
{code:java}
hdfs storagepolicies -listPaths{code}
as that would be more inline with the already existing command which lists just
the policies without paths:
{code:java}
hdfs storagepolicies -listPolicies{code}
Alternatively it could be a switch to -listPolicies such as (in a similar
fashion to what is done with hdfs fsck files blocks locations switches):
{code:java}
hdfs storagepolicies -listPolicies -paths{code}
[~brahmareddy] grep -i storagepolicy fsimage.xml returns no hits. I actually
had a colleague double check this for me last week too, dumped all xml tags to
sort uniq and there was no such tag or anything that looked related, but there
are definitely storage policies applied as my hortonworks colleague who
configured this one told me the path and the following returns the policy
{code:java}
hdfs storagepolicies -getStoragePolicy -path /data/blah
The storage policy of /data/blah:
BlockStoragePolicy{COLD:2, storageTypes=[ARCHIVE], creationFallbacks=[],
replicationFallbacks=[]}
{code}
> Storage Tiering Show Paths with Policies applied
>
>
> Key: HDFS-13724
> URL: https://issues.apache.org/jira/browse/HDFS-13724
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hari Sekhon
>Assignee: Yuanbo Liu
>Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for
> which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to
> determine if one has been applied, but if another hadoop admin has configured
> policies on anything but trivially obvious paths such as /archive then there
> is no way to find which paths have policies applied to them other than by
> querying every single directory and subdirectory one by one which might
> potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah
> which doesn't show when trying
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory
> in hdfs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13724) Storage Tiering Show Paths with Policies applied
[
https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16535988#comment-16535988
]
Yuanbo Liu commented on HDFS-13724:
---
how about implementing a new command called "-find" to get the results?
> Storage Tiering Show Paths with Policies applied
>
>
> Key: HDFS-13724
> URL: https://issues.apache.org/jira/browse/HDFS-13724
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for
> which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to
> determine if one has been applied, but if another hadoop admin has configured
> policies on anything but trivially obvious paths such as /archive then there
> is no way to find which paths have policies applied to them other than by
> querying every single directory and subdirectory one by one which might
> potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah
> which doesn't show when trying
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory
> in hdfs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13724) Storage Tiering Show Paths with Policies applied
[
https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534959#comment-16534959
]
Brahma Reddy Battula commented on HDFS-13724:
-
bq. but I can't find anything relating to 'policy' or the name of our storage
policy or the directory I know it's applied to.
you can check/grep with *"storagePolicyId"* for finding the storagepolicy of a
file (i.e Policyid will stored in the fsiamge.)
*Example :*
if you set policy as *cold*, you can see like below.
2
> Storage Tiering Show Paths with Policies applied
>
>
> Key: HDFS-13724
> URL: https://issues.apache.org/jira/browse/HDFS-13724
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for
> which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to
> determine if one has been applied, but if another hadoop admin has configured
> policies on anything but trivially obvious paths such as /archive then there
> is no way to find which paths have policies applied to them other than by
> querying every single directory and subdirectory one by one which might
> potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah
> which doesn't show when trying
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory
> in hdfs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13724) Storage Tiering Show Paths with Policies applied
[
https://issues.apache.org/jira/browse/HDFS-13724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16534859#comment-16534859
]
Hari Sekhon commented on HDFS-13724:
I tried a workaround of dupming the for now is to do the following as hdfs
superuser - dump the fsimage, convert to XML and then grep the tiers path info:
{code:java}
su - hdfs
kinit -kt /etc/security/keytabs/hdfs.headless.keytab hdfs
hdfs dfsadmin -fetchImage .
# this step might take a long time on big clusters (eg. 20 mins for 12GB
fsimage.xml result file from a moderate 600TB cluster)
hadoop oiv -i $(ls -tr fsimage_* | tail -n1) -p XML -o fsimage.xml
grep ...{code}
but I can't find anything relating to 'policy' or the name of our storage
policy or the directory I know it's applied to.
> Storage Tiering Show Paths with Policies applied
>
>
> Key: HDFS-13724
> URL: https://issues.apache.org/jira/browse/HDFS-13724
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Hari Sekhon
>Priority: Major
>
> Improvement Request to add an hdfs storagepolicies command to find paths for
> which storage policies have been applied.
> Right now you must explicitly query a single directory to get its policy to
> determine if one has been applied, but if another hadoop admin has configured
> policies on anything but trivially obvious paths such as /archive then there
> is no way to find which paths have policies applied to them other than by
> querying every single directory and subdirectory one by one which might
> potentially have a policy, eg:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir2
> hdfs storagepolicies -getStoragePolicy -path /dir3
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir1
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir2
> hdfs storagepolicies -getStoragePolicy -path /dir3/subdir3
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN
> ...
> hdfs storagepolicies -getStoragePolicy -path /dirN/subdirN/subsubdirN
> ...{code}
> In my current environment for example, a policy was configured for /data/blah
> which doesn't show when trying
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data{code}
> and I had no way of knowing that I had to do:
> {code:java}
> hdfs storagepolicies -getStoragePolicy -path /data/blah{code}
> other than trial and error of trying every directory and every subdirectory
> in hdfs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
