[jira] [Commented] (HDFS-15553) Improve NameNode RPC throughput with ReadWriteRpcCallQueue
[
https://issues.apache.org/jira/browse/HDFS-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188628#comment-17188628
]
Wang, Xinglong commented on HDFS-15553:
---
[~kihwal] Thank you for the comment. It is really helpful.
{quote}Also there are write calls that initially acquire the read lock then
reacquire the write lock.
{quote}
Yes, I do notice this, the addBlock rpc is a such example to get read lock and
then write lock. It's a trouble maker. Currently, I think as for the read lock
part, it can share with other read rpc. So I put it in my read rpc queue.
For the internal rpc calls, I agree with you. E.g, IRB may have a impact on
complete rpc call result if there is a reorder. For other scenarios, I don't
have concrete case for now.
{quote} are you using async edit logging and audit logging?
{quote}
Yes. We are using both. And we have some enhancement for async edit logging
part. I can share in another thread.
{quote} -HDFS-9198-
{quote}
This patch is also in our production env. As for the 1024 queue size, I don't
have data right now. Let me grab some logs and get back.
For Lock combination, it can't be too small neither can be too big. This is
tricky part. :)
{quote}Key to the success of this approach would depend on how smart the
dynamic read/write allocation mechanism works. This may be less critical if the
workload pattern is easily predictable or slowly changing, or if you want to
enforce a certain ratio or priority between reads and writes. In environments
where the workload is highly varied, there might be difficulty utilizing this
in its fullest extent.
{quote}
Yes. I can share my version of the dequeue strategy later by uploading some
code as well as some benchmark. It's very like FairCallQueue strategy a weight
based rpc scheduler. [~hexiaoqiao]
One of our cluster is mostly with 10:1 read to write ratio traffic. And this
approach will indeed not bring much benefit for cases where there are too many
read or there are too many write. It's designed for moderate case.
> Improve NameNode RPC throughput with ReadWriteRpcCallQueue
> ---
>
> Key: HDFS-15553
> URL: https://issues.apache.org/jira/browse/HDFS-15553
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Wang, Xinglong
>Priority: Major
>
> *Current*
> In our production cluster, a typical traffic model is read to write raito is
> 10:1 and sometimes the ratios goes to 30:1.
> NameNode is using ReEntrantReadWriteLock under the hood of FSNamesystemLock.
> Read lock is shared lock while write lock is exclusive lock.
> Read RPC and Write RPC comes randomly to namenode. This makes read and write
> mixed up. And then only a small fraction of read can really share their read
> lock.
> Currently we have default callqueue and faircallqueue. And we can
> refreshCallQueue on the fly. This opens room to design new call queue.
> *Idea*
> If we reorder the rpc call in callqueue to group read rpc together and write
> rpc together, we will have sort of control to let a batch of read rpc come to
> handlers together and possibly share the same read lock. Thus we can reduce
> Fragments of read locks.
> This will only improve the chance to share the read lock among the batch of
> read rpc due to there are some namenode internal write lock is out of call
> queue.
> Under ReEntrantReadWriteLock, there is a queue to manage threads asking for
> locks. We can give an example.
> R: stands for read rpc
> W: stands for write rpc
> e.g
> WWWWWWWW
> In this case, we need 16 lock timeslice.
> optimized
>
> In this case, we only need 9 lock timeslice.
> *Correctness*
> Since the execution order of any 2 concurrent or queued rpc in namenode is
> not guaranteed. We can reorder the rpc in callqueue into read group and write
> group. And then dequeue from these 2 queues by a designed strategy. let's say
> dequeue 100 read and then dequeue 5 write rpc and then dequeue read again and
> then write again.
> Since FairCallQueue also does rpc call reorder in callqueue, for this part I
> think they share the same logic to guarantee rpc result correctness.
> *Performance*
> In test environment, we can see a 15% - 20% NameNode RPC throughput
> improvement comparing with default callqueue.
> Test traffic is 30 read:3 write :1 list using NNLoadGeneratorMR
> This performance is not a surprise. Due to some write rpc is not managed in
> callqueue. We can't do reorder to them by reording calls in callqueue.
> But still we can do a fully read write reorder if we redesign
> ReEntrantReadWriteLock to achieve this. This will be further step after this.
--
This message was sent by Atlassian Jira
[jira] [Commented] (HDFS-15553) Improve NameNode RPC throughput with ReadWriteRpcCallQueue
[ https://issues.apache.org/jira/browse/HDFS-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188442#comment-17188442 ] Wang, Xinglong commented on HDFS-15553: --- Thank you for the commet [~hexiaoqiao] I think even in the current code base, we can't make sure Client B will always see result of client A. I will give some counter cases against the scenario in your case. 1 FairCallQueue case, if client A is downgraded to a low priority queue, then this rpc may be executed after client B rpc. 2 Unfair lock mode, latter rpc may be executed before former rpc due to unfair mechanism 3 Due to network latency, if T1 and T2 means client side timestamp, then the sequence of client A rpc and client B rpc is not guaranteed in server side. 4 Let's say client A and client B is in the same jvm. Usually client A will wait until it got response from NN and then it will perform other actions like notify others to do things. There needs a synchronization between client A and client B if there is dependency. If user just spawn client A first and client B later, and there is no synchronization between client A and client B. The result can be a lot of cases. We can't say for sure it will have a guaranteed result. 5 If client A and client B is in different jvm, I think it's even harder to guarantee result only based on rpc sent out time. For the dequeue strategy, currently I implemented a weight based dynamic dequeue strategy to dynamically decide how many rpc should be dequeued from read call queue and write callqueue based on current readqueue length and write queue length to make sure we can consume both queue to prevent one queue starving. > Improve NameNode RPC throughput with ReadWriteRpcCallQueue > --- > > Key: HDFS-15553 > URL: https://issues.apache.org/jira/browse/HDFS-15553 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Wang, Xinglong >Priority: Major > > *Current* > In our production cluster, a typical traffic model is read to write raito is > 10:1 and sometimes the ratios goes to 30:1. > NameNode is using ReEntrantReadWriteLock under the hood of FSNamesystemLock. > Read lock is shared lock while write lock is exclusive lock. > Read RPC and Write RPC comes randomly to namenode. This makes read and write > mixed up. And then only a small fraction of read can really share their read > lock. > Currently we have default callqueue and faircallqueue. And we can > refreshCallQueue on the fly. This opens room to design new call queue. > *Idea* > If we reorder the rpc call in callqueue to group read rpc together and write > rpc together, we will have sort of control to let a batch of read rpc come to > handlers together and possibly share the same read lock. Thus we can reduce > Fragments of read locks. > This will only improve the chance to share the read lock among the batch of > read rpc due to there are some namenode internal write lock is out of call > queue. > Under ReEntrantReadWriteLock, there is a queue to manage threads asking for > locks. We can give an example. > R: stands for read rpc > W: stands for write rpc > e.g > WWWWWWWW > In this case, we need 16 lock timeslice. > optimized > > In this case, we only need 9 lock timeslice. > *Correctness* > Since the execution order of any 2 concurrent or queued rpc in namenode is > not guaranteed. We can reorder the rpc in callqueue into read group and write > group. And then dequeue from these 2 queues by a designed strategy. let's say > dequeue 100 read and then dequeue 5 write rpc and then dequeue read again and > then write again. > Since FairCallQueue also does rpc call reorder in callqueue, for this part I > think they share the same logic to guarantee rpc result correctness. > *Performance* > In test environment, we can see a 15% - 20% NameNode RPC throughput > improvement comparing with default callqueue. > Test traffic is 30 read:3 write :1 list using NNLoadGeneratorMR > This performance is not a surprise. Due to some write rpc is not managed in > callqueue. We can't do reorder to them by reording calls in callqueue. > But still we can do a fully read write reorder if we redesign > ReEntrantReadWriteLock to achieve this. This will be further step after this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902599#comment-16902599 ] Wang, Xinglong commented on HDFS-14195: --- HDFS-14195.010.patch to address checkstyle in https://builds.apache.org/job/PreCommit-HDFS-Build/27439/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.010.patch Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch, HDFS-14195.010.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902090#comment-16902090 ] Wang, Xinglong commented on HDFS-14195: --- Thank you [~jojochuang] and [~adam.antal] for the comments. I think the comments should be addressed now. Submitting new patch. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.009.patch Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch, > HDFS-14195.009.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899421#comment-16899421 ] Wang, Xinglong commented on HDFS-14195: --- [~jojochuang] [~adam.antal] The failed tests are not related. Could you help to merge the patch? > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899298#comment-16899298 ] Wang, Xinglong commented on HDFS-14195: --- Failed TCs are not related with this patch. And How to trigger the test again on the same patch. I tried cancel patch and submit patch without uploading new patch file and seems can't trigger CI. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.008.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch, HDFS-14195.008.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.007.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch, HDFS-14195.007.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16898744#comment-16898744 ] Wang, Xinglong commented on HDFS-14195: --- Sure, I will do the rebase. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Assignee: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[
https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16833899#comment-16833899
]
Wang, Xinglong commented on HDFS-14195:
---
Updated the following part.
* import style
{code:java}
java.io.*
{code}
* try ()
try (
ByteArrayInputStream input =
new ByteArrayInputStream(output.toByteArray());
> OIV: print out storage policy id in oiv Delimited output
>
>
> Key: HDFS-14195
> URL: https://issues.apache.org/jira/browse/HDFS-14195
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Wang, Xinglong
>Priority: Minor
> Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch,
> HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch,
> HDFS-14195.006.patch
>
>
> There is lacking of a method to get all folders and files with sort of
> specified storage policy via command line, like ALL_SSD type.
> By adding storage policy id to oiv output, it will help with oiv
> post-analysis to have a overview of all folders/files with specified storage
> policy and to apply internal regulation based on this information.
>
> Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print
> out xattr which including storage policy already.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.006.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch, > HDFS-14195.006.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829924#comment-16829924 ] Wang, Xinglong commented on HDFS-14195: --- [~adam.antal] Your comments have been addressed. Could you please take a look？Thanks. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.005.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch, HDFS-14195.005.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.004.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch, HDFS-14195.004.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.003.patch > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch, > HDFS-14195.003.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741131#comment-16741131 ] Wang, Xinglong commented on HDFS-14195: --- Unrelated test failure and all tests passed in a local run. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740991#comment-16740991 ] Wang, Xinglong commented on HDFS-14195: --- Hi [~adam.antal]. I am not able to assign the ticket to myself. Could you please help to assign it to me? * reordered storage policy position in getEntry() * fixed 2 related test failure * Added TestOfflineImageViewerForStoragePolicy.java to test this field. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.002.patch Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch, HDFS-14195.002.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Status: Open (was: Patch Available) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Description: There is lacking of a method to get all folders and files with sort of specified storage policy via command line, like ALL_SSD type. By adding storage policy id to oiv output, it will help with oiv post-analysis to have a overview of all folders/files with specified storage policy and to apply internal regulation based on this information. Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print out xattr which including storage policy already. was: There is lacking of a method to get all specified folders and files with sort of specified storage policy via command line, like ALL_SSD type. By adding storage policy id to oiv output, it will help with oiv post-analysis to have a overview of all folders/files with specified storage policy and to apply internal regulation based on this information. Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print out xattr which including storage policy already. > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch > > > There is lacking of a method to get all folders and files with sort of > specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
[ https://issues.apache.org/jira/browse/HDFS-14195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang, Xinglong updated HDFS-14195: -- Attachment: HDFS-14195.001.patch Status: Patch Available (was: Open) > OIV: print out storage policy id in oiv Delimited output > > > Key: HDFS-14195 > URL: https://issues.apache.org/jira/browse/HDFS-14195 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Reporter: Wang, Xinglong >Priority: Minor > Attachments: HDFS-14195.001.patch > > > There is lacking of a method to get all specified folders and files with sort > of specified storage policy via command line, like ALL_SSD type. > By adding storage policy id to oiv output, it will help with oiv > post-analysis to have a overview of all folders/files with specified storage > policy and to apply internal regulation based on this information. > > Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print > out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14195) OIV: print out storage policy id in oiv Delimited output
Wang, Xinglong created HDFS-14195: - Summary: OIV: print out storage policy id in oiv Delimited output Key: HDFS-14195 URL: https://issues.apache.org/jira/browse/HDFS-14195 Project: Hadoop HDFS Issue Type: Improvement Components: tools Reporter: Wang, Xinglong There is lacking of a method to get all specified folders and files with sort of specified storage policy via command line, like ALL_SSD type. By adding storage policy id to oiv output, it will help with oiv post-analysis to have a overview of all folders/files with specified storage policy and to apply internal regulation based on this information. Currently, for PBImageXmlWriter.java, in HDFS-9835 it added function to print out xattr which including storage policy already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-8439) Adding more slow action log in critical read path
[ https://issues.apache.org/jira/browse/HDFS-8439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16160603#comment-16160603 ] Wang, Xinglong commented on HDFS-8439: -- This is very interesting feature for hbase, do we have plan to patch this? > Adding more slow action log in critical read path > - > > Key: HDFS-8439 > URL: https://issues.apache.org/jira/browse/HDFS-8439 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HDFS-8439.001.patch, HDFS-8439.002.patch, > HDFS-8439.003.patch > > > To dig a HBase read spike issue, we'd better to add more slow pread/seek log > in read flow to get the abnormal datanodes. > Patch will be uploaded soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
