[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16389003#comment-16389003 ] Jean-Marc Spaggiari commented on HBASE-20101: - I built a small improvement which in some usecases reduces the compaction IOs by 80%... I will for sure share that very soon, when cleaned a bit. I don't think it will ever make it into HBase. It needs HDFS admin rights to be able to move the blocks. And HBase doesn't have those right. I will provide more details soon. Tested in on 3 clusters right now. One 8 nodes dev cluster, one 20 nodes test cluster and one 60 nodes production cluster. Working well. More to come. > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Assignee: huaxiang sun >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16388239#comment-16388239 ] Dave Latham commented on HBASE-20101: - Can you share more about moving blocks to restore locality? Now that's something that would be awesome to see inside HBase, so that it doesn't require a compaction. > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Assignee: huaxiang sun >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383851#comment-16383851 ] Jean-Marc Spaggiari commented on HBASE-20101: - awesome! Thanks [~huaxiang]! > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Assignee: huaxiang sun >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383845#comment-16383845 ] huaxiang sun commented on HBASE-20101: -- talked with [~jmspaggi], I will take a look, thanks. > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Assignee: huaxiang sun >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381294#comment-16381294 ] Jean-Marc Spaggiari commented on HBASE-20101: - Kind of! There is 2 things. One Chore that can check, eventually, on a specified frequence. One API call that can trigger a reload for a specific region/CF... I need this 2nd option because I manually move some blocks to restore locality. I don't want to force HBase to re-check everything. Since I know which file the blocks are belonging too, I just need to say to HBase than this specific Region/CF changed... > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20101) HBase should provide a way to re-validate locality
[ https://issues.apache.org/jira/browse/HBASE-20101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379487#comment-16379487 ] churro morales commented on HBASE-20101: were you thinking of doing something like the StorefileRefresherChore but for primary regions? > HBase should provide a way to re-validate locality > -- > > Key: HBASE-20101 > URL: https://issues.apache.org/jira/browse/HBASE-20101 > Project: HBase > Issue Type: New Feature >Reporter: Jean-Marc Spaggiari >Priority: Major > > HDFS blocks can move for many reasons. HDFS balancing, lost of a disk, of a > node, etc. However, today, locality seems to be calculated when the files are > opened for the first time. Even disabling and re-enabling the regions doesn't > trigger a re-calculation of the locality. > We should provide a way to let the user ask for this number to be > re-calculated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
