[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16145796#comment-16145796 ] Ajay Jadhav commented on HBASE-18448: - No problem [~lhofhansl] Yes, since the data sits on S3, one can point as many clusters to the same bucket without the need of data replication etc. Do provide your feedback on the parent jira. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144729#comment-16144729 ] Lars Hofhansl commented on HBASE-18448: --- Thanks [~ajayjadhav], this make a lot of sense. I can see more use cases for this - such as quickly standing up an HBase cluster over existing data, etc. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142944#comment-16142944 ] Ajay Jadhav commented on HBASE-18448: - [~lhofhansl]: Mostly it was meant for providing high availability in case one of the availability zone (AZ) goes down. So, one can have multiple clusters pointing to the same S3 location spread across AZs. We have an umbrella jira (https://issues.apache.org/jira/browse/HBASE-18477) where we have discussed this approach. Let me know if you have more questions. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142551#comment-16142551 ] Lars Hofhansl commented on HBASE-18448: --- This is interesting. [~ajayjadhav], if you don't mind can you share why wonted to a read-only replica cluster? Is it for performance? Also how do you prevent the read-only replica from ever writing? Lastly, the read-only cluster might decide to do compaction, or perhaps even splits on its own. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16142303#comment-16142303 ] Ajay Jadhav commented on HBASE-18448: - Thanks [~anoop.hbase] > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141342#comment-16141342 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-2.0 #392 (See [https://builds.apache.org/job/HBase-2.0/392/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (anoopsamjohn: rev 81ccef83becbd756c05eebace8ebd7eef82b9e85) * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141326#comment-16141326 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3593 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3593/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (anoopsamjohn: rev 98bb5c05e3cfcd90f2a3a25902f1eff8d8d37619) * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141069#comment-16141069 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-1.4 #884 (See [https://builds.apache.org/job/HBase-1.4/884/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (stack: rev 75ab445eb1f99f1f44382a71b5f681aeb395c2db) * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (edit) hbase-examples/pom.xml * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RefreshHFilesProtos.java * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141035#comment-16141035 ] Anoop Sam John commented on HBASE-18448: Pushed again to branch-2 and master. I reverted it seeing the compile issue but that was because not having the protos generated java file. We stopped pushing such files from branch-2 onwards. Need to do mvn build to generate them. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16141022#comment-16141022 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-1.5 #30 (See [https://builds.apache.org/job/HBase-1.5/30/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (stack: rev 7d605fe9c2ebd0e98a9e14067a6b44d117fb53eb) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RefreshHFilesProtos.java * (edit) hbase-examples/pom.xml * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140962#comment-16140962 ] Ajay Jadhav commented on HBASE-18448: - Thanks [~stack] and [~apurtell]. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140941#comment-16140941 ] stack commented on HBASE-18448: --- Pushed to 1.4 too. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.4.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140935#comment-16140935 ] Andrew Purtell commented on HBASE-18448: +1 > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.5.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140904#comment-16140904 ] stack commented on HBASE-18448: --- I tried the patch against branch-1. Test passed. Looks good. Pushed to branch-1. [~apurtell] you want this in branch-1.4? Its a nice, clear example in hbase-example package of how to do a CPEP. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 1.5.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140891#comment-16140891 ] stack commented on HBASE-18448: --- +1 on patch. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140727#comment-16140727 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3590 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3590/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (anoopsamjohn: rev 612c23556d4d2b6ef609ce7c487efa9bed35e145) * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java Revert "HBASE-18448 Added refresh HFiles coprocessor endpoint" (anoopsamjohn: rev 88356029f17430b77df15aaedbe17651397494a7) * (delete) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (delete) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (delete) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java * (delete) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140540#comment-16140540 ] Hudson commented on HBASE-18448: FAILURE: Integrated in Jenkins build HBase-2.0 #389 (See [https://builds.apache.org/job/HBase-2.0/389/]) HBASE-18448 Added refresh HFiles coprocessor endpoint (anoopsamjohn: rev e5a8f162a2ded86b42f40650d879fb730afb84dc) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (add) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (add) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (add) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java Revert "HBASE-18448 Added refresh HFiles coprocessor endpoint" (anoopsamjohn: rev adbe844ea5c0bbb9d11ccab222e25cfbfdae3208) * (delete) hbase-examples/src/test/java/org/apache/hadoop/hbase/coprocessor/example/TestRefreshHFilesEndpoint.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (delete) hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/RefreshHFilesClient.java * (delete) hbase-examples/src/main/protobuf/RefreshHFiles.proto * (delete) hbase-examples/src/main/java/org/apache/hadoop/hbase/coprocessor/example/RefreshHFilesEndpoint.java > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140462#comment-16140462 ] Ajay Jadhav commented on HBASE-18448: - Thanks everyone for the reviews. Can we push it to branch-1 too? > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 3.0.0, 2.0.0-alpha-3 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16140089#comment-16140089 ] Ted Yu commented on HBASE-18448: lgtm > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16139566#comment-16139566 ] Anoop Sam John commented on HBASE-18448: My bad eyes. Did not observe that u pass the table name in getOnlineRegions. +1 on patch. Ted, Ram u guys ok? > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[
https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16138667#comment-16138667
]
Ajay Jadhav commented on HBASE-18448:
-
[~anoop.hbase]
public int getNumHFilesForRS(final HRegionServer rs, final TableName
tableName,
final byte[] family) {
int numHFiles = 0;
for (Region region : rs.getOnlineRegions(tableName)) {
numHFiles += region.getStore(family).getStorefilesCount();
}
return numHFiles;
}
Above we are getting the online regions based on tableName. Not sure, what is
missing?
> EndPoint example for refreshing HFiles for stores
> --
>
> Key: HBASE-18448
> URL: https://issues.apache.org/jira/browse/HBASE-18448
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Affects Versions: 2.0.0, 1.3.1
>Reporter: Ajay Jadhav
>Assignee: Ajay Jadhav
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-18448.branch-1.001.patch,
> HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch,
> HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch,
> HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch,
> HBASE-18448.master.001.patch, HBASE-18448.master.002.patch
>
>
> In the case where multiple HBase clusters are sharing a common rootDir, even
> after flushing the data from
> one cluster doesn't mean that other clusters (replicas) will automatically
> pick the new HFile. Through this patch,
> we are exposing the refresh HFiles API which when issued from a replica will
> update the in-memory file handle list
> with the newly added file.
> This allows replicas to be consistent with the data written through the
> primary cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-18448) EndPoint example for refreshing HFiles for stores
[ https://issues.apache.org/jira/browse/HBASE-18448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16137918#comment-16137918 ] Anoop Sam John commented on HBASE-18448: bq.getNumHFilesForRS Here getting #files from all the online regions. There might be other table regions also in this RS. So we need check the table name for that region and then only get files# from Store. > EndPoint example for refreshing HFiles for stores > -- > > Key: HBASE-18448 > URL: https://issues.apache.org/jira/browse/HBASE-18448 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 2.0.0, 1.3.1 >Reporter: Ajay Jadhav >Assignee: Ajay Jadhav >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-18448.branch-1.001.patch, > HBASE-18448.branch-1.002.patch, HBASE-18448.branch-1.003.patch, > HBASE-18448.branch-1.004.patch, HBASE-18448.branch-1.005.patch, > HBASE-18448.branch-1.006.patch, HBASE-18448.branch-1.007.patch, > HBASE-18448.master.001.patch, HBASE-18448.master.002.patch > > > In the case where multiple HBase clusters are sharing a common rootDir, even > after flushing the data from > one cluster doesn't mean that other clusters (replicas) will automatically > pick the new HFile. Through this patch, > we are exposing the refresh HFiles API which when issued from a replica will > update the in-memory file handle list > with the newly added file. > This allows replicas to be consistent with the data written through the > primary cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
