[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346935#comment-16346935 ] Ashish Singhi commented on HBASE-13153: --- These issues as you said are out of date and also they are not related to bulk load replication feature. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi >Priority: Major > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16345294#comment-16345294 ] Anoop Sam John commented on HBASE-13153: bq.which one's you are pointing at ? Recently deleted data reappearing Bulk-loaded data cannot be overwritten by another bulk load > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi >Priority: Major > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344518#comment-16344518 ] Ashish Singhi commented on HBASE-13153: --- Hi [~anoop.hbase], I will try to post a patch for document update coming weekend. On week days I am little busy with my paid job. bq. Many bugs were resolved in bulk load and all such fix should be there in 1.3+ versions. AFAIK all those bugs are fixed in 1.3+ versions. I usually keep a eye on it. bq. The mentioned potential issues also seems out of date. which one's you are pointing at ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi >Priority: Major > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16344508#comment-16344508 ] Anoop Sam John commented on HBASE-13153: This feature as such is not described in book? If so, we have to add. Ya we have to update the section with removing the replication related limitation. Many bugs were resolved in bulk load and all such fix should be there in 1.3+ versions. The mentioned potential issues also seems out of date. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi >Priority: Major > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290715#comment-16290715 ] Akshay Gehi commented on HBASE-13153: - Understood. You are saying that the documentation should be updated since the limitation no longer applies to Bulk load but this section should not be removed. If someone uses an API such as Put.setDurability(SKIP_WAL), the limitation would still apply. My second question was related to the bulk load feature. Are there any other known problems using Bulk load feature on a regular basis on a production system. I have seen a lot of problems in various articles such as the following link but those articles seem to be dated and those limitations don't seem to apply any more. http://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/ Some snippets from the (perhaps outdated) article: *Potential Issues* * Recently deleted data reappearing * Bulk-loaded data cannot be overwritten by another bulk load We intend to use bulk load on a production system on a daily basis since we expect to receive millions of insert/updates a day. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290641#comment-16290641 ] Anoop Sam John commented on HBASE-13153: I mean the #2 point is there already in the ref guide as a limitation. That is a limitation for the replication feature and with normal write ops. That still applies as is. No way to overcome that yet. bq.Is bulk load meant for production use or could it cause other severe problems? U mean bulk load operation or the replication of bulk loaded files operation? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290464#comment-16290464 ] Akshay Gehi commented on HBASE-13153: - Thanks [~anoop.hbase] Regarding your second point - "The other limitation abt the data written with Durability = SKIP_WAL, still applies." Are you specifically talking about another limitation mentioned in the reference guide or do you mean that it is bound to have other limitations? Is bulk load meant for production use or could it cause other severe problems? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290416#comment-16290416 ] Anoop Sam John commented on HBASE-13153: Thanks for bringing in the issue. Ya the ref guide to be updated. 2 things to keep in mind 1. This feature allows the bulk loaded files to be replicated. But by default off. Need configs so as to enable to bulk loaded files replication 2. The other limitation abt the data written with Durability = SKIP_WAL, still applies. cc [~ashish singhi] > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290411#comment-16290411
]
Akshay Gehi commented on HBASE-13153:
-
HBase document specifies the following limitation:
{noformat}
- Bulk Load Limitations
As bulk loading bypasses the write path, the WAL doesn’t get written to as part
of the process. Replication works by reading the WAL files so it won’t see the
bulk loaded data – and the same goes for the edits that use
Put.setDurability(SKIP_WAL). One way to handle that is to ship the raw files or
the HFiles to the other cluster and do the other processing there.
{noformat}
Does this limitation still apply or should this Reference guide be updated?
HBASE-14939 talks about document update but I cannot be certain if it relates
to this problem.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v20.patch,
> HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch,
> HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568251#comment-15568251 ] Ashish Singhi commented on HBASE-13153: --- [~tedyu], the above is handled in the code. Take a look [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java#L372-L379]. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565504#comment-15565504 ] Ted Yu commented on HBASE-13153: bq. thinking on the HFiles that are being replicated but are moved to archive due to compaction. Here again we need to retry the operation like in case of WAL as per discussion [~ashish singhi]: Can you point me to the code where retry is done ? Or the above is not applicable to the current design ? Thanks > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050561#comment-15050561 ] Hudson commented on HBASE-13153: FAILURE: Integrated in HBase-1.3 #428 (See [https://builds.apache.org/job/HBase-1.3/428/]) HBASE-13153 Bulk Loaded HFile Replication (Ashish Singhi) (ramkrishna: rev 65117d3d04349f2ebb445a34f761fa83ce8e6aa3) * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SourceFSConfigurationProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeers.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSourceFSConfigurationProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpToolWithBulkLoadedData.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-protocol/src/main/protobuf/Admin.proto * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DefaultSourceFSConfigurationProvider.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java *
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050557#comment-15050557 ] Ashish Singhi commented on HBASE-13153: --- Thanks a lot to everyone who involved in this jira starting from the design review till the commit phase. bq. Ensure that the documentation talks about the proper config knobs and other areas that a user need to look out for while enabling this feature. Surely will do as part of HBASE-14939, once I address HBASE-14937 and HBASE-14938 as they may likely introduce 2/3 more configuration properties. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, > HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch, > HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050576#comment-15050576
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776746/HBASE-13153-v21.patch
against master branch at commit 9647fee3f0f196d064879afd41b9eff51d5aa036.
ATTACHMENT ID: 12776746
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16826//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16826//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16826//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16826//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v20.patch,
> HBASE-13153-branch-1-v21.patch, HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v19.patch,
> HBASE-13153-v2.patch, HBASE-13153-v20.patch, HBASE-13153-v21.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050920#comment-15050920 ] Hudson commented on HBASE-13153: FAILURE: Integrated in HBase-Trunk_matrix #544 (See [https://builds.apache.org/job/HBase-Trunk_matrix/544/]) HBASE-13153 Bulk Loaded HFile Replication (Ashish Singhi) (ramkrishna: rev 26ac60b03f80c9215103a02db783341e67037753) * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSourceFSConfigurationProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HBaseInterClusterReplicationEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DefaultSourceFSConfigurationProvider.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpToolWithBulkLoadedData.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SourceFSConfigurationProvider.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateZKImpl.java * hbase-protocol/src/main/protobuf/Admin.proto * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeers.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java *
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15051064#comment-15051064 ] Hudson commented on HBASE-13153: SUCCESS: Integrated in HBase-1.3-IT #365 (See [https://builds.apache.org/job/HBase-1.3-IT/365/]) HBASE-13153 Bulk Loaded HFile Replication (Ashish Singhi) (ramkrishna: rev 65117d3d04349f2ebb445a34f761fa83ce8e6aa3) * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSink.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateZKImpl.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeers.java * hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SourceFSConfigurationProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpToolWithBulkLoadedData.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ReplicationSinkService.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java * hbase-protocol/src/main/protobuf/Admin.proto * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSmallTests.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java * hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStateBasic.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/ScopeWALEntryFilter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestSourceFSConfigurationProvider.java * hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMasterReplication.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClient.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationPeersZKImpl.java * hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSinkSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/SecureBulkLoadEndpoint.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueues.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/ReplicationSourceDummy.java * hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java * hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesClientZKImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DefaultSourceFSConfigurationProvider.java * hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/Replication.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSyncUpTool.java *
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048876#comment-15048876
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776537/HBASE-13153-v19.patch
against master branch at commit 7bfbb6a3c9af4b0e2853b5ea2580a05bb471211b.
ATTACHMENT ID: 12776537
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
new checkstyle errors. Check build console for list of new errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16810//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16810//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16810//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16810//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch,
> HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch,
> HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch,
> HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch,
> HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048898#comment-15048898 ] Ted Yu commented on HBASE-13153: Please address checkstyle warnings. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load > Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049413#comment-15049413
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12776600/HBASE-13153-v20.patch
against master branch at commit 0e147a9d6e53e71ad2e57f512b4d3e1eeeac0b78.
ATTACHMENT ID: 12776600
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not generate new
checkstyle errors.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this
patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 zombies{color}. No zombie tests found running at the end of
the build.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16811//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16811//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16811//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16811//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v20.patch, HBASE-13153-v1.patch,
> HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch,
> HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch,
> HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch,
> HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v20.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049414#comment-15049414
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}.
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16812//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16812//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16812//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16812//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v20.patch, HBASE-13153-v1.patch,
> HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch,
> HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch,
> HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch,
> HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v20.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050237#comment-15050237 ] ramkrishna.s.vasudevan commented on HBASE-13153: Just a need a rebased patch . Ashish will give a new patch which we can commit. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v20.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v20.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049073#comment-15049073 ] Anoop Sam John commented on HBASE-13153: +1 > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v19.patch, HBASE-13153-v2.patch, HBASE-13153-v20.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15044714#comment-15044714 ] Ashish Singhi commented on HBASE-13153: --- Thanks [~jerryhe] for the comment. bq. I think we should add notes/best practice/warning in the release notes These we have mentioned in the design doc, do you want me to copy paste the same in release notes ? We are also planning to update the replication section in HBase book and note down all these there too. bq. And recommendations like the rpc timeout values need to be increased. These we are planning to make it adaptive for replication. I have raised HBASE-14937 for the same and plan to implement. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15046222#comment-15046222 ] Ted Yu commented on HBASE-13153: Any more review comment ? If not, planning to integrate. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042321#comment-15042321 ] Ted Yu commented on HBASE-13153: [~anoop.hbase] [~jerryhe]: Any more comment(s) ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15042453#comment-15042453 ] Jerry He commented on HBASE-13153: -- I have a use case that this feature would be quite useful. We have a SQL on Hadoop/HBase. When inserting into HBase, we try to be smart and optimize using bulk load some times. For example, when doing 'INSERT INTO my-hbase-table SELECT col1 from table1', we will try to see if the cardinalities are big (say > 2). If yes, we will generate hfile to bulk load, not running table puts. The problem is that replication will not kick in for this new data. For across cluster bulk load, people would probably use an external tool (e.g distCp) to move the MR generated hfiles to the target cluster. But in this case, it would be difficult to save and transport the hfiles for bulk load to the peer cluster since they are generated on-the-fly inside the SQL engine. So this is a good feature to have. Regarding the network latency and impact on HBase instances, I think we should add notes/best practice/warning in the release notes. Mention that potentially large files need to copied over the network by HBase handlers, and potential impact on the source and peer clusters. And recommendations like the rpc timeout values need to be increased. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15031386#comment-15031386 ] ramkrishna.s.vasudevan commented on HBASE-13153: Is there any more review comments. If not we can commit this. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15028645#comment-15028645 ] Ashish Singhi commented on HBASE-13153: --- Any more review comments ? I will be on vacation from tomorrow for next 10 days, so pardon me for my absence (If possible will try to reply through mobile). I have attached branch-1 patch also hoping that we can commit this there also. As by default it is disabled. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch, > HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch, > HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch, > HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15027470#comment-15027470
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12774378/HBASE-13153-branch-1-v18.patch
against branch-1 branch at commit 90bdb0dc7412f3ae8fc1e14ce033bf115c6332f3.
ATTACHMENT ID: 12774378
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16663//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16663//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16663//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16663//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-13153-branch-1-v18.patch, HBASE-13153-v1.patch,
> HBASE-13153-v10.patch, HBASE-13153-v11.patch, HBASE-13153-v12.patch,
> HBASE-13153-v13.patch, HBASE-13153-v14.patch, HBASE-13153-v15.patch,
> HBASE-13153-v16.patch, HBASE-13153-v17.patch, HBASE-13153-v18.patch,
> HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch,
> HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch,
> HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021743#comment-15021743 ] Ashish Singhi commented on HBASE-13153: --- [~jerryhe], thanks for the comments. bq. Another region server RPC handler --> holds region write lock --> transfers files to be bulk loaded into the region from remote cluster synchronously Not remote cluster, it will be local at this point as all the files are copied first from source to peer cluster and then only bulk load is initiated which will just rename these files. bq. Multiple handlers on the peer cluster can potentially be blocked Yes agress, this point was earlier raised by [~devaraj] also I have noted it down and have plan to may be add another QoS for bulk load as part of another jira as it will also help in normal bulk load case. bq. Now that the peer cluster 'server id' needs to read files directly from source cluster hbase.root directory. In a secure cluster, I recall that the hbase.root has been changed to be only accessible by the current 'server id'. Now they need to match I did not get what you mean. But we have done internal testing for this by providing peer cluster user read permission on the source cluster FS as mentioned in the design doc. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023827#comment-15023827 ] Ashish Singhi commented on HBASE-13153: --- bq. This HBASE-13780 restricts the access to the hbase.root. Would it impact this patch? We have tested in secure mode but with same user in active and peer cluster, so we were not required to give any extra permissions to the user in peer cluster to access active cluster FS. But if users are different then the operator has to provide read permission to the peer cluster user on the source cluster file system and we have mentioned the same in the document also in section 3 & 6. So with this things assured it will not impact this patch. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023516#comment-15023516 ] Jerry He commented on HBASE-13153: -- bq. Not remote cluster, it will be local at this point as all the files are copied first from source to peer cluster Okay bq. I did not get what you mean. Let me find the old JIRA. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023530#comment-15023530 ] Jerry He commented on HBASE-13153: -- This HBASE-13780 restricts the access to the hbase.root. Would it impact this patch? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021382#comment-15021382 ] Jerry He commented on HBASE-13153: -- Another question related to secure cluster. Now that the peer cluster 'server id' needs to read files directly from source cluster hbase.root directory. In a secure cluster, I recall that the hbase.root has been changed to be only accessible by the current 'server id'. Now they need to match. Will this work our ok? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021370#comment-15021370 ] Jerry He commented on HBASE-13153: -- A general comment. Disclaimer: I have not closely read thru the code, but I have read the doc and follow thru the comments roughly. On the source cluster: The source replication handler --> sends WALs entries to the peer, including bulkload entries, synchronously blocking for response. On the peer cluster: The peer region server RPC handler --> sees bulkload WAL entry --> invokes bulkload client RPC to another region server --> synchronously blocking Another region server RPC handler --> holds region write lock --> transfers files to be bulk loaded into the region from remote cluster synchronously Multiple handlers on the peer cluster can potentially be blocked. Multiple regions can be blocked from reading as well. In the normal replication case, the granularity is a few WAL entries. The granularity of failure is at the entire file level with bulk load. This is probably going to be ok in low network latency. But what happens when the network latency is less ideal? In an active-active case? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021068#comment-15021068
]
Ashish Singhi commented on HBASE-13153:
---
Thanks [~appy] for the prompt response.
bq. Help review and submit?
By submit if you mean commit then sorry I can't, I don't have that special
power! :-)
Left over check style warnings are all related to either IndentationCheck or
ImportOrderCheck as mentioned in my previous comment which are in total 19,
which are not valid for HBase project and will be fixed by Appy as part of
HBASE-14859.
{noformat}
$ dev-support/checkstyle_report.py ~/Downloads/trunkCheckstyle.xml
~/Downloads/patchCheckstyle.xml
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
85 87
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
7 8
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java
3 4
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
232 234
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SourceFSConfigurationProvider.java
0 1
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicationCallable.java
0 4
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALReplicationCallable.java
0 1
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DefaultSourceFSConfigurationProvider.java
0 1
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java
0 5
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java
1 2
{noformat}
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020457#comment-15020457
]
Ashish Singhi commented on HBASE-13153:
---
Total check style warnings found in patch by checkstyle_report.py are:
{noformat}
$ dev-support/checkstyle_report.py trunkCheckstyle.xml patchCheckstyle.xml
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
85 87
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationQueuesZKImpl.java
7 11
hbase-client/src/main/java/org/apache/hadoop/hbase/replication/ReplicationStateZKBase.java
3 4
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
232 234
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALReplicationCallable.java
0 2
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicationCallable.java
0 4
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/DefaultSourceFSConfigurationProvider.java
0 1
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/SourceFSConfigurationProvider.java
0 2
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSink.java
2 3
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/master/ReplicationHFileCleaner.java
0 6
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/TableCfWALEntryFilter.java
1 2
{noformat}
but Hadoop QA something has something else to say!
Out of all those I have not fixed below the one's:
ZKUtil.java (2)
{noformat}
{noformat}
ReplicationQueuesZKImpl.java (1)
{noformat}
{noformat}
ReplicationStateZKBase.java (1)
{noformat}
{noformat}
RSRpcServices.java (2)
{noformat}
{noformat}
WALReplicationCallable.java (1)
{noformat}
{noformat}
HFileReplicationCallable.java (4)
{noformat}
{noformat}
DefaultSourceFSConfigurationProvider.java (1)
{noformat}
{noformat}
SourceFSConfigurationProvider.java (1)
{noformat}
{noformat}
ReplicationHFileCleaner.java (5)
{noformat}
{noformat}
TableCfWALEntryFilter.java (1)
{noformat}
{noformat}
These belongs to two categories,
1. IndentationCheck: The code formatter which we
[use|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
adds only two spaces for the next line which is continuation of previous line.
2. ImportOrderCheck: I am not sure what to fix here! I am using eclipse IDE
and it by default organizes the imports in that way as mentioned
[here|http://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.jdt.doc.user%2Freference%2Fpreferences%2Fjava%2Fcodestyle%2Fref-preferences-organize-imports.htm]
[~appy] can you throw some light, what can I do here ?
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020582#comment-15020582
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12773664/HBASE-13153-v18.patch
against master branch at commit 52edd83baf2e8bbf519c6c4927cbdd8dce9ae406.
ATTACHMENT ID: 12773664
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
18692 checkstyle errors (more than the master's current 18689 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16628//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16628//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16628//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16628//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15020709#comment-15020709 ] Appy commented on HBASE-13153: -- The difference is because hadoop QA compares total number of error across all files where as the tool checks at file level. Adding the numbers given by tool, there are 26 new errors. The patch probably fixed 16 errors in some other files, which is why hadoop qa shows only +10 increase. As you can see, looking at overall count like presubmit does isn't the best way, HBASE-14859 will fix it. As for the exact errors, it's my fault for not setting indentations for all options to 2. When i was setting it, I looked at couple of code examples, and seemed like 4 was standard. Same for imports. For now, updated hee patch in HBASE-14859 to only check for common things between idea and eclipse. Help review and submit? :-) [~ashish singhi] Sorry for the trouble. :-( > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v17.patch, HBASE-13153-v18.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15018903#comment-15018903
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12773558/HBASE-13153-v17.patch
against master branch at commit 86be690b0723e814a655ad0ae8a6577d7111c1f2.
ATTACHMENT ID: 12773558
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
18700 checkstyle errors (more than the master's current 18690 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16622//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16622//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16622//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16622//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v17.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008636#comment-15008636 ] Ted Yu commented on HBASE-13153: +1 from me as well > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15010390#comment-15010390 ] ramkrishna.s.vasudevan commented on HBASE-13153: Any more reviews here [~saint@gmail.com], [~lhofhansl], [~anoop.hbase], [~apurtell]? SF folks would be interested in this as it is related to replication. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006670#comment-15006670 ] Ashish Singhi commented on HBASE-13153: --- Thank you so much Ted and Ram for the reviews. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006942#comment-15006942 ] Ted Yu commented on HBASE-13153: See if the checkstyle warning was from latest patch. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006940#comment-15006940
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12772488/HBASE-13153-v15.patch
against master branch at commit dd5f454b03b9ccd788398fa52220c7690381eb6f.
ATTACHMENT ID: 12772488
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1727 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16533//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16533//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16533//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16533//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006973#comment-15006973 ] Ashish Singhi commented on HBASE-13153: --- bq. See if the checkstyle warning was from latest patch. Yes, it was due to one unused import. Fixed in v16. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007276#comment-15007276
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12772529/HBASE-13153-v16.patch
against master branch at commit f218e499ae12805cc35ed938c1f6d07ace048a25.
ATTACHMENT ID: 12772529
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16535//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16535//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16535//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16535//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch,
> HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch,
> HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch,
> HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008063#comment-15008063 ] ramkrishna.s.vasudevan commented on HBASE-13153: +1 on patch. Better to change the release notes to indicate the configs needed to make this feature run along with that new configs added for HA. Also add some details on the tests performed to verify this feature - just for others to know. Thanks for the work Ashish. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15008128#comment-15008128 ] Ashish Singhi commented on HBASE-13153: --- bq. Better to change the release notes to indicate the configs needed to make this feature run along with that new configs added for HA Done. bq. Also add some details on the tests performed to verify this feature - just for others to know I have asked [~sreenivasulureddy] to provide the details as he has done most of the testing and let him take due credits for this. Thanks Ram. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v15.patch, HBASE-13153-v16.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15006319#comment-15006319 ] ramkrishna.s.vasudevan commented on HBASE-13153: Completed the reviews on the JIRA. Most of the comments are minors and few questions. Should be good to commit IMHO once those nits are done. Any more reviews or feedback? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load > Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005809#comment-15005809
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12772399/HBASE-13153-v14.patch
against master branch at commit bb6581345fd9ecac964e19cea2293477162801ca.
ATTACHMENT ID: 12772399
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16524//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16524//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16524//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16524//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v14.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load
> Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15005742#comment-15005742 ] Ashish Singhi commented on HBASE-13153: --- Attached patch(v14) which corrects the logic for bulk load entry filter in WALEntryFilter's. Please review. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v14.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication-v3.pdf, HBase Bulk Load > Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15003699#comment-15003699 ] Ashish Singhi commented on HBASE-13153: --- In our internal testing [~sreenivasulureddy] today found one more issue. The list of StoreDescriptor's in BulkLoadDescriptor is unmodifiable and in result the patch will fail work to with WALEntryFilters as expected. I will handle this in the patch and will try to add UT for it and also if there are any review comments to address by that time then will happily address them also. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002066#comment-15002066 ] Ashish Singhi commented on HBASE-13153: --- Attached patch(v13) addressing Ted's comments. Please review. If no further reviews I request please commit this, I have already spent lot of efforts in re-basing it! If any more issue I can fix them as part of sub tasks. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15002433#comment-15002433
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771967/HBASE-13153-v13.patch
against master branch at commit 1f62a487284b57fca505bc1b3d04c1f86b2e7d76.
ATTACHMENT ID: 12771967
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16499//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16499//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16499//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16499//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v13.patch,
> HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch,
> HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch,
> HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14995324#comment-14995324
]
Ashish Singhi commented on HBASE-13153:
---
Thanks for the comments [~tedyu]
bq. 1. secure bulk loading (without replication)
There is no change in flow of secure bulk load without replication. We just
added a check if the input hfile path and staging dir hfile path are same avoid
FS rename. As in replication the staging dir is managed by it and all the
hfiles are already copied in it so we save this FS rename call.
{quote}
2. bulk loaded hfiles replicated across secure clusters
3. 2. bulk loaded hfiles replicated across secure HA clusters
{quote}
As for secure clusters we need to configure kerberos settings required for
below operations across two secure clusters
1. HDFS distcp and
2. Existing HBase replication(mutations).
These settings users taking bulk load data backup may be already configuring it
in their secure clusters env. Nothing additional for this feature.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994071#comment-14994071 ] Ashish Singhi commented on HBASE-13153: --- Attached updated design doc and patch. Please review. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994388#comment-14994388
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12771067/HBASE-13153-v12.patch
against master branch at commit bfa36891901b96b95d82f5307642c35fd2b9f534.
ATTACHMENT ID: 12771067
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 42 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1728 checkstyle errors (more than the master's current 1726 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { "Entry", "ReplicationClusterId",
"SourceBaseNamespaceDirPath", "SourceHFileArchiveDirPath", });
+ private void validateFamiliesInHFiles(Table table, Deque
queue) throws IOException {
++ ". Hence will load all the xml files present in its
configured replication cluster"
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16433//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16433//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16433//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16433//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14994373#comment-14994373 ] Ted Yu commented on HBASE-13153: Left a few comments on reviewboard. Can you illustrate the cluster tests you have performed ? 1. secure bulk loading (without replication) 2. bulk loaded hfiles replicated across secure clusters 3. 2. bulk loaded hfiles replicated across secure HA clusters Please try to add more unit tests for the 3 points you mentioned on Oct 28th > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v12.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication-v3.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14982969#comment-14982969 ] Ashish Singhi commented on HBASE-13153: --- Thanks Ted, after more testing on this internally will post a patch here. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14977904#comment-14977904
]
Ashish Singhi commented on HBASE-13153:
---
Following problems were faced in our internal testing,
*Problem 1:*
Replication for bulk loaded data was not working HDFS HA cluster.
*Solution:*
Sink cluster will need the source cluster hdfs client configuration to be able
to perform its operation. So we have taken the below approach,
a) Configure unique replication cluster ID for
each Source cluster. Example: dc1, dc2
b) Each source cluster needs to place the HDFS client configurations in peer
cluster, in of source cluster under
directory.
c) During replication, source cluster will send its unique replication id to
peer cluster in the request. Peer cluster will identify this source cluster
hdfs client configurations based on this id and configured replication
configuration directory and will perform replication.
!https://imageshack.com/i/p3Ik2xanp!
This approach requires two more configurations,
i. At source cluster, hbase.replication.cluster.id (mandatory when
hbase.replication.bulkload.enabled is set to true)
ii. At sink cluster, hbase.replication.conf.dir (default hbase configuration
directory)
*Problem 2:*
If source and sink cluster are sharing same HDFS then the hfile from source
cluster is moved to sink cluster, instead of copy as per the logic in
{{SecureBulkLoadEndPoint}}
*Solution:*
To avoid this we will copy the hfiles from source cluster into sink cluster
staging directory parallely in the {{HFileReplicationCallable}} class itself
before sending the request to {{LoadIncrementalHFiles}} and will ensure to
avoid copy/rename again in {{SecureBulkLoadEndPoint}} to staging directory.
This will also solve the problem "the bulk load to peer cluster needs a split,
we will do split by reading each cell from remote src cluster HFile. This will
be a costly op" found during a offline discussion with [~anoopsamjohn].
*Problem 3:*
If add_peer is done before enabling replication for bulk load data, then to
that peer, bulk loaded data was not getting replicated.
*Solution:*
As part RS startup, we init {{ReplicationSourceManager}} there we will check if
all the peers in the cluster have their node in hfile-refs znode.
Please review the above solutions, if it seems ok then I will post a new patch
based on above solutions and update the doc.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14978132#comment-14978132 ] Ted Yu commented on HBASE-13153: Sounds good to me > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf, HDFS_HA_Solution.PNG > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14964824#comment-14964824 ] Ashish Singhi commented on HBASE-13153: --- During a offline discussion with Anoop on this, we found that when the source hfiles are in a different FS and if the hfile requires a split then LoadIncrementalHFiles will open a remote reader to source hfile, scan the file and append the data to each of the file split. Since we anyway copy the hfiles to the local FS if the source hfiles are in remote FS later, so we thought we can optimize this by copying the hfiles to a temp directory in local FS if source hfiles are in a different FS first and then do a local read and write. This is related to LoadIncrementalHFiles, when ever the source hfiles are in a different FS so I will handle this as part of another jira which will be subtask of this. So in this jira there will be no change in the patch or doc related to this. Any further review comments on the patch will be really appreciated. Thanks Ted, Ram, Anoop and Matteo for the reviews till now. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963077#comment-14963077
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12767313/HBASE-13153-v11.patch
against master branch at commit 8e6316a80cf96f4d4cd6bd10f4c647ebf45c7e02.
ATTACHMENT ID: 12767313
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16091//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16091//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16091//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16091//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963386#comment-14963386 ] Anoop Sam John commented on HBASE-13153: Checking the op flow : Considering the scenario where peer cluster is not secure (no secure EP) and the bulk load to peer cluster needs a split, we will do split by reading each cell from remote src cluster HFile. This will be a costly op. Suggestion will be like when we have to do bulk load to peer cluster, make sure the big file is copied to dest peer cluster first and then do the split and read of the file. Had a call with Ashish and discussed this. He will come with change in the flow and doc > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14963626#comment-14963626 ] ramkrishna.s.vasudevan commented on HBASE-13153: Is this slow down related to the point mentioned in the Performance Scenarios section? Or something else? Because we are going to do copy of the files from src to remote right and then do the incremental bulk load? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch, > HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch, > HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960491#comment-14960491 ] Ashish Singhi commented on HBASE-13153: --- [~mbertozzi], [~anoop.hbase], do you have any more review comments ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch, > HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch, > HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch, > HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk > Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk > Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960496#comment-14960496
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12767009/HBASE-13153-v10.patch
against master branch at commit d5ed46bc9f9285f75d2d906ec9c120cb408827df.
ATTACHMENT ID: 12767009
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16060//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16060//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16060//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16060//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v2.patch, HBASE-13153-v3.patch, HBASE-13153-v4.patch,
> HBASE-13153-v5.patch, HBASE-13153-v6.patch, HBASE-13153-v7.patch,
> HBASE-13153-v8.patch, HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14960770#comment-14960770
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12767052/HBASE-13153-v11.patch
against master branch at commit 30cf4e3761e95f3cceaf8c1aa154695e18198cd6.
ATTACHMENT ID: 12767052
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16064//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16064//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16064//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16064//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v10.patch,
> HBASE-13153-v11.patch, HBASE-13153-v2.patch, HBASE-13153-v3.patch,
> HBASE-13153-v4.patch, HBASE-13153-v5.patch, HBASE-13153-v6.patch,
> HBASE-13153-v7.patch, HBASE-13153-v8.patch, HBASE-13153-v9.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14958899#comment-14958899 ] Ashish Singhi commented on HBASE-13153: --- Added a test case for verifying the offline replication sync up tool. All the changes from my side are done. Other than addressing review comments if any. Please review. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch, > HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14959185#comment-14959185
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766785/HBASE-13153-v9.patch
against master branch at commit e7defd7d9a76f44e3089db3fe522fe400fe6dcd7.
ATTACHMENT ID: 12766785
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 41 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { "Entry",
"SourceBaseNamespaceDirPath", "SourceHFileArchiveDirPath", });
+loadAndValidateHFileReplication("HFileReplication_1", row, famName,
ht1Source, hfileRanges, 100);
+loadAndValidateHFileReplication("HFileReplication_1", row, famName,
ht2Source, hfileRanges, 200);
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16027//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16027//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16027//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16027//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153-v9.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14957182#comment-14957182
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766536/HBASE-13153-v8.patch
against master branch at commit 94bfe909aff9fd74cb1a5d0c3f9209a19704c6cf.
ATTACHMENT ID: 12766536
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16006//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16006//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16006//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16006//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16006//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153-v8.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956451#comment-14956451
]
Ashish Singhi commented on HBASE-13153:
---
Fixed the test case and check style warning.
javadoc was not mine.
May be it is complaining about,
{noformat}
[WARNING]
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/util/KeyLocker.java:83:
warning - @throw is an unknown tag.
{noformat}
Added a test case for hfile cyclic and multi slave replications.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14956750#comment-14956750
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766494/HBASE-13153-v7.patch
against master branch at commit 94bfe909aff9fd74cb1a5d0c3f9209a19704c6cf.
ATTACHMENT ID: 12766494
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1749 checkstyle errors (more than the master's current 1748 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16001//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16001//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16001//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16001//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16001//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153-v7.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955396#comment-14955396 ] Ashish Singhi commented on HBASE-13153: --- Rebased patch after HBASE-14501. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153-v6.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955752#comment-14955752
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766380/HBASE-13153-v6.patch
against master branch at commit 6323cd95442bf6c2617b7e8267e0c0b76e4c47eb.
ATTACHMENT ID: 12766380
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1754 checkstyle errors (more than the master's current 1753 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15990//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15990//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15990//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15990//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15990//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14955851#comment-14955851
]
Ted Yu commented on HBASE-13153:
Please fix the following test failure:
{code}
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager.testBulkLoadWALEdits(TestReplicationSourceManager.java:458)
{code}
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153-v6.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951863#comment-14951863 ] Ted Yu commented on HBASE-13153: +1 > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch, > HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication-v2.pdf, HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951861#comment-14951861
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765980/HBASE-13153-v5.patch
against master branch at commit 1b66ea71b69e8b24ba7ecb9ba6548d775777f417.
ATTACHMENT ID: 12765980
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15952//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15952//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15952//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15952//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153-v5.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14951730#comment-14951730
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765969/HBASE-13153-v4.patch
against master branch at commit 2e593a9d3801a42751244ab4478650a581437875.
ATTACHMENT ID: 12765969
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1764 checkstyle errors (more than the master's current 1762 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { "Entry",
"SourceBaseNamespaceDirPath", "SourceHFileArchiveDirPath", });
+ * {@link LoadIncrementalHFiles#bulkLoadPhase(Table, Connection,
ExecutorService, Deque, Multimap)}
+ HConstants.REPLICATION_BULKLOAD_ENABLE_KEY,
HConstants.REPLICATION_BULKLOAD_ENABLE_DEFAULT))) {
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15948//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15948//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15948//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15948//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153-v4.patch, HBASE-13153.patch, HBase Bulk
> Load Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk
> Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950646#comment-14950646 ] Ted Yu commented on HBASE-13153: Ashish: Mind fixing the javadoc warning in your next patch ? Thanks > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14950357#comment-14950357 ] Ashish Singhi commented on HBASE-13153: --- Thanks [~devaraj]. bq. One question - when the peer cluster does a bulkload, I am wondering if the network is slow, would it hold up the peer RegionServer handlers for a longer duration for bigger bulkloads, and thereby affect the throughput of the peer cluster significantly. A very valid point. I think this problem is not only related to replication but also in a normal bulk load where source and destination hdfs clusters are different. May be we can add a new QoS priority to handle bulk load ? and also can we do that as part of another jira ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch, > HBASE-13153-v3.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14949320#comment-14949320
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765631/HBASE-13153-v3.patch
against master branch at commit 7e30436e3fa84525b85b05b9e23cb01b2ada7c12.
ATTACHMENT ID: 12765631
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15918//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15918//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15918//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15918//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15918//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153-v3.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load
> Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14943990#comment-14943990
]
Hadoop QA commented on HBASE-13153:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12765026/HBASE-13153-v2.patch
against master branch at commit ceafa09d3cf6102d21c66745ca80e132021890c9.
ATTACHMENT ID: 12765026
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15875//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15875//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15875//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15875//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153-v2.patch,
> HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load
> Replication-v2.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939965#comment-14939965
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12764608/HBASE-13153-v1.patch
against master branch at commit 76463a36f5648f42cdcf64019a825c1e3f0c4fe1.
ATTACHMENT ID: 12764608
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15842//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15842//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15842//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15842//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153-v1.patch, HBASE-13153.patch, HBase Bulk Load
> Replication-v1-1.pdf, HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940160#comment-14940160 ] Ashish Singhi commented on HBASE-13153: --- bq. org.apache.hadoop.hbase.io.TestHeapSize Valid failure. Added a new field clusterId in HRegion class whose size needs to be added to FIXED_OVERHEAD. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14940269#comment-14940269 ] Devaraj Das commented on HBASE-13153: - Ashish, nice work. One question - when the peer cluster does a bulkload, I am wondering if the network is slow, would it hold up the peer RegionServer handlers for a longer duration for bigger bulkloads, and thereby affect the throughput of the peer cluster significantly. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153-v1.patch, HBASE-13153.patch, HBase Bulk Load > Replication-v1-1.pdf, HBase Bulk Load Replication-v2.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939372#comment-14939372 ] Ashish Singhi commented on HBASE-13153: --- Thanks for the review and comments. bq. Can you share your experience testing this feature on clusters ? Till now I have tested only on one node master slave cluster and all seems ok. Metrics, zk dump on shell and UI all looks ok as well. The main things pending for testing is, 1. Testing on a kerberos cluster 2. Replication offline sync up tool 3. Cyclic replication 4. Multi slave cluster Open points for this are same as wal replication. For example, HBASE-12769. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, > HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939378#comment-14939378
]
Ashish Singhi commented on HBASE-13153:
---
bq. Please fill in release notes
Will do in some time.
Adding well documented public methods or broadening the scope of a package
private class to a public in a {{@InterfaceAudience.Public}},
{{@InterfaceStability.Stable}} class should not cause any compatibility issues
IMO.
In my first patch I have done something like that to {{LoadIncrementalHFiles}}
class. [~mbertozzi], do you have any concerns with it ?
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf,
> HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938787#comment-14938787 ] Ted Yu commented on HBASE-13153: Thanks for the hard work. Can you share your experience testing this feature on clusters ? > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, > HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938872#comment-14938872 ] Ted Yu commented on HBASE-13153: Please fill in release notes > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf, > HBase Bulk Load Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938828#comment-14938828
]
Hadoop QA commented on HBASE-13153:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12764470/HBASE-13153.patch
against master branch at commit 24370547c500df0026a71944d8be88cd5b51b23e.
ATTACHMENT ID: 12764470
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 26 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1788 checkstyle errors (more than the master's current 1781 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ new java.lang.String[] { "Entry",
"SourceBaseNamespaceDirPath", "SourceHFileArchiveDirPath", });
+ * {@link LoadIncrementalHFiles#bulkLoadPhase(Table, Connection,
ExecutorService, Deque, Multimap)}
+long trx = writeMarker(wal, htd, hri, WALEdit.createBulkLoadEvent(hri,
desc), mvcc, clusterId, true);
+ new HLogKey(hri.getEncodedNameAsBytes(), hri.getTable(),
System.currentTimeMillis(), mvcc);
+ HConstants.REPLICATION_BULKLOAD_ENABLE_KEY,
HConstants.REPLICATION_BULKLOAD_ENABLE_DEFAULT))) {
+conf.getInt("hbase.loadincremental.threads.max",
Runtime.getRuntime().availableProcessors());
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15833//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15833//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15833//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15833//console
This message is automatically generated.
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf,
> HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[ https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14938738#comment-14938738 ] Ashish Singhi commented on HBASE-13153: --- I have uploaded the first version of the patch on the RB, https://reviews.apache.org/r/38903/ Will work on a integration test, once the patch addresses all the concerns if any and good to commit. Please review. Thanks. > Bulk Loaded HFile Replication > - > > Key: HBASE-13153 > URL: https://issues.apache.org/jira/browse/HBASE-13153 > Project: HBase > Issue Type: New Feature > Components: Replication >Reporter: sunhaitao >Assignee: Ashish Singhi > Fix For: 2.0.0 > > Attachments: HBase Bulk Load Replication-v1-1.pdf, HBase Bulk Load > Replication.pdf > > > Currently we plan to use HBase Replication feature to deal with disaster > tolerance scenario.But we encounter an issue that we will use bulkload very > frequently,because bulkload bypass write path, and will not generate WAL, so > the data will not be replicated to backup cluster. It's inappropriate to > bukload twice both on active cluster and backup cluster. So i advise do some > modification to bulkload feature to enable bukload to both active cluster and > backup cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13153) Bulk Loaded HFile Replication
[
https://issues.apache.org/jira/browse/HBASE-13153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14939017#comment-14939017
]
Ted Yu commented on HBASE-13153:
Test run was incomplete.
{code}
[INFO] Apache HBase - Common . SUCCESS [1:24.284s]
[INFO] Apache HBase - Procedure .. FAILURE [1:47.734s]
{code}
> Bulk Loaded HFile Replication
> -
>
> Key: HBASE-13153
> URL: https://issues.apache.org/jira/browse/HBASE-13153
> Project: HBase
> Issue Type: New Feature
> Components: Replication
>Reporter: sunhaitao
>Assignee: Ashish Singhi
> Fix For: 2.0.0
>
> Attachments: HBASE-13153.patch, HBase Bulk Load Replication-v1-1.pdf,
> HBase Bulk Load Replication.pdf
>
>
> Currently we plan to use HBase Replication feature to deal with disaster
> tolerance scenario.But we encounter an issue that we will use bulkload very
> frequently,because bulkload bypass write path, and will not generate WAL, so
> the data will not be replicated to backup cluster. It's inappropriate to
> bukload twice both on active cluster and backup cluster. So i advise do some
> modification to bulkload feature to enable bukload to both active cluster and
> backup cluster
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
