[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324038075 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: hbase-testing-util is used for running unit tests only by downstreamers? like just to bring up HBase minicluster by downstreamers and run their own tests on top of it? if no, then let me change this to `compile` Edit: I think better to include at `compile` given downstreames requirement This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
Apache-HBase commented on issue #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/619#issuecomment-531103122 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 19m 38s | Docker mode activated. | | :broken_heart: | patch | 0m 9s | https://github.com/apache/hbase/pull/619 does not apply to branch-1. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-619/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/619 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-619/1/console | | versions | git=1.9.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22929) MemStoreLAB ChunkCreator may memory leak
[ https://issues.apache.org/jira/browse/HBASE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-22929: --- Fix Version/s: (was: 2.1.6) (was: 2.2.1) 2.2.2 2.1.7 > MemStoreLAB ChunkCreator may memory leak > - > > Key: HBASE-22929 > URL: https://issues.apache.org/jira/browse/HBASE-22929 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.2 >Reporter: Yechao Chen >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: OOM_log.png, hbase-site.xml, hbase_heap_monitor.png, > hbase_rs_heap_dump_mat_1.png, > hbase_rs_heap_dump_mat_ChunkCreator_chunkIdMap.png, hbase_rs_mem_used.png > > > We use hbase 2.1.2 with memstorelab enable > RegionServer crashed case of oom > I dump the heap ,found the ChunkCreator may be memory leak > The heap is 32GB, > hbase.regionserver.global.memstore.size=0.4, > hbase.hregion.memstore.mslab.enabled=true > hbase.hregion.memstore.chunkpool.initialsize=0.5, > hbase.hregion.memstore.chunkpool.maxsize=1.0 > BucketCache with offheap -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324038075 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: hbase-testing-util is used for running unit tests only by downstreamers? like just to bring up HBase minicluster by downstreamers and run their own tests on top of it? if no, then let me change this to `compile` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (HBASE-22929) MemStoreLAB ChunkCreator may memory leak
[ https://issues.apache.org/jira/browse/HBASE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan resolved HBASE-22929. Fix Version/s: 2.1.6 2.2.1 2.3.0 3.0.0 Resolution: Fixed Thanks for all the reviews. Pushed to master, branch-2, branch-2.1 and branch-2.2. > MemStoreLAB ChunkCreator may memory leak > - > > Key: HBASE-22929 > URL: https://issues.apache.org/jira/browse/HBASE-22929 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.2 >Reporter: Yechao Chen >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.2.1, 2.1.6 > > Attachments: OOM_log.png, hbase-site.xml, hbase_heap_monitor.png, > hbase_rs_heap_dump_mat_1.png, > hbase_rs_heap_dump_mat_ChunkCreator_chunkIdMap.png, hbase_rs_mem_used.png > > > We use hbase 2.1.2 with memstorelab enable > RegionServer crashed case of oom > I dump the heap ,found the ChunkCreator may be memory leak > The heap is 32GB, > hbase.regionserver.global.memstore.size=0.4, > hbase.hregion.memstore.mslab.enabled=true > hbase.hregion.memstore.chunkpool.initialsize=0.5, > hbase.hregion.memstore.chunkpool.maxsize=1.0 > BucketCache with offheap -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (HBASE-22929) MemStoreLAB ChunkCreator may memory leak
[ https://issues.apache.org/jira/browse/HBASE-22929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-22929: -- Assignee: ramkrishna.s.vasudevan > MemStoreLAB ChunkCreator may memory leak > - > > Key: HBASE-22929 > URL: https://issues.apache.org/jira/browse/HBASE-22929 > Project: HBase > Issue Type: Bug >Affects Versions: 2.1.2 >Reporter: Yechao Chen >Assignee: ramkrishna.s.vasudevan >Priority: Major > Attachments: OOM_log.png, hbase-site.xml, hbase_heap_monitor.png, > hbase_rs_heap_dump_mat_1.png, > hbase_rs_heap_dump_mat_ChunkCreator_chunkIdMap.png, hbase_rs_mem_used.png > > > We use hbase 2.1.2 with memstorelab enable > RegionServer crashed case of oom > I dump the heap ,found the ChunkCreator may be memory leak > The heap is 32GB, > hbase.regionserver.global.memstore.size=0.4, > hbase.hregion.memstore.mslab.enabled=true > hbase.hregion.memstore.chunkpool.initialsize=0.5, > hbase.hregion.memstore.chunkpool.maxsize=1.0 > BucketCache with offheap -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324037433 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: If it's needed for Hadoop then it can't be at `provided` scope because it'll need to be present for a downstream user as well. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
Apache-HBase commented on issue #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/618#issuecomment-531099707 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 0s | Docker mode activated. | | :broken_heart: | patch | 0m 8s | https://github.com/apache/hbase/pull/618 does not apply to branch-2. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/618 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-618/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] anoopsjohn merged pull request #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
anoopsjohn merged pull request #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] anoopsjohn merged pull request #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
anoopsjohn merged pull request #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324037181 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java ## @@ -35,6 +30,13 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.metrics.impl.FastLongHistogram; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.GsonUtil; + + +import org.apache.hbase.thirdparty.com.google.gson.Gson; Review comment: no, modules should expressly list those dependencies they directly use. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] anoopsjohn merged pull request #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
anoopsjohn merged pull request #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/617 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani opened a new pull request #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani opened a new pull request #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/619 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani commented on issue #619: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/619#issuecomment-531098010 @anoopsjohn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani commented on issue #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/618#issuecomment-531097617 @anoopsjohn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani opened a new pull request #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani opened a new pull request #618: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/618 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani commented on issue #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/617#issuecomment-531097204 FYI @anoopsjohn This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani opened a new pull request #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity
virajjasani opened a new pull request #617: HBASE-22760 : Pause/Resume/Query Snapshot Auto Cleanup Activity URL: https://github.com/apache/hbase/pull/617 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] asf-ci commented on issue #31: HBASE-23018 - [HBCK2] Add useful messages when report/fixing missing regions in meta
asf-ci commented on issue #31: HBASE-23018 - [HBCK2] Add useful messages when report/fixing missing regions in meta URL: https://github.com/apache/hbase-operator-tools/pull/31#issuecomment-531096296 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/100/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase-operator-tools] daisukebe opened a new pull request #31: [HBCK2] Add useful messages when report/fixing missing regions in meta
daisukebe opened a new pull request #31: [HBCK2] Add useful messages when report/fixing missing regions in meta URL: https://github.com/apache/hbase-operator-tools/pull/31 Informing the user that - No missing regions found in meta - Likely a table name is passed without the NS portion - See also the command usage when executing reportMissingRegionsInMeta or addFsRegionsMissingInMeta. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] ramkrish86 merged pull request #614: HBASE-22929 - MemStoreLAB ChunkCreator may memory leak(ram)
ramkrish86 merged pull request #614: HBASE-22929 - MemStoreLAB ChunkCreator may memory leak(ram) URL: https://github.com/apache/hbase/pull/614 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324031522 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java ## @@ -35,6 +30,13 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.metrics.impl.FastLongHistogram; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.GsonUtil; + + +import org.apache.hbase.thirdparty.com.google.gson.Gson; Review comment: hbase-server is getting it from hbase-common transitively, it should be fine? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324031028 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: without Jackson1, minicluster is not coming up, this is mostly because Hadoop requires Jackson1 and hence, just kept this dependency at provided scope This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r324031028 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: without Jackson1, minicluster is not coming up, this is mostly because Hadoop requires Jackson1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-23018) [HBCK2] Add useful messages when report/fixing missing regions in meta
[
https://issues.apache.org/jira/browse/HBASE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HBASE-23018:
--
Description:
When a table name is wrongly passed without the namespace portion,
*reportMissingRegionsInMeta* just reports:
{code}
Missing Regions for each table:
{code}
similarly, *addFsRegionsMissingInMeta* just reports:
{code}
Regions re-added into Meta: 0
{code}
Let' make it more friendly by letting the user know that the table name is
wrongly passed while it's mentioned in the command usage.
was:
When a table name is wrongly passed without the namespace portion,
*reportMissingRegionsInMeta* just reports:
{code}
Missing Regions for each table:
{code}
similarly, *addFsRegionsMissingInMeta* just reports:
{code}
Regions re-added into Meta: 0
{code}
It's friendly to let the user know that the table name is wrongly passed while
it's mentioned in the command usage.
> [HBCK2] Add useful messages when report/fixing missing regions in meta
> --
>
> Key: HBASE-23018
> URL: https://issues.apache.org/jira/browse/HBASE-23018
> Project: HBase
> Issue Type: Improvement
> Components: hbck2, Operability
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
>
> When a table name is wrongly passed without the namespace portion,
> *reportMissingRegionsInMeta* just reports:
> {code}
> Missing Regions for each table:
> {code}
> similarly, *addFsRegionsMissingInMeta* just reports:
> {code}
> Regions re-added into Meta: 0
> {code}
> Let' make it more friendly by letting the user know that the table name is
> wrongly passed while it's mentioned in the command usage.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (HBASE-23018) [HBCK2] Add useful messages when report/fixing missing regions in meta
Daisuke Kobayashi created HBASE-23018:
-
Summary: [HBCK2] Add useful messages when report/fixing missing
regions in meta
Key: HBASE-23018
URL: https://issues.apache.org/jira/browse/HBASE-23018
Project: HBase
Issue Type: Improvement
Components: hbck2, Operability
Reporter: Daisuke Kobayashi
When a table name is wrongly passed without the namespace portion,
*reportMissingRegionsInMeta* just reports:
{code}
Missing Regions for each table:
{code}
similarly, *addFsRegionsMissingInMeta* just reports:
{code}
Regions re-added into Meta: 0
{code}
It's friendly to let the user know that the table name is wrongly passed while
it's mentioned in the command usage.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HBASE-23018) [HBCK2] Add useful messages when report/fixing missing regions in meta
[
https://issues.apache.org/jira/browse/HBASE-23018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi reassigned HBASE-23018:
-
Assignee: Daisuke Kobayashi
> [HBCK2] Add useful messages when report/fixing missing regions in meta
> --
>
> Key: HBASE-23018
> URL: https://issues.apache.org/jira/browse/HBASE-23018
> Project: HBase
> Issue Type: Improvement
> Components: hbck2, Operability
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
>
> When a table name is wrongly passed without the namespace portion,
> *reportMissingRegionsInMeta* just reports:
> {code}
> Missing Regions for each table:
> {code}
> similarly, *addFsRegionsMissingInMeta* just reports:
> {code}
> Regions re-added into Meta: 0
> {code}
> It's friendly to let the user know that the table name is wrongly passed
> while it's mentioned in the command usage.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[
https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928983#comment-16928983
]
Hudson commented on HBASE-22796:
Results for branch master
[build #1431 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1431/]: (/)
*{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1431//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1431//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1431//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [HBCK2] Add fix of overlaps to fixMeta hbck Service
> ---
>
> Key: HBASE-22796
> URL: https://issues.apache.org/jira/browse/HBASE-22796
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
> Attachments:
> 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch,
> HBASE-22796.master.001.patch, HBASE-22796.master.002.patch,
> HBASE-22796.master.003.patch, HBASE-22796.master.004.patch,
> HBASE-22796.master.005.patch
>
>
> fixMeta currently does holes in meta only courtesy of HBASE-22771 which added
> fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about
> adding fix of overlaps to general fixMeta call.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[
https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928974#comment-16928974
]
HBase QA commented on HBASE-22796:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m
49s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 13s{color} | {color:green} Patch does not cause any errors with Hadoop
2.8.5 2.9.2 or 3.1.2. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}296m 54s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}355m 11s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterShutdown |
| | hadoop.hbase.master.procedure.TestSCPWithReplicasWithoutZKCoordinated |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/PreCommit-HBASE-Build/899/artifact/patchprocess/Dockerfile
|
| JIRA Issue | HBASE-22796 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12980226/0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch
|
| Optional Tests | dupname asflicense javac javadoc unit spotbugs findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux d2349327da3a 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22
16:55:30 UTC 2019 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[
https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928972#comment-16928972
]
Hudson commented on HBASE-22796:
Results for branch branch-2.1
[build #1593 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1593/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1593//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1593//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/1593//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [HBCK2] Add fix of overlaps to fixMeta hbck Service
> ---
>
> Key: HBASE-22796
> URL: https://issues.apache.org/jira/browse/HBASE-22796
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
> Attachments:
> 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch,
> HBASE-22796.master.001.patch, HBASE-22796.master.002.patch,
> HBASE-22796.master.003.patch, HBASE-22796.master.004.patch,
> HBASE-22796.master.005.patch
>
>
> fixMeta currently does holes in meta only courtesy of HBASE-22771 which added
> fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about
> adding fix of overlaps to general fixMeta call.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[
https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928970#comment-16928970
]
Hudson commented on HBASE-22796:
Results for branch branch-2.2
[build #602 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/602/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/602//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/602//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.2/602//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [HBCK2] Add fix of overlaps to fixMeta hbck Service
> ---
>
> Key: HBASE-22796
> URL: https://issues.apache.org/jira/browse/HBASE-22796
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
> Attachments:
> 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch,
> HBASE-22796.master.001.patch, HBASE-22796.master.002.patch,
> HBASE-22796.master.003.patch, HBASE-22796.master.004.patch,
> HBASE-22796.master.005.patch
>
>
> fixMeta currently does holes in meta only courtesy of HBASE-22771 which added
> fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about
> adding fix of overlaps to general fixMeta call.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[
https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928966#comment-16928966
]
Hudson commented on HBASE-22796:
Results for branch branch-2
[build #2256 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2256/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2256//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2256//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2256//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> [HBCK2] Add fix of overlaps to fixMeta hbck Service
> ---
>
> Key: HBASE-22796
> URL: https://issues.apache.org/jira/browse/HBASE-22796
> Project: HBase
> Issue Type: Sub-task
>Reporter: stack
>Assignee: Sakthi
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2
>
> Attachments:
> 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch,
> HBASE-22796.master.001.patch, HBASE-22796.master.002.patch,
> HBASE-22796.master.003.patch, HBASE-22796.master.004.patch,
> HBASE-22796.master.005.patch
>
>
> fixMeta currently does holes in meta only courtesy of HBASE-22771 which added
> fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about
> adding fix of overlaps to general fixMeta call.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22955) Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
[
https://issues.apache.org/jira/browse/HBASE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928965#comment-16928965
]
Sean Busbey commented on HBASE-22955:
-
I was hoping I could do this with just changes to Jenkinsfile, by making the
use of jdk7 explicit in the appropriate places. However, that won't work with
our current precommit job for testing patches on jira. because in that case
there's only the 1 bootstrapping run and we don't know what branch we're on
until yetus is already running.
So for *all* of our testing to work, we need to get the Dockerfile to properly
set JAVA_HOME to point at jdk7 on branches-1.
> Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
>
>
> Key: HBASE-22955
> URL: https://issues.apache.org/jira/browse/HBASE-22955
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.3.6, 1.4.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> branch-1 nightly job is failing while trying to do our yetus shaded jar test
> because the enforcer plugin complains about a jdk8 jdk-tools jar being
> present.
>
> looking at the yetus footer, it's because jdk8 is being used instead of jdk7:
> {code:java}
> [2019-08-30T13:30:34.353Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T13:30:34.353Z] | Default Java | 1.8.0_222 |
> {code}
> Same thing in the footer of the jdk7 build (which passes, even though it
> shouldn't)
> {code}
> [2019-08-30T15:43:29.103Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T15:43:29.103Z] | Default Java | 1.8.0_222 |
> {code}
> Both builds appear to be properly running in Docker mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Work started] (HBASE-22955) Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
[
https://issues.apache.org/jira/browse/HBASE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-22955 started by Sean Busbey.
---
> Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
>
>
> Key: HBASE-22955
> URL: https://issues.apache.org/jira/browse/HBASE-22955
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.3.6, 1.4.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> branch-1 nightly job is failing while trying to do our yetus shaded jar test
> because the enforcer plugin complains about a jdk8 jdk-tools jar being
> present.
>
> looking at the yetus footer, it's because jdk8 is being used instead of jdk7:
> {code:java}
> [2019-08-30T13:30:34.353Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T13:30:34.353Z] | Default Java | 1.8.0_222 |
> {code}
> Same thing in the footer of the jdk7 build (which passes, even though it
> shouldn't)
> {code}
> [2019-08-30T15:43:29.103Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T15:43:29.103Z] | Default Java | 1.8.0_222 |
> {code}
> Both builds appear to be properly running in Docker mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (HBASE-22955) Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
[
https://issues.apache.org/jira/browse/HBASE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-22955:
---
Assignee: Sean Busbey
> Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
>
>
> Key: HBASE-22955
> URL: https://issues.apache.org/jira/browse/HBASE-22955
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.3.6, 1.4.11
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
>
> branch-1 nightly job is failing while trying to do our yetus shaded jar test
> because the enforcer plugin complains about a jdk8 jdk-tools jar being
> present.
>
> looking at the yetus footer, it's because jdk8 is being used instead of jdk7:
> {code:java}
> [2019-08-30T13:30:34.353Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T13:30:34.353Z] | Default Java | 1.8.0_222 |
> {code}
> Same thing in the footer of the jdk7 build (which passes, even though it
> shouldn't)
> {code}
> [2019-08-30T15:43:29.103Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T15:43:29.103Z] | Default Java | 1.8.0_222 |
> {code}
> Both builds appear to be properly running in Docker mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil…
asf-ci commented on issue #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil… URL: https://github.com/apache/hbase-operator-tools/pull/30#issuecomment-531020723 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/99/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #593: HBASE-22927 Upgrade Mockito version for jdk11
Apache-HBase commented on issue #593: HBASE-22927 Upgrade Mockito version for jdk11 URL: https://github.com/apache/hbase/pull/593#issuecomment-531015350 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 8s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 0s | No case conflicting files found. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :yellow_heart: | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ master Compile Tests _ | | :green_heart: | mvninstall | 5m 53s | master passed | | :green_heart: | compile | 3m 13s | master passed | | :green_heart: | shadedjars | 4m 55s | branch has no errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 2m 50s | master passed | ||| _ Patch Compile Tests _ | | :green_heart: | mvninstall | 5m 25s | the patch passed | | :green_heart: | compile | 3m 18s | the patch passed | | :green_heart: | javac | 3m 18s | the patch passed | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | xml | 0m 1s | The patch has no ill-formed XML file. | | :broken_heart: | shadedjars | 5m 7s | patch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 17m 27s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2. | | :green_heart: | javadoc | 3m 25s | the patch passed | ||| _ Other Tests _ | | :broken_heart: | unit | 277m 46s | root in the patch failed. | | :green_heart: | asflicense | 0m 31s | The patch does not generate ASF License warnings. | | | | 337m 57s | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-593/5/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hbase/pull/593 | | Optional Tests | dupname asflicense javac javadoc unit shadedjars hadoopcheck xml compile | | uname | Linux 8174f2aa71a3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 GNU/Linux | | Build tool | maven | | Personality | /home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-593/out/precommit/personality/provided.sh | | git revision | master / c0e5c15887 | | Default Java | 1.8.0_181 | | shadedjars | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-593/5/artifact/out/patch-shadedjars.txt | | unit | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-593/5/artifact/out/patch-unit-root.txt | | Test Results | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-593/5/testReport/ | | Max. process+thread count | 4594 (vs. ulimit of 1) | | modules | C: . U: . | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-593/5/console | | versions | git=2.11.0 maven=2018-06-17T18:33:14Z) | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928897#comment-16928897 ] Sean Busbey commented on HBASE-22749: - It looks like the difference is that you're building with Eclipse and our lifecycle mapping was missing an entry in hbase-protocol? If I have that right, could you please file a different jira for correcting that and get that bit posted on its own? > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, > HBASE-22749-branch-2.2-v4.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928890#comment-16928890 ] Sean Busbey commented on HBASE-22749: - please make a github PR, the patch is large enough that it'll be hard to provide review feedback otherwise. > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, > HBASE-22749-branch-2.2-v4.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928889#comment-16928889 ] Sean Busbey commented on HBASE-22749: - bq. Nevertheless, failed again: what's the output of your maven version command? > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, > HBASE-22749-branch-2.2-v4.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-connectors] busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0.
busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0. URL: https://github.com/apache/hbase-connectors/pull/41#discussion_r323948011 ## File path: spark/hbase-spark-it/src/main/appended-resources/META-INF/LICENSE ## @@ -0,0 +1,1454 @@ + +Apache HBase - Spark Integration Tests contained works + +This product contains additional works that are distributed under licenses +other than ASL v2. Details below. + + + +This product includes the following under BSD License. +* Kryo Shaded, Copyright (c) 2008-2018, Nathan Sweet All rights reserved. +* Minlog, Copyright (c) 2008, Nathan Sweet, Nathan Sweet All rights reserved. +* Protocol Buffer Java API, Copyright 2008 Google Inc. All rights reserved. +* JSch, Copyright (c) 2002-2015 Atsuhiko Yamanaka, JCraft,Inc. All rights reserved. +* Paranamer Core, Portions copyright (c) 2006-2018 Paul Hammant & ThoughtWorks Inc +Portions copyright (c) 2000-2007 INRIA, France Telecom +All rights reserved. +* Py4J, Copyright (c) 2009-2018, Barthelemy Dagenais and individual contributors. +All rights reserved. +* ANTLR 4 Runtime, Copyright (c) 2012-2017 The ANTLR Project. All rights reserved. +* commons-compiler, Copyright (c) 2001-2016, Arno Unkrig +Copyright (c) 2015-2016 TIBCO Software Inc. +All rights reserved. +* janino, Copyright (c) 2001-2016, Arno Unkrig +Copyright (c) 2015-2016 TIBCO Software Inc. +All rights reserved. +* levelbdjni-all, Copyright (c) 2011 FuseSource Corp. All rights reserved. +* Hamcrest Core, Copyright (c) 2000-2015 www.hamcrest.org. All rights reserved. +* Scala Library, Scala Compiler, scala-parser-combinators, scala-xml +* xmlenc library + +BSD License + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are +met: + +* Redistributions of source code must retain the above copyright +notice, this list of conditions and the following disclaimer. +* Redistributions in binary form must reproduce the above +copyright notice, this list of conditions and the following disclaimer +in the documentation and/or other materials provided with the +distribution. +* Neither the name of Google Inc. nor the names of its +contributors may be used to endorse or promote products derived from +this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS +"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT +LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR +A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT +OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, +SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + + +This product includes jersey-client, jersey-core, jersey-json, jersey-server, +jersey-guice, JAXB RI, javax.annotation API, Java Servlet API, JavaServer Pages(TM) API, +javax.ws.rs-api, JAXB API bundle for GlassFish V3, Expression Language 3.0, +HK2 API module, ServiceLocator Default Implementation, HK2 Implementation Utilities, +OSGi resource locator bundle, aopalliance version 1.0 repackaged as a module, +javax.inject:1 as OSGi bundle, jersey-repackaged-guava, jersey-container-servlet, +jersey-container-servlet-core, jersey-core-client, jersey-core-common, jersey-core-server, +jersey-media-jaxb, JSP implementation under the +license COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.1 + +COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.1 + +1. Definitions. + +1.1. "Contributor" means each individual or entity that creates or +contributes to the creation of Modifications. + +1.2. "Contributor Version" means the combination of the Original +Software, prior Modifications used by a Contributor (if any), and +the Modifications made by that particular Contributor. + +1.3. "Covered Software" means (a) the Original Software, or (b) +Modifications, or (c) the combination of files containing Original +Software with files containing Modifications, in each case including +portions thereof. + +1.4. "Executable" means the Covered Software in any form other than +Source Code. + +1.5. "Initial Developer" means the individual or entity that first +makes Original Software available under this License. + +1.6. "Larger Work" means a work which combines Covered Software or +portions thereof with code not governed by the terms of this License. + +1.7. "License" means this document. + +1.8. "Licensable"
[GitHub] [hbase-connectors] busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0.
busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0. URL: https://github.com/apache/hbase-connectors/pull/41#discussion_r323945661 ## File path: kafka/hbase-kafka-model/src/main/appended-resources/META-INF/LICENSE ## @@ -0,0 +1,69 @@ + +Apache HBase - Model Objects for Kafka Proxy contained works + +This product contains additional works that are distributed under licenses +other than ASL v2. Details below. + + + +This product includes SLF4J API Module version +1.7.25 under the terms of the MIT license which can be found at: +https://www.slf4j.org/license.html + +Copyright (c) 2004-2017 QOS.ch + All rights reserved. + + Permission is hereby granted, free of charge, to any person obtaining + a copy of this software and associated documentation files (the + "Software"), to deal in the Software without restriction, including + without limitation the rights to use, copy, modify, merge, publish, + distribute, sublicense, and/or sell copies of the Software, and to + permit persons to whom the Software is furnished to do so, subject to + the following conditions: + + The above copyright notice and this permission notice shall be + included in all copies or substantial portions of the Software. + + THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + MERCHANTABILITY,FITNESSFORA PARTICULARPURPOSEAND + NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE + LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION + OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION + WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. + + +This product includes ParaNamer Core Module version 2.3 +under the terms of the BSD license which can be found at: +https://github.com/paul-hammant/paranamer/releases/tag/paranamer-2.3 Review comment: I think the correct link is https://github.com/paul-hammant/paranamer/blob/paranamer-2.3/LICENSE.txt This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] Apache-HBase commented on issue #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
Apache-HBase commented on issue #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#issuecomment-531004668 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 1m 26s | Docker mode activated. | ||| _ Prechecks _ | | :green_heart: | dupname | 0m 1s | No case conflicting files found. | | :green_heart: | hbaseanti | 0m 0s | Patch does not have any anti-patterns. | | :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | | :green_heart: | test4tests | 0m 0s | The patch appears to include 3 new or modified test files. | ||| _ branch-1 Compile Tests _ | | :blue_heart: | mvndep | 6m 8s | Maven dependency ordering for branch | | :green_heart: | mvninstall | 3m 17s | branch-1 passed | | :green_heart: | compile | 1m 52s | branch-1 passed | | :green_heart: | checkstyle | 9m 34s | branch-1 passed | | :broken_heart: | shadedjars | 0m 12s | branch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | javadoc | 4m 14s | branch-1 passed | | :blue_heart: | spotbugs | 8m 50s | Used deprecated FindBugs config; considering switching to SpotBugs. | | :blue_heart: | findbugs | 0m 22s | branch/hbase-resource-bundle no findbugs output file (findbugsXml.xml) | | :blue_heart: | findbugs | 0m 16s | branch/hbase-testing-util no findbugs output file (findbugsXml.xml) | | :blue_heart: | findbugs | 0m 13s | branch/hbase-shaded/hbase-shaded-server no findbugs output file (findbugsXml.xml) | | :blue_heart: | findbugs | 0m 16s | branch/hbase-shaded/hbase-shaded-testing-util no findbugs output file (findbugsXml.xml) | | :blue_heart: | findbugs | 0m 10s | branch/hbase-shaded/hbase-shaded-testing-util-tester no findbugs output file (findbugsXml.xml) | ||| _ Patch Compile Tests _ | | :blue_heart: | mvndep | 0m 22s | Maven dependency ordering for patch | | :green_heart: | mvninstall | 1m 58s | the patch passed | | :green_heart: | compile | 1m 53s | the patch passed | | :green_heart: | javac | 1m 53s | the patch passed | | :green_heart: | checkstyle | 0m 8s | The patch passed checkstyle in hbase-resource-bundle | | :green_heart: | checkstyle | 0m 27s | The patch passed checkstyle in hbase-common | | :green_heart: | checkstyle | 0m 33s | The patch passed checkstyle in hbase-client | | :green_heart: | checkstyle | 1m 49s | hbase-server: The patch generated 0 new + 260 unchanged - 25 fixed = 260 total (was 285) | | :green_heart: | checkstyle | 0m 7s | The patch passed checkstyle in hbase-testing-util | | :green_heart: | checkstyle | 0m 7s | The patch passed checkstyle in hbase-shaded-server | | :green_heart: | checkstyle | 0m 9s | The patch passed checkstyle in hbase-shaded-testing-util | | :green_heart: | checkstyle | 0m 8s | The patch passed checkstyle in hbase-shaded-testing-util-tester | | :green_heart: | checkstyle | 6m 6s | root: The patch generated 0 new + 260 unchanged - 25 fixed = 260 total (was 285) | | :green_heart: | whitespace | 0m 0s | The patch has no whitespace issues. | | :green_heart: | xml | 0m 10s | The patch has no ill-formed XML file. | | :broken_heart: | shadedjars | 0m 11s | patch has 10 errors when building our shaded downstream artifacts. | | :green_heart: | hadoopcheck | 4m 58s | Patch does not cause any errors with Hadoop 2.8.5 2.9.2. | | :green_heart: | javadoc | 4m 11s | the patch passed | | :blue_heart: | findbugs | 0m 9s | hbase-resource-bundle has no data from findbugs | | :blue_heart: | findbugs | 0m 17s | hbase-testing-util has no data from findbugs | | :blue_heart: | findbugs | 0m 14s | hbase-shaded/hbase-shaded-server has no data from findbugs | | :blue_heart: | findbugs | 0m 15s | hbase-shaded/hbase-shaded-testing-util has no data from findbugs | | :blue_heart: | findbugs | 0m 10s | hbase-shaded/hbase-shaded-testing-util-tester has no data from findbugs | ||| _ Other Tests _ | | :broken_heart: | unit | 185m 54s | root in the patch failed. | | :green_heart: | asflicense | 3m 54s | The patch does not generate ASF License warnings. | | | | 275m 14s | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hbase.util.TestHBaseFsck | | | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2 | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.0 Server=19.03.0 base:
[GitHub] [hbase-connectors] busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0.
busbey commented on a change in pull request #41: HBASE-22338 LICENSE file only contains Apache 2.0. URL: https://github.com/apache/hbase-connectors/pull/41#discussion_r323947041 ## File path: kafka/hbase-kafka-model/src/main/appended-resources/META-INF/NOTICE.txt ## @@ -0,0 +1,26 @@ +Apache HBase - Kafka Model +Copyright 2019 The Apache Software Foundation + +This product includes software developed at +The Apache Software Foundation (http://www.apache.org/). + + +Apache HBase - Kafka Model contained works + +This product contains additional works that are distributed under licenses +other than Apache Software License Version 2. See LICENSE for full details + + +This product includes the SLF4J API Module Review comment: that file just tells some tools that look at the jar what dependencies are needed at runtime. It doesn't mean we're packaging and distributing them. a given LICENSE/NOTICE set of files should only cover the artifact they're found in. So each of these module should have a `src/main/appended-resources/META-INF/LICENSE`/`NOTICE` based on the contents of the jar that will be made by that module. the only case where these would include dependencies is if we are making a "fat jar" or using the maven shade plugin to package our dependency into the jar. I *think* that means for the connectors repo, it's only the convenience binary tarball that needs the aggregate information about our dependencies. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r323943116 ## File path: hbase-testing-util/pom.xml ## @@ -164,6 +164,11 @@ + +org.codehaus.jackson Review comment: this jackson dependency shouldn't be needed anymore This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#discussion_r323943767 ## File path: hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCacheUtil.java ## @@ -35,6 +30,13 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.metrics.impl.FastLongHistogram; import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.GsonUtil; + + +import org.apache.hbase.thirdparty.com.google.gson.Gson; Review comment: hbase-server's pom needs to include a dependency on hbase-shaded-miscellaneous. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-22749: -- Attachment: HBASE-22749-branch-2.2-v4.patch > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, > HBASE-22749-branch-2.2-v4.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928879#comment-16928879 ] Vladimir Rodionov commented on HBASE-22749: --- v4 should build on 2.2. > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, > HBASE-22749-branch-2.2-v4.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil…
asf-ci commented on issue #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil… URL: https://github.com/apache/hbase-operator-tools/pull/30#issuecomment-530996367 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/98/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928876#comment-16928876 ] stack edited comment on HBASE-22796 at 9/12/19 8:32 PM: Pushed addendum on branch-2.1 to master. Thanks for review [~psomogyi]. Made your changes on push. was (Author: stack): Pushed on branch-2.1 to master. Thanks for review [~psomogyi]. Made your chagnes on push. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: > 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch, > HBASE-22796.master.001.patch, HBASE-22796.master.002.patch, > HBASE-22796.master.003.patch, HBASE-22796.master.004.patch, > HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22796: -- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed on branch-2.1 to master. Thanks for review [~psomogyi]. Made your chagnes on push. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: > 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch, > HBASE-22796.master.001.patch, HBASE-22796.master.002.patch, > HBASE-22796.master.003.patch, HBASE-22796.master.004.patch, > HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-23002) [HBCK2/hbase-operator-tools] Create an assembly that builds an hbase-operator-tools tgz
[ https://issues.apache.org/jira/browse/HBASE-23002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928873#comment-16928873 ] stack commented on HBASE-23002: --- I put up a PR with what have so far. Could commit it and then keep working on it. There is no site doc. There is the high-level README but not enough. > [HBCK2/hbase-operator-tools] Create an assembly that builds an > hbase-operator-tools tgz > --- > > Key: HBASE-23002 > URL: https://issues.apache.org/jira/browse/HBASE-23002 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Major > > Was going to build a convenience binary tgz as part of the first release of > hbase-operator-tools. Not sure how just yet; best would be if it were a > fatjar with all dependencies but that'd be kinda insane at same time since > the tgz would be massive. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-operator-tools] saintstack opened a new pull request #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil…
saintstack opened a new pull request #30: HBASE-23002 [HBCK2/hbase-operator-tools] Create an assembly that buil… URL: https://github.com/apache/hbase-operator-tools/pull/30 …ds an hbase-operator-tools tgz Add an hbase-assembly that is just for creating assemblies. Also changed groupid from org.apache.hbase to org.apache.hbase.operator.tools so lands into a different area in repository -- less crowded, easier to differentiate from core hbase. Creates a bin assembly (src is made elsewhere in release script). Adds a first-cut at RELEASENOTES and CHANGES (release script will fix up what is here based off what is in JIRA to make it suit the 1.0.0RC0) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928870#comment-16928870 ] Peter Somogyi commented on HBASE-22796: --- +1 Tested on branch-2 and master, TestMetaFixer passes. Can you change the commit message when committing? It has a different JIRA's commit message and all the content is in a single line (Signed-off-by also in the first line) Thanks [~stack] for the fix! > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: > 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch, > HBASE-22796.master.001.patch, HBASE-22796.master.002.patch, > HBASE-22796.master.003.patch, HBASE-22796.master.004.patch, > HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22796: -- Status: Patch Available (was: Reopened) Here is an addendum. Was a bug. Worked on master because its hbase:meta is different w/ integrated namespace content. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: > 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch, > HBASE-22796.master.001.patch, HBASE-22796.master.002.patch, > HBASE-22796.master.003.patch, HBASE-22796.master.004.patch, > HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-22796: -- Attachment: 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: > 0001-HBASE-23014-Should-not-show-split-parent-regions-in-.patch, > HBASE-22796.master.001.patch, HBASE-22796.master.002.patch, > HBASE-22796.master.003.patch, HBASE-22796.master.004.patch, > HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-operator-tools] asf-ci commented on issue #29: HBASE-23003 [HBCK2/hbase-operator-tools] Release-making scripts
asf-ci commented on issue #29: HBASE-23003 [HBCK2/hbase-operator-tools] Release-making scripts URL: https://github.com/apache/hbase-operator-tools/pull/29#issuecomment-530975411 Refer to this link for build results (access rights to CI server needed): https://builds.apache.org/job/PreCommit-HBASE-OPERATOR-TOOLS-Build/97/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-23003) [HBCK2/hbase-operator-tools] Release-making scripts
[ https://issues.apache.org/jira/browse/HBASE-23003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928845#comment-16928845 ] stack commented on HBASE-23003: --- [~psomogyi] Thanks for trying to review. Patch posted was incomplete. I just put up a PR that I've tested. It seems to do the right thing. > [HBCK2/hbase-operator-tools] Release-making scripts > --- > > Key: HBASE-23003 > URL: https://issues.apache.org/jira/browse/HBASE-23003 > Project: HBase > Issue Type: Task >Reporter: stack >Priority: Major > Attachments: > 0001-HBASE-23003-HBCK2-hbase-operator-tools-Release-makin.patch > > > Make some scripts for creating hbase-operator-tools releases so its easy to > do and not subject to vagaries of environment or RM's attention-to-detail (or > not). -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase-operator-tools] saintstack opened a new pull request #29: HBASE-23003 [HBCK2/hbase-operator-tools] Release-making scripts
saintstack opened a new pull request #29: HBASE-23003 [HBCK2/hbase-operator-tools] Release-making scripts URL: https://github.com/apache/hbase-operator-tools/pull/29 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928832#comment-16928832
]
Vladimir Rodionov commented on HBASE-22749:
---
Nevertheless, failed again:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade
(aggregate-into-a-jar-with-relocated-third-parties) on project
hbase-shaded-client: Error creating shaded jar: duplicate entry:
META-INF/services/org.apache.hadoop.hbase.shaded.com.fasterxml.jackson.core.ObjectCodec
-> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-shaded-client
{code}
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf,
> HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase] xcangCRM merged pull request #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
xcangCRM merged pull request #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] smerkerg commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
smerkerg commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530964452 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] xcangCRM edited a comment on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
xcangCRM edited a comment on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530960337 This patch's related tests are all passing. And canary class is separated to rest of the code. I think it's safe and good to commit now, unless any objections. thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] xcangCRM commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
xcangCRM commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530960337 This change's related tests are all passing. And canary class is separated to rest of the code. I think it's safe and good to commit now, unless any objections. thanks This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928823#comment-16928823
]
Sean Busbey commented on HBASE-22749:
-
I just got a clean build off of branch-2.2 ref {{ }}.
Since branch-2.2 is going through release candidates ATM, the version numbers
don't have SNAPSHOT in them, so you have to be careful to keep Maven from using
an artifact from an old build. I use the following line, but you have to be
careful because it will forcibly delete any untracked changes in the local
working directory:
{code}
git clean -xdf && rm -rf ~/.m2/repository/org/apache/hbase/* && mvn
-Dtest=NoTests -Dit.test=NOITs package install verify
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 10:41 min
[INFO] Finished at: 2019-09-12T13:51:11-05:00
[INFO]
{code}
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf,
> HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Comment Edited] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928823#comment-16928823
]
Sean Busbey edited comment on HBASE-22749 at 9/12/19 6:52 PM:
--
I just got a clean build off of branch-2.2 ref
{{a010bf154ae31c7aa4cb165a34bc7d42a6b70f2f}}.
Since branch-2.2 is going through release candidates ATM, the version numbers
don't have SNAPSHOT in them, so you have to be careful to keep Maven from using
an artifact from an old build. I use the following line, but you have to be
careful because it will forcibly delete any untracked changes in the local
working directory:
{code}
git clean -xdf && rm -rf ~/.m2/repository/org/apache/hbase/* && mvn
-Dtest=NoTests -Dit.test=NOITs package install verify
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 10:41 min
[INFO] Finished at: 2019-09-12T13:51:11-05:00
[INFO]
{code}
was (Author: busbey):
I just got a clean build off of branch-2.2 ref {{ }}.
Since branch-2.2 is going through release candidates ATM, the version numbers
don't have SNAPSHOT in them, so you have to be careful to keep Maven from using
an artifact from an old build. I use the following line, but you have to be
careful because it will forcibly delete any untracked changes in the local
working directory:
{code}
git clean -xdf && rm -rf ~/.m2/repository/org/apache/hbase/* && mvn
-Dtest=NoTests -Dit.test=NOITs package install verify
...
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 10:41 min
[INFO] Finished at: 2019-09-12T13:51:11-05:00
[INFO]
{code}
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf,
> HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase] caroliney14 commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
caroliney14 commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530955761 @busbey do you mean run the shaded jars test locally? how do I run just that test suite? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
busbey commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530950734 the shadedjars test failure is [HBASE-22955](https://issues.apache.org/jira/browse/HBASE-22955). you should run the test locally with jdk7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22866) Multiple slf4j-log4j provider versions included in binary package (branch-1)
[
https://issues.apache.org/jira/browse/HBASE-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928812#comment-16928812
]
Sean Busbey commented on HBASE-22866:
-
the issue here and the issue on HBASE-22728 is HBASE-22955.
> Multiple slf4j-log4j provider versions included in binary package (branch-1)
>
>
> Key: HBASE-22866
> URL: https://issues.apache.org/jira/browse/HBASE-22866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22866.branch-1.000.patch,
> HBASE-22866.branch-1.000.patch
>
>
> Examining binary assembly results there are multiple versions of slf4j-log4j
> in lib/
> {noformat}
> slf4j-api-1.7.7.jar
> slf4j-log4j12-1.6.1.jar
> slf4j-log4j12-1.7.10.jar
> slf4j-log4j12-1.7.7.jar
> {noformat}
> We aren't managing slf4j-log4j12 dependency versions correctly, somehow.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22955) Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
[
https://issues.apache.org/jira/browse/HBASE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928811#comment-16928811
]
Sean Busbey commented on HBASE-22955:
-
also happening with precommit checks.
> Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
>
>
> Key: HBASE-22955
> URL: https://issues.apache.org/jira/browse/HBASE-22955
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.3.6, 1.4.11
>Reporter: Sean Busbey
>Priority: Major
>
> branch-1 nightly job is failing while trying to do our yetus shaded jar test
> because the enforcer plugin complains about a jdk8 jdk-tools jar being
> present.
>
> looking at the yetus footer, it's because jdk8 is being used instead of jdk7:
> {code:java}
> [2019-08-30T13:30:34.353Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T13:30:34.353Z] | Default Java | 1.8.0_222 |
> {code}
> Same thing in the footer of the jdk7 build (which passes, even though it
> shouldn't)
> {code}
> [2019-08-30T15:43:29.103Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T15:43:29.103Z] | Default Java | 1.8.0_222 |
> {code}
> Both builds appear to be properly running in Docker mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (HBASE-22955) Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
[
https://issues.apache.org/jira/browse/HBASE-22955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-22955:
Summary: Branches-1 precommit and nightly yetus jobs are using jdk8 for
jdk7 jobs (was: Branches-1 nightly yetus jobs are using jdk8 for jdk7 jobs)
> Branches-1 precommit and nightly yetus jobs are using jdk8 for jdk7 jobs
>
>
> Key: HBASE-22955
> URL: https://issues.apache.org/jira/browse/HBASE-22955
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0, 1.3.6, 1.4.11
>Reporter: Sean Busbey
>Priority: Major
>
> branch-1 nightly job is failing while trying to do our yetus shaded jar test
> because the enforcer plugin complains about a jdk8 jdk-tools jar being
> present.
>
> looking at the yetus footer, it's because jdk8 is being used instead of jdk7:
> {code:java}
> [2019-08-30T13:30:34.353Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T13:30:34.353Z] | Default Java | 1.8.0_222 |
> {code}
> Same thing in the footer of the jdk7 build (which passes, even though it
> shouldn't)
> {code}
> [2019-08-30T15:43:29.103Z] | maven | version: Apache Maven 3.0.5 |
> [2019-08-30T15:43:29.103Z] | Default Java | 1.8.0_222 |
> {code}
> Both builds appear to be properly running in Docker mode.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase] xcangCRM commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary
xcangCRM commented on issue #612: HBASE-22804 Provide an API to get list of successful regions and total expected regions in Canary URL: https://github.com/apache/hbase/pull/612#issuecomment-530949492 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] virajjasani commented on issue #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on issue #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587) URL: https://github.com/apache/hbase/pull/616#issuecomment-530947480 Except for supplemental-models.xml and adding hbase-shaded-miscellaneous dependency, mostly this is backport of HBASE-20587 for relevant files(a couple of files on branch-1 don't have Jackson2 and hence, unchanged in this PR) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
busbey commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323883325
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[jira] [Comment Edited] (HBASE-22866) Multiple slf4j-log4j provider versions included in binary package (branch-1)
[
https://issues.apache.org/jira/browse/HBASE-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928798#comment-16928798
]
Viraj Jasani edited comment on HBASE-22866 at 9/12/19 6:10 PM:
---
[~xucang] This seems to be issue with branch-1, I am not much aware of it
though.
[~busbey] has provided some context about this in HBASE-22728,
In his own words:
{quote}I think we used to accidentally include the tools.jar from the JDK in
our assembly? we can't do that because it's under the same terms as the JDK. if
it's also in the branch and shows up on nightly, then I wouldn't worry about
fixing it here. But we should figure out what's going on, generally. I'll make
a jira and dig in when I can. If anyone else has time before a linked jira
shows up, please feel free to not wait for me.
{quote}
was (Author: vjasani):
This seems to be issue with branch-1, I am not much aware of it though.
[~busbey] has provided some context about this in HBASE-22728,
In his own words:
```
I think we used to accidentally include the tools.jar from the JDK in our
assembly? we can't do that because it's under the same terms as the JDK. if
it's also in the branch and shows up on nightly, then I wouldn't worry about
fixing it here. But we should figure out what's going on, generally. I'll make
a jira and dig in when I can. If anyone else has time before a linked jira
shows up, please feel free to not wait for me.
```
> Multiple slf4j-log4j provider versions included in binary package (branch-1)
>
>
> Key: HBASE-22866
> URL: https://issues.apache.org/jira/browse/HBASE-22866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22866.branch-1.000.patch,
> HBASE-22866.branch-1.000.patch
>
>
> Examining binary assembly results there are multiple versions of slf4j-log4j
> in lib/
> {noformat}
> slf4j-api-1.7.7.jar
> slf4j-log4j12-1.6.1.jar
> slf4j-log4j12-1.7.10.jar
> slf4j-log4j12-1.7.7.jar
> {noformat}
> We aren't managing slf4j-log4j12 dependency versions correctly, somehow.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22866) Multiple slf4j-log4j provider versions included in binary package (branch-1)
[
https://issues.apache.org/jira/browse/HBASE-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928798#comment-16928798
]
Viraj Jasani commented on HBASE-22866:
--
This seems to be issue with branch-1, I am not much aware of it though.
[~busbey] has provided some context about this in HBASE-22728,
In his own words:
```
I think we used to accidentally include the tools.jar from the JDK in our
assembly? we can't do that because it's under the same terms as the JDK. if
it's also in the branch and shows up on nightly, then I wouldn't worry about
fixing it here. But we should figure out what's going on, generally. I'll make
a jira and dig in when I can. If anyone else has time before a linked jira
shows up, please feel free to not wait for me.
```
> Multiple slf4j-log4j provider versions included in binary package (branch-1)
>
>
> Key: HBASE-22866
> URL: https://issues.apache.org/jira/browse/HBASE-22866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22866.branch-1.000.patch,
> HBASE-22866.branch-1.000.patch
>
>
> Examining binary assembly results there are multiple versions of slf4j-log4j
> in lib/
> {noformat}
> slf4j-api-1.7.7.jar
> slf4j-log4j12-1.6.1.jar
> slf4j-log4j12-1.7.10.jar
> slf4j-log4j12-1.7.7.jar
> {noformat}
> We aren't managing slf4j-log4j12 dependency versions correctly, somehow.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928796#comment-16928796 ] Vladimir Rodionov commented on HBASE-22749: --- It seems, that tip of branch-2.2 is broken. Not a patch related. I tried to build 2.2 w/o patch and it failed with multiple errors. > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[ https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928789#comment-16928789 ] Vladimir Rodionov commented on HBASE-22749: --- Oops, will fix it. > Distributed MOB compactions > > > Key: HBASE-22749 > URL: https://issues.apache.org/jira/browse/HBASE-22749 > Project: HBase > Issue Type: New Feature > Components: mob >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov >Priority: Major > Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf, > HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf > > > There are several drawbacks in the original MOB 1.0 (Moderate Object > Storage) implementation, which can limit the adoption of the MOB feature: > # MOB compactions are executed in a Master as a chore, which limits > scalability because all I/O goes through a single HBase Master server. > # Yarn/Mapreduce framework is required to run MOB compactions in a scalable > way, but this won’t work in a stand-alone HBase cluster. > # Two separate compactors for MOB and for regular store files and their > interactions can result in a data loss (see HBASE-22075) > The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible > implementation, which is free of the above drawbacks and can be used as a > drop in replacement in existing MOB deployments. So, these are design goals > of a MOB 2.0: > # Make MOB compactions scalable without relying on Yarn/Mapreduce framework > # Provide unified compactor for both MOB and regular store files > # Make it more robust especially w.r.t. to data losses. > # Simplify and reduce the overall MOB code. > # Provide 100% compatible implementation with MOB 1.0. > # No migration of data should be required between MOB 1.0 and MOB 2.0 - just > software upgrade. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22866) Multiple slf4j-log4j provider versions included in binary package (branch-1)
[
https://issues.apache.org/jira/browse/HBASE-22866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928782#comment-16928782
]
Xu Cang commented on HBASE-22866:
-
question for [~vjasani] and [~apurtell]
this patch's hadoop-qa run had "shadedjars" with -1 and the patch is still
committed. Do you already know the reason of this shadedjars failure?
The reason I am asking is I am trying to get HBASE-22804 committed but having
the same shadedjars failure. And I am not sure how to add flag to show more
info on HBase Jenkins to reveal which rule was offended. Thanks!
cc. [~caroliney14]
> Multiple slf4j-log4j provider versions included in binary package (branch-1)
>
>
> Key: HBASE-22866
> URL: https://issues.apache.org/jira/browse/HBASE-22866
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.5.0
>Reporter: Andrew Purtell
>Assignee: Viraj Jasani
>Priority: Minor
> Fix For: 1.5.0, 1.3.6, 1.4.11
>
> Attachments: HBASE-22866.branch-1.000.patch,
> HBASE-22866.branch-1.000.patch
>
>
> Examining binary assembly results there are multiple versions of slf4j-log4j
> in lib/
> {noformat}
> slf4j-api-1.7.7.jar
> slf4j-log4j12-1.6.1.jar
> slf4j-log4j12-1.7.10.jar
> slf4j-log4j12-1.7.7.jar
> {noformat}
> We aren't managing slf4j-log4j12 dependency versions correctly, somehow.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323861908
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[jira] [Commented] (HBASE-22749) Distributed MOB compactions
[
https://issues.apache.org/jira/browse/HBASE-22749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928761#comment-16928761
]
Sean Busbey commented on HBASE-22749:
-
The patch {{HBASE-22749-branch-2.2.v3.patch}} fails to build when I apply it on
top of branch-2.2 ref {{a010bf154ae31c7aa4cb165a34bc7d42a6b70f2f}}.
> Distributed MOB compactions
>
>
> Key: HBASE-22749
> URL: https://issues.apache.org/jira/browse/HBASE-22749
> Project: HBase
> Issue Type: New Feature
> Components: mob
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
>Priority: Major
> Attachments: HBASE-22749-branch-2.2-v3.patch, HBase-MOB-2.0-v1.pdf,
> HBase-MOB-2.0-v2.1.pdf, HBase-MOB-2.0-v2.2.pdf, HBase-MOB-2.0-v2.pdf
>
>
> There are several drawbacks in the original MOB 1.0 (Moderate Object
> Storage) implementation, which can limit the adoption of the MOB feature:
> # MOB compactions are executed in a Master as a chore, which limits
> scalability because all I/O goes through a single HBase Master server.
> # Yarn/Mapreduce framework is required to run MOB compactions in a scalable
> way, but this won’t work in a stand-alone HBase cluster.
> # Two separate compactors for MOB and for regular store files and their
> interactions can result in a data loss (see HBASE-22075)
> The design goals for MOB 2.0 were to provide 100% MOB 1.0 - compatible
> implementation, which is free of the above drawbacks and can be used as a
> drop in replacement in existing MOB deployments. So, these are design goals
> of a MOB 2.0:
> # Make MOB compactions scalable without relying on Yarn/Mapreduce framework
> # Provide unified compactor for both MOB and regular store files
> # Make it more robust especially w.r.t. to data losses.
> # Simplify and reduce the overall MOB code.
> # Provide 100% compatible implementation with MOB 1.0.
> # No migration of data should be required between MOB 1.0 and MOB 2.0 - just
> software upgrade.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22956) [HBCK2/hbase-operator-tools] Make first release, 1.0.0
[ https://issues.apache.org/jira/browse/HBASE-22956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928720#comment-16928720 ] stack commented on HBASE-22956: --- [~psomogyi] I don't. We need one though HBASE-22853 > [HBCK2/hbase-operator-tools] Make first release, 1.0.0 > -- > > Key: HBASE-22956 > URL: https://issues.apache.org/jira/browse/HBASE-22956 > Project: HBase > Issue Type: Bug > Components: hbck2 >Reporter: stack >Priority: Major > Fix For: hbase-operator-tools-1.0.0 > > > Make our first release of hbck2/hbase-operator-tools. > First release should have the coverage hbck1 had at least. When the parent > for this issue is done, we'll be at hbck1+. Let us release then (week or > two?). > A release will help operators who have been struggling having to build hbck2 > against different hbase versions. The release should be a "fat > jar"/completely contained with all dependency satisfied so operator can just > fire up hbck2 w/o having to build against an hbase or provide some magic mix > of jars to satisfy hbck2 tool needs. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] busbey commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
busbey commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323840520
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928712#comment-16928712 ] stack commented on HBASE-22796: --- I can do it since I committed this. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22796.master.001.patch, > HBASE-22796.master.002.patch, HBASE-22796.master.003.patch, > HBASE-22796.master.004.patch, HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323838187
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323837823
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323837823
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928705#comment-16928705 ] Sakthi commented on HBASE-22796: Ah, I see you reopened this one. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22796.master.001.patch, > HBASE-22796.master.002.patch, HBASE-22796.master.003.patch, > HBASE-22796.master.004.patch, HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928705#comment-16928705 ] Sakthi edited comment on HBASE-22796 at 9/12/19 4:24 PM: - Ah, I see you reopened this one. Let me start debugging the failure was (Author: jatsakthi): Ah, I see you reopened this one. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22796.master.001.patch, > HBASE-22796.master.002.patch, HBASE-22796.master.003.patch, > HBASE-22796.master.004.patch, HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928703#comment-16928703 ] Peter Somogyi commented on HBASE-22796: --- It can be fixed here with an addendum. The patch was only committed yesterday. > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22796.master.001.patch, > HBASE-22796.master.002.patch, HBASE-22796.master.003.patch, > HBASE-22796.master.004.patch, HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] Apache-HBase commented on issue #566: HBASE-22380 break circle replication when doing bulkload
Apache-HBase commented on issue #566: HBASE-22380 break circle replication when doing bulkload URL: https://github.com/apache/hbase/pull/566#issuecomment-530899023 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | :blue_heart: | reexec | 0m 0s | Docker mode activated. | | :broken_heart: | patch | 0m 6s | https://github.com/apache/hbase/pull/566 does not apply to master. Rebase required? Wrong Branch? See https://yetus.apache.org/documentation/in-progress/precommit-patchnames for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hbase/pull/566 | | Console output | https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-566/8/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.11.0 https://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-22796) [HBCK2] Add fix of overlaps to fixMeta hbck Service
[ https://issues.apache.org/jira/browse/HBASE-22796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928700#comment-16928700 ] Sakthi commented on HBASE-22796: [~psomogyi] shall we file a separate Jira for fixing the failures on branch-2s ? > [HBCK2] Add fix of overlaps to fixMeta hbck Service > --- > > Key: HBASE-22796 > URL: https://issues.apache.org/jira/browse/HBASE-22796 > Project: HBase > Issue Type: Sub-task >Reporter: stack >Assignee: Sakthi >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.1.7, 2.2.2 > > Attachments: HBASE-22796.master.001.patch, > HBASE-22796.master.002.patch, HBASE-22796.master.003.patch, > HBASE-22796.master.004.patch, HBASE-22796.master.005.patch > > > fixMeta currently does holes in meta only courtesy of HBASE-22771 which added > fixMeta to hbck Service; missing was fix of overlaps too. This JIRA is about > adding fix of overlaps to general fixMeta call. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323820464
##
File path:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java
##
@@ -0,0 +1,304 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hbase.regionserver;
+
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID;
+import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertTrue;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hbase.Cell;
+import org.apache.hadoop.hbase.CellBuilder;
+import org.apache.hadoop.hbase.CellBuilderFactory;
+import org.apache.hadoop.hbase.CellBuilderType;
+import org.apache.hadoop.hbase.HBaseClassTestRule;
+import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.KeyValue;
+import org.apache.hadoop.hbase.client.Admin;
+import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder;
+import org.apache.hadoop.hbase.client.Connection;
+import org.apache.hadoop.hbase.client.ConnectionFactory;
+import org.apache.hadoop.hbase.client.Get;
+import org.apache.hadoop.hbase.client.Result;
+import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
+import org.apache.hadoop.hbase.client.TableDescriptorBuilder;
+import org.apache.hadoop.hbase.coprocessor.ObserverContext;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
+import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.coprocessor.RegionObserver;
+import org.apache.hadoop.hbase.io.hfile.HFile;
+import org.apache.hadoop.hbase.io.hfile.HFileContext;
+import org.apache.hadoop.hbase.replication.ReplicationPeerConfig;
+import org.apache.hadoop.hbase.replication.TestReplicationBase;
+import org.apache.hadoop.hbase.testclassification.MediumTests;
+import org.apache.hadoop.hbase.testclassification.ReplicationTests;
+import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool;
+import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.hadoop.hbase.util.Pair;
+import org.apache.hadoop.hdfs.MiniDFSCluster;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.ClassRule;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TemporaryFolder;
+import org.junit.rules.TestName;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Integration test for bulk load replication. Defines two clusters, with two
way replication.
+ * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on
the bulk loaded file
+ * gets into the related table in UTIL1, then also validates the same got
replicated to cluster
+ * UTIL2. Then, bulk loads another file into UTIL2, and checks if related
values are present on
+ * UTIL2, and also gets replicated to UTIL1.
+ * It also defines a preBulkLoad coprocessor that is added to all test table
regions on each of the
+ * clusters, in order to count amount of times bulk load actually gets
invoked. This is to certify
+ * we are not entered in the infinite loop condition addressed by HBASE-22380.
+ */
+@Category({ ReplicationTests.class, MediumTests.class})
+public class TestBulkLoadReplication extends TestReplicationBase {
+
+ @ClassRule
+ public static final HBaseClassTestRule CLASS_RULE =
+HBaseClassTestRule.forClass(TestBulkLoadReplication.class);
+
+ protected static final Logger LOG =
+LoggerFactory.getLogger(TestBulkLoadReplication.class);
+
+ private
[GitHub] [hbase] virajjasani commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
virajjasani commented on a change in pull request #616: HBASE-23015 : Moving
from Jackson2 to shaded Gson (Backport HBASE-20587)
URL: https://github.com/apache/hbase/pull/616#discussion_r323820422
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
##
@@ -52,11 +51,9 @@
* This class is not thread safe.
*/
@InterfaceAudience.Private
-@JsonIgnoreProperties({"indexStatistics", "freeSize", "usedSize"})
Review comment:
Just verified the significant difference in JSON string for Gson and
Jackson2. Sure, will call out in release note.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] ravowlga123 commented on issue #593: HBASE-22927 Upgrade Mockito version for jdk11
ravowlga123 commented on issue #593: HBASE-22927 Upgrade Mockito version for jdk11 URL: https://github.com/apache/hbase/pull/593#issuecomment-530889920 Made the changes @busbey . Hopefully this will pass the test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (HBASE-19320) document the mysterious direct memory leak in hbase
[ https://issues.apache.org/jira/browse/HBASE-19320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928672#comment-16928672 ] Sakthi commented on HBASE-19320: I think we should get this one in. > document the mysterious direct memory leak in hbase > > > Key: HBASE-19320 > URL: https://issues.apache.org/jira/browse/HBASE-19320 > Project: HBase > Issue Type: Improvement >Affects Versions: 1.2.6, 2.0.0 >Reporter: huaxiang sun >Assignee: huaxiang sun >Priority: Critical > Labels: Replication > Attachments: HBASE-19320-master-v001.patch, Screen Shot 2017-11-21 at > 4.43.36 PM.png, Screen Shot 2017-11-21 at 4.44.22 PM.png > > > Recently we run into a direct memory leak case, which takes some time to > trace and debug. Internally discussed with our [~saint@gmail.com], we > thought we had some findings and want to share with the community. > Basically, it is the issue described in > http://www.evanjones.ca/java-bytebuffer-leak.html and it happened to one of > our hbase clusters. > Create the jira first and will fill in more details later. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle
replication when doing bulkload
URL: https://github.com/apache/hbase/pull/566#discussion_r323811028
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
##
@@ -245,14 +252,26 @@ public void replicateEntries(List entries,
final CellScanner cells,
LOG.debug("Finished replicating mutations.");
}
- if (bulkLoadHFileMap != null && !bulkLoadHFileMap.isEmpty()) {
-LOG.debug("Started replicating bulk loaded data.");
-HFileReplicator hFileReplicator =
-new HFileReplicator(this.provider.getConf(this.conf,
replicationClusterId),
+ if(bulkLoadsPerClusters != null) {
+for (List clusterIds : bulkLoadsPerClusters.keySet()) {
+ Map>>> bulkLoadHFileMap =
+bulkLoadsPerClusters.get(clusterIds);
+ if (bulkLoadHFileMap != null && !bulkLoadHFileMap.isEmpty()) {
+if(LOG.isDebugEnabled()) {
+ LOG.debug("Started replicating bulk loaded data from cluster
ids: {}.",
+clusterIds.stream().reduce("", (idsList, id) -> idsList + ", "
+ id));
Review comment:
Indeed (embarrassed)... fixing it on next commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload
wchevreuil commented on a change in pull request #566: HBASE-22380 break circle replication when doing bulkload URL: https://github.com/apache/hbase/pull/566#discussion_r323807923 ## File path: hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkLoadReplication.java ## @@ -0,0 +1,304 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.hadoop.hbase.regionserver; + +import static org.apache.hadoop.hbase.HConstants.REPLICATION_CLUSTER_ID; +import static org.apache.hadoop.hbase.HConstants.REPLICATION_CONF_DIR; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FSDataOutputStream; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hbase.Cell; +import org.apache.hadoop.hbase.CellBuilder; +import org.apache.hadoop.hbase.CellBuilderFactory; +import org.apache.hadoop.hbase.CellBuilderType; +import org.apache.hadoop.hbase.HBaseClassTestRule; +import org.apache.hadoop.hbase.HBaseTestingUtility; +import org.apache.hadoop.hbase.HConstants; +import org.apache.hadoop.hbase.KeyValue; +import org.apache.hadoop.hbase.client.Admin; +import org.apache.hadoop.hbase.client.ColumnFamilyDescriptorBuilder; +import org.apache.hadoop.hbase.client.Connection; +import org.apache.hadoop.hbase.client.ConnectionFactory; +import org.apache.hadoop.hbase.client.Get; +import org.apache.hadoop.hbase.client.Result; +import org.apache.hadoop.hbase.client.Table; +import org.apache.hadoop.hbase.client.TableDescriptor; +import org.apache.hadoop.hbase.client.TableDescriptorBuilder; +import org.apache.hadoop.hbase.coprocessor.ObserverContext; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor; +import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment; +import org.apache.hadoop.hbase.coprocessor.RegionObserver; +import org.apache.hadoop.hbase.io.hfile.HFile; +import org.apache.hadoop.hbase.io.hfile.HFileContext; +import org.apache.hadoop.hbase.replication.ReplicationPeerConfig; +import org.apache.hadoop.hbase.replication.TestReplicationBase; +import org.apache.hadoop.hbase.testclassification.MediumTests; +import org.apache.hadoop.hbase.testclassification.ReplicationTests; +import org.apache.hadoop.hbase.tool.BulkLoadHFilesTool; +import org.apache.hadoop.hbase.util.Bytes; +import org.apache.hadoop.hbase.util.Pair; +import org.apache.hadoop.hdfs.MiniDFSCluster; +import org.junit.After; +import org.junit.Before; +import org.junit.BeforeClass; +import org.junit.ClassRule; +import org.junit.Rule; +import org.junit.Test; +import org.junit.experimental.categories.Category; +import org.junit.rules.TemporaryFolder; +import org.junit.rules.TestName; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Integration test for bulk load replication. Defines two clusters, with two way replication. + * Performs a bulk load on cluster defined by UTIL1 first, asserts the Cell on the bulk loaded file + * gets into the related table in UTIL1, then also validates the same got replicated to cluster + * UTIL2. Then, bulk loads another file into UTIL2, and checks if related values are present on + * UTIL2, and also gets replicated to UTIL1. Review comment: Indeed. Fixing it on next commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from
Jackson2 to shaded Gson (Backport HBASE-20587)
URL: https://github.com/apache/hbase/pull/616#discussion_r323807270
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/http/jmx/JMXJsonServlet.java
##
@@ -163,14 +163,14 @@ public void doGet(HttpServletRequest request,
HttpServletResponse response) thro
try {
jsonpcb = checkCallbackName(request.getParameter(CALLBACK_PARAM));
writer = response.getWriter();
-beanWriter = this.jsonBeanWriter.open(writer);
// "callback" parameter implies JSONP outpout
if (jsonpcb != null) {
response.setContentType("application/javascript; charset=utf8");
writer.write(jsonpcb + "(");
} else {
response.setContentType("application/json; charset=utf8");
}
+beanWriter = this.jsonBeanWriter.open(writer);
Review comment:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [hbase] busbey commented on a change in pull request #616: HBASE-23015 : Moving from Jackson2 to shaded Gson (Backport HBASE-20587)
busbey commented on a change in pull request #616: HBASE-23015 : Moving from
Jackson2 to shaded Gson (Backport HBASE-20587)
URL: https://github.com/apache/hbase/pull/616#discussion_r323806868
##
File path:
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java
##
@@ -52,11 +51,9 @@
* This class is not thread safe.
*/
@InterfaceAudience.Private
-@JsonIgnoreProperties({"indexStatistics", "freeSize", "usedSize"})
Review comment:
That works for me. We should take a look at the output JSON and make sure
the release note for this calls out that folks will start getting some extra
info, since that's (IMHO) surprising for a maintenance release.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
