[jira] [Updated] (HBASE-21740) NPE happens while shutdown the RS
[
https://issues.apache.org/jira/browse/HBASE-21740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lujie updated HBASE-21740:
--
Attachment: (was: HBASE-21740_branch_2.1_1.patch)
> NPE happens while shutdown the RS
> -
>
> Key: HBASE-21740
> URL: https://issues.apache.org/jira/browse/HBASE-21740
> Project: HBase
> Issue Type: Bug
>Reporter: lujie
>Assignee: lujie
>Priority: Major
> Attachments: HBASE-21740_1.patch, HBASE-21740_2.patch,
> HBASE-21740_branch_1.4_1.patch, HBASE-21740_branch_2.1_2.patch
>
>
> while shutdown a NM, we meet a NPE:
> {code:java}
> 2019-01-18 16:52:05,500 INFO [Thread-4] regionserver.HRegionServer: STOPPED:
> Shutdown hook
> 2019-01-18 16:52:05,896 INFO [regionserver/hadoop15:16020]
> regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics
> every 5000 milliseconds
> 2019-01-18 16:52:05,978 INFO [regionserver/hadoop15:16020.Chore.1]
> hbase.ScheduledChore: Chore: CompactedHFilesCleaner was stopped
> 2019-01-18 16:52:05,996 ERROR [regionserver/hadoop15:16020]
> regionserver.HRegionServer: Failed init
> java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.startServices(HRegionServer.java:1978)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1572)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:975)
> at java.lang.Thread.run(Thread.java:745)
> 2019-01-18 16:52:06,011 ERROR [regionserver/hadoop15:16020]
> regionserver.HRegionServer: * ABORTING region server
> hadoop15,16020,1547801516426: Unhandled: Region server startup failed *
> java.io.IOException: Region server startup failed
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.convertThrowableToIOE(HRegionServer.java:3392)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1591)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:975)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.startServices(HRegionServer.java:1978)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.handleReportForDutyResponse(HRegionServer.java:1572)
> ... 2 more
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21413) Empty meta log doesn't get split when restart whole cluster
[ https://issues.apache.org/jira/browse/HBASE-21413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748434#comment-16748434 ] Hudson commented on HBASE-21413: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #520 (See [https://builds.apache.org/job/HBase-1.3-IT/520/]) HBASE-21561 Backport HBASE-21413 (Empty meta log doesn't get split when (apurtell: [https://github.com/apache/hbase/commit/3bee9d1bc9d92888fd85950e001a7767e043d0ed]) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java > Empty meta log doesn't get split when restart whole cluster > --- > > Key: HBASE-21413 > URL: https://issues.apache.org/jira/browse/HBASE-21413 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.1.1, 2.0.2 >Reporter: Jingyun Tian >Assignee: Allan Yang >Priority: Major > Fix For: 3.0.0, 2.2.0, 2.1.2, 2.0.4 > > Attachments: HBASE-21413.branch-2.1.001.patch, > HBASE-21413.branch-2.1.002.patch, Screenshot from 2018-10-31 18-11-02.png, > Screenshot from 2018-10-31 18-11-11.png > > > After I restart whole cluster, there is a splitting directory still exists on > hdfs. Then I found there is only an empty meta wal file in it. I'll dig into > this later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748433#comment-16748433 ] Hudson commented on HBASE-21561: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #520 (See [https://builds.apache.org/job/HBase-1.3-IT/520/]) HBASE-21561 Backport HBASE-21413 (Empty meta log doesn't get split when (apurtell: [https://github.com/apache/hbase/commit/3bee9d1bc9d92888fd85950e001a7767e043d0ed]) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCleanupMetaWAL.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/DefaultWALProvider.java > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10, 1.3.4 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21745) Make HBCK2 be able to fix issues other than region assignment
[
https://issues.apache.org/jira/browse/HBASE-21745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748425#comment-16748425
]
Duo Zhang commented on HBASE-21745:
---
[~stack] In general sir, HBCK2 is for fixing the broken cluster where we have
bugs. For AMv1, we know that the design itself has problems so we can see lots
of broken states in real production, and AMv2 aims to solve this and now it
seems work pretty well. But anyway, we could have bugs in code, so we haven't
seen it now does not mean it will not happen in the future...
FWIW, I think we should have the ability to fix region holes, and failed
split/merge, etc.
Thanks.
> Make HBCK2 be able to fix issues other than region assignment
> -
>
> Key: HBASE-21745
> URL: https://issues.apache.org/jira/browse/HBASE-21745
> Project: HBase
> Issue Type: Umbrella
> Components: hbase-operator-tools, hbck2
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
>
> This is what [~apurtell] posted on mailing-list, HBCK2 should support
> {quote}
>- Rebuild meta from region metadata in the filesystem, aka offline meta
>rebuild.
>- Fix assignment errors (undeployed regions, double assignments (yes,
>should not be possible), etc)
>- Fix region holes, overlaps, and other errors in the region chain
>- Fix failed split and merge transactions that have failed to roll back
>due to some bug (related to previous)
>- Enumerate store files to determine file level corruption and sideline
>corrupt files
>- Fix hfile link problems (dangling / broken)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21716) Add toStringCustomizedValues to TableDescriptor
[
https://issues.apache.org/jira/browse/HBASE-21716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748423#comment-16748423
]
Duo Zhang commented on HBASE-21716:
---
[~pingsutw]
See below
{code}
+ /**
+ * @return Name of this table and then a map of all of the column family
+ * descriptors (with only the non-default column family attributes) <
this line should have indent instead of align with the @return above.
+ */
+ String toStringCustomizedValues();
+
{code}
> Add toStringCustomizedValues to TableDescriptor
> ---
>
> Key: HBASE-21716
> URL: https://issues.apache.org/jira/browse/HBASE-21716
> Project: HBase
> Issue Type: Task
>Reporter: Duo Zhang
>Assignee: kevin su
>Priority: Major
> Labels: beginner
> Attachments: HBASE-21716.patch01, HBASE-21716.v1.patch
>
>
> HTableDescriptor.toStringCustomizedValues is used in our code base,especially
> that on the web page, we should add this method to TableDescriptor if we want
> to remove HTableDescriptor completely
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21753) Support getting the locations for all the replicas of a region
[ https://issues.apache.org/jira/browse/HBASE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748417#comment-16748417 ] Duo Zhang commented on HBASE-21753: --- Yes it is for region replication, not HDFS replication. And here we will return a HRegionLocation, which tells you where the region is, not only a RegionInfo. > Support getting the locations for all the replicas of a region > -- > > Key: HBASE-21753 > URL: https://issues.apache.org/jira/browse/HBASE-21753 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748407#comment-16748407 ] stack commented on HBASE-7191: -- [~johou] Another heuristic in hbck1 prevails, the one that considers empty dirs as corrupt or remainders from incomplete-deletes? You are trying to teach hbck1 to create the version and id files? The version should be easy enough. You could create one offline and copy it into place. On the id, I don't recall -- does it have to be just unique so any id will do or does it have to be the id that was there previous? > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Priority: Major > Attachments: 7191-2.patch, hbck1-1.4-v1.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21751) WAL creation fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748405#comment-16748405
]
Hadoop QA commented on HBASE-21751:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
39s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
49s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 49s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}129m
26s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21751 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955718/HBASE-21751v2.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 641de1d5e511 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f2820ea16f |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15672/testReport/ |
| Max. process+thread count | 4496 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15672/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> WAL creation fails
[jira] [Commented] (HBASE-21745) Make HBCK2 be able to fix issues other than region assignment
[
https://issues.apache.org/jira/browse/HBASE-21745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748404#comment-16748404
]
stack commented on HBASE-21745:
---
bq. Rebuild meta from region metadata in the filesystem, aka offline meta
rebuild.
Has anyone ever had use for this facility in recent memory? This was always a
hack dependent on possible stale schema info echo'd to the FS. It would be
blind to splits and merges. I'd actually like to remove the echoing of schema
to the FS -- we'd have too if we want to change the layout in the FS so we can
scale.
But we could do a version for hbase2 that gave some comfort in the unlikely
case we lost a whole hbase:meta.
bq. Fix assignment errors (undeployed regions, double assignments (yes, should
not be possible), etc)
I've not seen double assign in hbase2. On assign, between shell and hbck2,
there are a variety of toolings -- some that were not available in hbase1 such
as bulk assign/unassign. I think we are covered here.
Perhaps a check on the wholesomeness of hbase:meta? hbck2 has done work in the
Canary to serve this verification function. Let me take another look and see if
we need to hoist up stuff from hbck1.
bq. Fix region holes, overlaps, and other errors in the region chain
Haven't seen this in amv2. Doesn't seem to happen.
bq. Fix failed split and merge transactions that have failed to roll back due
to some bug (related to previous)
Yeah. Haven't seen this either.
bq. Enumerate store files to determine file level corruption and sideline
corrupt files
This seems to be an hfile tool variant. Let me look into adding it.
bq. Fix hfile link problems (dangling / broken)
Ditto.
Will be back.
> Make HBCK2 be able to fix issues other than region assignment
> -
>
> Key: HBASE-21745
> URL: https://issues.apache.org/jira/browse/HBASE-21745
> Project: HBase
> Issue Type: Umbrella
> Components: hbase-operator-tools, hbck2
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
>
> This is what [~apurtell] posted on mailing-list, HBCK2 should support
> {quote}
>- Rebuild meta from region metadata in the filesystem, aka offline meta
>rebuild.
>- Fix assignment errors (undeployed regions, double assignments (yes,
>should not be possible), etc)
>- Fix region holes, overlaps, and other errors in the region chain
>- Fix failed split and merge transactions that have failed to roll back
>due to some bug (related to previous)
>- Enumerate store files to determine file level corruption and sideline
>corrupt files
>- Fix hfile link problems (dangling / broken)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21716) Add toStringCustomizedValues to TableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-21716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748400#comment-16748400 ] cxorm commented on HBASE-21716: --- Thanks for sharing. > Add toStringCustomizedValues to TableDescriptor > --- > > Key: HBASE-21716 > URL: https://issues.apache.org/jira/browse/HBASE-21716 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: kevin su >Priority: Major > Labels: beginner > Attachments: HBASE-21716.patch01, HBASE-21716.v1.patch > > > HTableDescriptor.toStringCustomizedValues is used in our code base,especially > that on the web page, we should add this method to TableDescriptor if we want > to remove HTableDescriptor completely -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21745) Make HBCK2 be able to fix issues other than region assignment
[
https://issues.apache.org/jira/browse/HBASE-21745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack reassigned HBASE-21745:
-
Assignee: stack
> Make HBCK2 be able to fix issues other than region assignment
> -
>
> Key: HBASE-21745
> URL: https://issues.apache.org/jira/browse/HBASE-21745
> Project: HBase
> Issue Type: Umbrella
> Components: hbase-operator-tools, hbck2
>Reporter: Duo Zhang
>Assignee: stack
>Priority: Critical
>
> This is what [~apurtell] posted on mailing-list, HBCK2 should support
> {quote}
>- Rebuild meta from region metadata in the filesystem, aka offline meta
>rebuild.
>- Fix assignment errors (undeployed regions, double assignments (yes,
>should not be possible), etc)
>- Fix region holes, overlaps, and other errors in the region chain
>- Fix failed split and merge transactions that have failed to roll back
>due to some bug (related to previous)
>- Enumerate store files to determine file level corruption and sideline
>corrupt files
>- Fix hfile link problems (dangling / broken)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21753) Support getting the locations for all the replicas of a region
[ https://issues.apache.org/jira/browse/HBASE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748374#comment-16748374 ] Wellington Chevreuil commented on HBASE-21753: -- Hey [~beeshma], I thought this was actually regarding [region replicas|http://hbase.apache.org/book.html#arch.timelineconsistent.reads], not hdfs block replication related. [~Apache9], can you share more details? Isn't the current *org.apache.hadoop.hbase.client**Admin.getRegions(TableName)* method not solving this? I think it returns a separate *RegionInfo* for each replica already. > Support getting the locations for all the replicas of a region > -- > > Key: HBASE-21753 > URL: https://issues.apache.org/jira/browse/HBASE-21753 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21734) Some optimization in FilterListWithOR
[ https://issues.apache.org/jira/browse/HBASE-21734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21734: - Resolution: Fixed Status: Resolved (was: Patch Available) > Some optimization in FilterListWithOR > - > > Key: HBASE-21734 > URL: https://issues.apache.org/jira/browse/HBASE-21734 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5 > > Attachments: HBASE-21734.branch-1.v1.patch, HBASE-21734.v1.patch, > columnkey.txt, perf-ut.patch > > > In HBASE-21620, [~KarthickRam] and [~mohamed.meeran] complaind that their > performance of filter list has been degraded after that patch in here [1]. > I wrote a UT for this, and test under my host. It's true. I gussed there > may be two reasons: > 1. the comparator.compare(nextKV, cell) > 0 StoreScanner; > 2. the filter list concated by OR will choose the minimal forward step among > all sub-filters. in this patch, we have stricter restrictions on all sub > filters, include those sub-filter whose has non-null RC returned in > calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this > sub-filter's rc, but it's wrong in some case), and merge all of the > sub-filter's RC, this is also some time cost. > The former one seems not the main problem, because the UT still cost ~ 3s > even if I comment the compare. the second one has some impact indeed, > because after i skip to merge the sub-filters's RC if > calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, > it's improvement but the logic is not wrong. > 1. > https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21734) Some optimization in FilterListWithOR
[ https://issues.apache.org/jira/browse/HBASE-21734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748369#comment-16748369 ] Zheng Hu commented on HBASE-21734: -- Pushed to branch-1/branch-1.4/branch-2.0/branch-2.1/branch-2/master, Thanks [~psomogyi] & [~zghaobac] for reviewing. > Some optimization in FilterListWithOR > - > > Key: HBASE-21734 > URL: https://issues.apache.org/jira/browse/HBASE-21734 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21734.branch-1.v1.patch, HBASE-21734.v1.patch, > columnkey.txt, perf-ut.patch > > > In HBASE-21620, [~KarthickRam] and [~mohamed.meeran] complaind that their > performance of filter list has been degraded after that patch in here [1]. > I wrote a UT for this, and test under my host. It's true. I gussed there > may be two reasons: > 1. the comparator.compare(nextKV, cell) > 0 StoreScanner; > 2. the filter list concated by OR will choose the minimal forward step among > all sub-filters. in this patch, we have stricter restrictions on all sub > filters, include those sub-filter whose has non-null RC returned in > calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this > sub-filter's rc, but it's wrong in some case), and merge all of the > sub-filter's RC, this is also some time cost. > The former one seems not the main problem, because the UT still cost ~ 3s > even if I comment the compare. the second one has some impact indeed, > because after i skip to merge the sub-filters's RC if > calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, > it's improvement but the logic is not wrong. > 1. > https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21734) Some optimization in FilterListWithOR
[ https://issues.apache.org/jira/browse/HBASE-21734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21734: - Hadoop Flags: Reviewed Release Note: After HBASE-21620, the filterListWithOR has been a bit slow because we need to merge each sub-filter's RC , while before HBASE-21620, we will skip many RC merging, but the logic was wrong. So here we choose another way to optimaze the performance: removing the KeyValueUtil#toNewKeyCell. Anoop Sam John suggested that the KeyValueUtil#toNewKeyCell can save some GC before because if we copy key part of cell into a single byte[], then the block the cell refering won't be refered by the filter list any more, the upper layer can GC the data block quickly. while after HBASE-21620, we will update the prevCellList for every encountered cell now, so the lifecycle of cell in prevCellList for FilterList will be quite shorter. so just use the cell ref for saving cpu. BTW, we removed all the arrays streams usage in filter list, because it's also quite time-consuming in our test. > Some optimization in FilterListWithOR > - > > Key: HBASE-21734 > URL: https://issues.apache.org/jira/browse/HBASE-21734 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5 > > Attachments: HBASE-21734.branch-1.v1.patch, HBASE-21734.v1.patch, > columnkey.txt, perf-ut.patch > > > In HBASE-21620, [~KarthickRam] and [~mohamed.meeran] complaind that their > performance of filter list has been degraded after that patch in here [1]. > I wrote a UT for this, and test under my host. It's true. I gussed there > may be two reasons: > 1. the comparator.compare(nextKV, cell) > 0 StoreScanner; > 2. the filter list concated by OR will choose the minimal forward step among > all sub-filters. in this patch, we have stricter restrictions on all sub > filters, include those sub-filter whose has non-null RC returned in > calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this > sub-filter's rc, but it's wrong in some case), and merge all of the > sub-filter's RC, this is also some time cost. > The former one seems not the main problem, because the UT still cost ~ 3s > even if I comment the compare. the second one has some impact indeed, > because after i skip to merge the sub-filters's RC if > calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, > it's improvement but the logic is not wrong. > 1. > https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang resolved HBASE-14223.
Resolution: Fixed
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:50,497
> INFO [main-EventThread] wal.WALSplitter: Archived processed
[jira] [Comment Edited] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748372#comment-16748372
]
Allan Yang edited comment on HBASE-14223 at 1/22/19 4:09 AM:
-
This issue was resolved in HBASE-21413 and backported to branch-1 in
HBASE-21561. Closing this one.
was (Author: allan163):
This issue was resolved in HBASE-21413. Closing this one.
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
>
[jira] [Commented] (HBASE-14223) Meta WALs are not cleared if meta region was closed and RS aborts
[
https://issues.apache.org/jira/browse/HBASE-14223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748372#comment-16748372
]
Allan Yang commented on HBASE-14223:
This issue was resolved in HBASE-21413. Closing this one.
> Meta WALs are not cleared if meta region was closed and RS aborts
> -
>
> Key: HBASE-14223
> URL: https://issues.apache.org/jira/browse/HBASE-14223
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-14223logs, hbase-14223_v0.patch,
> hbase-14223_v1-branch-1.patch, hbase-14223_v2-branch-1.patch,
> hbase-14223_v3-branch-1.patch, hbase-14223_v3-branch-1.patch,
> hbase-14223_v3-master.patch
>
>
> When an RS opens meta, and later closes it, the WAL(FSHlog) is not closed.
> The last WAL file just sits there in the RS WAL directory. If RS stops
> gracefully, the WAL file for meta is deleted. Otherwise if RS aborts, WAL for
> meta is not cleaned. It is also not split (which is correct) since master
> determines that the RS no longer hosts meta at the time of RS abort.
> From a cluster after running ITBLL with CM, I see a lot of {{-splitting}}
> directories left uncleaned:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs
> Found 31 items
> drwxr-xr-x - hbase hadoop 0 2015-06-05 01:14
> /apps/hbase/data/WALs/hregion-58203265
> drwxr-xr-x - hbase hadoop 0 2015-06-05 07:54
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433489308745-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 09:28
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433494382959-splitting
> drwxr-xr-x - hbase hadoop 0 2015-06-05 10:01
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-1.openstacklocal,16020,1433498252205-splitting
> ...
> {code}
> The directories contain WALs from meta:
> {code}
> [root@os-enis-dal-test-jun-4-7 cluster-os]# sudo -u hdfs hadoop fs -ls
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting
> Found 2 items
> -rw-r--r-- 3 hbase hadoop 201608 2015-06-05 03:15
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> -rw-r--r-- 3 hbase hadoop 44420 2015-06-05 04:36
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> The RS hosted the meta region for some time:
> {code}
> 2015-06-05 03:14:28,692 INFO [PostOpenDeployTasks:1588230740]
> zookeeper.MetaTableLocator: Setting hbase:meta region location in ZooKeeper
> as os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285
> ...
> 2015-06-05 03:15:17,302 INFO
> [RS_CLOSE_META-os-enis-dal-test-jun-4-5:16020-0] regionserver.HRegion: Closed
> hbase:meta,,1.1588230740
> {code}
> In between, a WAL is created:
> {code}
> 2015-06-05 03:15:11,707 INFO
> [RS_OPEN_META-os-enis-dal-test-jun-4-5:16020-0-MetaLogRoller] wal.FSHLog:
> Rolled WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433470511501.meta
> with entries=385, filesize=196.88 KB; new WAL
> /apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285..meta.1433474111645.meta
> {code}
> When CM killed the region server later master did not see these WAL files:
> {code}
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:46,075
> INFO [MASTER_SERVER_OPERATIONS-os-enis-dal-test-jun-4-3:16000-0]
> master.SplitLogManager: started splitting 2 logs in
> [hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting]
> for [os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285]
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05 03:36:47,300
> INFO [main-EventThread] wal.WALSplitter: Archived processed log
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/WALs/os-enis-dal-test-jun-4-5.openstacklocal,16020,1433466904285-splitting/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> to
> hdfs://os-enis-dal-test-jun-4-1.openstacklocal:8020/apps/hbase/data/oldWALs/os-enis-dal-test-jun-4-5.openstacklocal%2C16020%2C1433466904285.default.1433475074436
> ./hbase-hbase-master-os-enis-dal-test-jun-4-3.log:2015-06-05
[jira] [Updated] (HBASE-21734) Some optimization in FilterListWithOR
[ https://issues.apache.org/jira/browse/HBASE-21734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21734: - Fix Version/s: 2.0.5 2.1.3 1.4.10 2.2.0 1.5.0 3.0.0 > Some optimization in FilterListWithOR > - > > Key: HBASE-21734 > URL: https://issues.apache.org/jira/browse/HBASE-21734 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 1.5.0, 2.2.0, 1.4.10, 2.1.3, 2.0.5 > > Attachments: HBASE-21734.branch-1.v1.patch, HBASE-21734.v1.patch, > columnkey.txt, perf-ut.patch > > > In HBASE-21620, [~KarthickRam] and [~mohamed.meeran] complaind that their > performance of filter list has been degraded after that patch in here [1]. > I wrote a UT for this, and test under my host. It's true. I gussed there > may be two reasons: > 1. the comparator.compare(nextKV, cell) > 0 StoreScanner; > 2. the filter list concated by OR will choose the minimal forward step among > all sub-filters. in this patch, we have stricter restrictions on all sub > filters, include those sub-filter whose has non-null RC returned in > calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this > sub-filter's rc, but it's wrong in some case), and merge all of the > sub-filter's RC, this is also some time cost. > The former one seems not the main problem, because the UT still cost ~ 3s > even if I comment the compare. the second one has some impact indeed, > because after i skip to merge the sub-filters's RC if > calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, > it's improvement but the logic is not wrong. > 1. > https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748365#comment-16748365
]
Hadoop QA commented on HBASE-21561:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
28s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 1 new + 145
unchanged - 0 fixed = 146 total (was 145) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 36s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}106m
0s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}123m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955717/HBASE-21561.branch-1.002.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 425bf35c1af9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64
[jira] [Commented] (HBASE-21753) Support getting the locations for all the replicas of a region
[ https://issues.apache.org/jira/browse/HBASE-21753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748366#comment-16748366 ] beeshma commented on HBASE-21753: - HI [~Apache9] , you mean information's of Region server(Data-node) which have the particular Region ? For Example ( input => Region R) if dfs.replicaion =3 at hdfs-site.xml, then three Region server information's output > Support getting the locations for all the replicas of a region > -- > > Key: HBASE-21753 > URL: https://issues.apache.org/jira/browse/HBASE-21753 > Project: HBase > Issue Type: New Feature > Components: Client >Reporter: Duo Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748364#comment-16748364 ] Allan Yang commented on HBASE-21561: Yes, if it is as simple as adding a API, just include it in this patch. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10, 1.3.4 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21751) WAL creation fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-21751:
---
Summary: WAL creation fails during region open may cause region assign
forever fail (was: WAL create fails during region open may cause region assign
forever fail)
> WAL creation fails during region open may cause region assign forever fail
> --
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch, HBASE-21751v2.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[
https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748353#comment-16748353
]
Zheng Hu commented on HBASE-21750:
--
Well, It's a mistake which was introduced in HBASE-21657, for KeyOnlyCell,
the serializedSize should be (NOT the cell.getSerializedSize())
{code}
KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE + keyLen + getValueLength();
{code}
> Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for
> better performance because the latter one has been optimized
> --
>
> Key: HBASE-21750
> URL: https://issues.apache.org/jira/browse/HBASE-21750
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21750.v1.patch, HBASE-21750.v1.patch,
> HBASE-21750.v2.patch
>
>
> After HBASE-21657, Most subclass of Cell has a cached serialized size (Except
> those cells with tags), so I think most of the KeyValueUtil#length can be
> replaced by cell#getSerializedSize. Such as:
> - KeyValueUtil.length in StoreFlusher#performFlush;
> - KeyValueUtil.length in Compactor#performCompaction ;
> and so on..
> Will prepare a patch for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21750: - Attachment: HBASE-21750.v3.patch > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch, HBASE-21750.v1.patch, > HBASE-21750.v2.patch, HBASE-21750.v3.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-7191) HBCK - Add offline create/fix hbase.version and hbase.id
[ https://issues.apache.org/jira/browse/HBASE-7191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748354#comment-16748354 ] xufeng commented on HBASE-7191: --- Do we maintain the branch-1.* ？ in fact, I am writing the handbook of maintenance for the maintenance team。 The hbck tool is very important so I dig it 。but also found others problem. for example: 1. create the new table with some empty regions. 2. delete the .regioninfo file in one region folder(I want to make the INCONSISTENT and to fix it) 3. hbase hbck -fixHdfsOrphans result：the region folder will be sidelined because no data。 > HBCK - Add offline create/fix hbase.version and hbase.id > - > > Key: HBASE-7191 > URL: https://issues.apache.org/jira/browse/HBASE-7191 > Project: HBase > Issue Type: Improvement > Components: hbck >Affects Versions: 0.94.1 >Reporter: Enis Soztutar >Priority: Major > Attachments: 7191-2.patch, hbck1-1.4-v1.patch > > > One of our clients run into a problem, in which they have the hbase.root.dir, > and cluster data, but their hbase.id and hbase.version files are corrupted. > HMaster creates those on start, but not if there is already existing data. > We can add smt like --fixIdFile, and ability for HBCK to do some offline > repairs for the version file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21750: - Attachment: HBASE-21750.v2.patch > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch, HBASE-21750.v1.patch, > HBASE-21750.v2.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Allan Yang updated HBASE-21751:
---
Attachment: HBASE-21751v2.patch
> WAL create fails during region open may cause region assign forever fail
>
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch, HBASE-21751v2.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748347#comment-16748347
]
Allan Yang commented on HBASE-21751:
{quote}
But I suppose that any time when we fail to roll a wal writer we will abort the
RS? No? \
{quote}
Aborting RS if roll writer fail is to prevent data loss. But in this case, we
haven't create a WAL yet, no data is written, so aborting RS is not necessary.
I uploaded a v2 patch, recover lease before getLen(), how about this one?
> WAL create fails during region open may cause region assign forever fail
>
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch, HBASE-21751v2.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[
https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748346#comment-16748346
]
Zheng Hu commented on HBASE-21750:
--
The failed ut is indeed a problem, let me fix this.
{code}
java.lang.IllegalArgumentException: Some redundant bytes in KeyValue's buffer,
startOffset=33, endOffset=36,
KeyValueBytesHex=\x00\x00\x00\x17\x00\x00\x00\x00\x00\x03row\x03famqual1\x7F\xFF\xFF\xFF\xFF\xFF\xFF\xFF\x04\x00\x00\x00\x00\x00,
offset=0, length=36
at
org.apache.hadoop.hbase.KeyValueUtil.checkKeyValueBytes(KeyValueUtil.java:635)
at org.apache.hadoop.hbase.KeyValue.(KeyValue.java:344)
at
org.apache.hadoop.hbase.KeyValueUtil.copyToNewKeyValue(KeyValueUtil.java:94)
at
org.apache.hadoop.hbase.KeyValueUtil.ensureKeyValue(KeyValueUtil.java:478)
at
org.apache.hadoop.hbase.filter.TestFilterList.testTransformMPO(TestFilterList.java:582)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
{code}
> Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for
> better performance because the latter one has been optimized
> --
>
> Key: HBASE-21750
> URL: https://issues.apache.org/jira/browse/HBASE-21750
> Project: HBase
> Issue Type: Sub-task
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Major
> Fix For: 3.0.0, 2.2.0
>
> Attachments: HBASE-21750.v1.patch, HBASE-21750.v1.patch
>
>
> After HBASE-21657, Most subclass of Cell has a cached serialized size (Except
> those cells with tags), so I think most of the KeyValueUtil#length can be
> replaced by cell#getSerializedSize. Such as:
> - KeyValueUtil.length in StoreFlusher#performFlush;
> - KeyValueUtil.length in Compactor#performCompaction ;
> and so on..
> Will prepare a patch for this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20952) Re-visit the WAL API
[
https://issues.apache.org/jira/browse/HBASE-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748342#comment-16748342
]
Hudson commented on HBASE-20952:
Results for branch HBASE-20952
[build #65 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/65/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/65//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/65//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20952/65//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Re-visit the WAL API
>
>
> Key: HBASE-20952
> URL: https://issues.apache.org/jira/browse/HBASE-20952
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Reporter: Josh Elser
>Priority: Major
> Attachments: 20952.v1.txt
>
>
> Take a step back from the current WAL implementations and think about what an
> HBase WAL API should look like. What are the primitive calls that we require
> to guarantee durability of writes with a high degree of performance?
> The API needs to take the current implementations into consideration. We
> should also have a mind for what is happening in the Ratis LogService (but
> the LogService should not dictate what HBase's WAL API looks like RATIS-272).
> Other "systems" inside of HBase that use WALs are replication and
> backup Replication has the use-case for "tail"'ing the WAL which we
> should provide via our new API. B doesn't do anything fancy (IIRC). We
> should make sure all consumers are generally going to be OK with the API we
> create.
> The API may be "OK" (or OK in a part). We need to also consider other methods
> which were "bolted" on such as {{AbstractFSWAL}} and
> {{WALFileLengthProvider}}. Other corners of "WAL use" (like the
> {{WALSplitter}} should also be looked at to use WAL-APIs only).
> We also need to make sure that adequate interface audience and stability
> annotations are chosen.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21561: --- Fix Version/s: 1.3.4 > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10, 1.3.4 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21561: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[
https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748339#comment-16748339
]
Hadoop QA commented on HBASE-21750:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
4s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} hbase-common: The patch generated 0 new + 27
unchanged - 1 fixed = 27 total (was 28) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} hbase-client: The patch generated 1 new + 50 unchanged
- 0 fixed = 51 total (was 50) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
19s{color} | {color:red} hbase-server: The patch generated 12 new + 311
unchanged - 1 fixed = 323 total (was 312) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
30s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 31s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
3s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 11s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 85m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.filter.TestFilterList |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue |
[jira] [Created] (HBASE-21753) Support getting the locations for all the replicas of a region
Duo Zhang created HBASE-21753: - Summary: Support getting the locations for all the replicas of a region Key: HBASE-21753 URL: https://issues.apache.org/jira/browse/HBASE-21753 Project: HBase Issue Type: New Feature Components: Client Reporter: Duo Zhang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21729) Extract ProcedureCoordinatorRpcs and ProcedureMemberRpcs from CoordinatedStateManager
[ https://issues.apache.org/jira/browse/HBASE-21729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748327#comment-16748327 ] Jingyun Tian commented on HBASE-21729: -- [~zghaobac] Could you help check this out? This patch could let us remove CoordinatedStateManager in the future. > Extract ProcedureCoordinatorRpcs and ProcedureMemberRpcs from > CoordinatedStateManager > - > > Key: HBASE-21729 > URL: https://issues.apache.org/jira/browse/HBASE-21729 > Project: HBase > Issue Type: Sub-task >Reporter: Jingyun Tian >Assignee: Jingyun Tian >Priority: Major > Attachments: HBASE-21729.master.001.patch, > HBASE-21729.master.002.patch > > > If procedureV2 based WAL splitting is enabled, CoordinatedStateManager will > not be initialized. Then ProcedureCoordinatorRpcs and ProcedureMemberRpcs > will make backup not work. Let me extract these two method to another class. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748326#comment-16748326 ] Andrew Purtell edited comment on HBASE-21561 at 1/22/19 2:05 AM: - I think Allan might have been saying it's fine to add getProcedures() as part of this change, but the followup jira is fine too IMHO. I like that the API change has its own issue. We can do that and improve the test added here on HBASE-21752 . was (Author: apurtell): I think Allan might have been saying it's fine to add getProcedures() as part of this change, but the followup jira is fine too IMHO. I like that the API change has its own issue. We can do that and improve the test on HBASE-21752 . > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748326#comment-16748326 ] Andrew Purtell commented on HBASE-21561: I think Allan might have been saying it's fine to add getProcedures() as part of this change, but the followup jira is fine too IMHO. I like that the API change has its own issue. We can do that and improve the test on HBASE-21752 . > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748324#comment-16748324 ] Andrew Purtell commented on HBASE-21561: I think the test change is fine [~xucang]. Patch looks good, will commit after some local checks > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21752) Backport getProcedures() to branch-1 from branch-2 in HMaster class
[ https://issues.apache.org/jira/browse/HBASE-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21752: --- Fix Version/s: 1.5.0 > Backport getProcedures() to branch-1 from branch-2 in HMaster class > --- > > Key: HBASE-21752 > URL: https://issues.apache.org/jira/browse/HBASE-21752 > Project: HBase > Issue Type: Improvement > Environment: Backport getProcedures() to branch-1 from branch-2 in > HMaster class >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21680: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21749) RS UI may throw NPE and make rs-status page inaccessible with multiwal and replication
[
https://issues.apache.org/jira/browse/HBASE-21749?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-21749:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.1.3
2.2.0
1.5.0
3.0.0
Status: Resolved (was: Patch Available)
> RS UI may throw NPE and make rs-status page inaccessible with multiwal and
> replication
> --
>
> Key: HBASE-21749
> URL: https://issues.apache.org/jira/browse/HBASE-21749
> Project: HBase
> Issue Type: Bug
> Components: Replication, UI
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Fix For: 3.0.0, 1.5.0, 2.2.0, 2.1.3
>
> Attachments: HBASE-21749.master.001.patch
>
>
> Sometimes RS UI is unable to open as we get a NPE; This happens because
> {{shipper.getCurrentPath()}} may return null.
> We should have a null check @
> [ReplicationSource.java#L331|https://github.com/apache/hbase/blob/a2f6768acdc30b789c7cb8482b9f4352803f60a1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L331]
>
> {code:java}
> Path currentPath = shipper.getCurrentPath();
> try {
> fileSize = getFileSize(currentPath);
> } catch (IOException e) {
> LOG.warn("Ignore the exception as the file size of HLog only affects
> the web ui", e);
> fileSize = -1;
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[
https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748319#comment-16748319
]
Hadoop QA commented on HBASE-21720:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
36s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} hbase-client: The patch generated 15 new + 8 unchanged
- 39 fixed = 23 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
35s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 44s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}133m
29s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}184m 23s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21720 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955704/HBASE-21720.master.004.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 1e50fa719be3 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 35df6147ee |
| maven | version: Apache Maven 3.5.4
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748317#comment-16748317 ] Xu Cang commented on HBASE-21561: - Thanks [~allan163] I think it's the APIs in coprocessors were not backported. I will see what I can do. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21752) Backport getProcedures() to branch-1 from branch-2 in HMaster class
[ https://issues.apache.org/jira/browse/HBASE-21752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang reassigned HBASE-21752: --- Assignee: Xu Cang > Backport getProcedures() to branch-1 from branch-2 in HMaster class > --- > > Key: HBASE-21752 > URL: https://issues.apache.org/jira/browse/HBASE-21752 > Project: HBase > Issue Type: Improvement > Environment: Backport getProcedures() to branch-1 from branch-2 in > HMaster class >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21752) Backport getProcedures() to branch-1 from branch-2 in HMaster class
Xu Cang created HBASE-21752: --- Summary: Backport getProcedures() to branch-1 from branch-2 in HMaster class Key: HBASE-21752 URL: https://issues.apache.org/jira/browse/HBASE-21752 Project: HBase Issue Type: Improvement Environment: Backport getProcedures() to branch-1 from branch-2 in HMaster class Reporter: Xu Cang -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748316#comment-16748316 ] Xu Cang commented on HBASE-21561: - uploaded patch .001 to fix checkstyle errors. The unit test failure is not related, it ran fine locally. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748316#comment-16748316 ] Xu Cang edited comment on HBASE-21561 at 1/22/19 1:42 AM: -- uploaded patch .002 to fix checkstyle errors. The unit test failure is not related, it ran fine locally. was (Author: xucang): uploaded patch .001 to fix checkstyle errors. The unit test failure is not related, it ran fine locally. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-21561: Attachment: HBASE-21561.branch-1.002.patch > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch-1.002.patch, HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748315#comment-16748315
]
Allan Yang commented on HBASE-21751:
Not multi WAL, when getting the WAL first time, a new wal will be created by
calling rollWriter(), it is a sync call, it can fail due to HDFS glitches. In
some cases, it will leave a wal with size 0. But since the rollWriter is called
in the constructor, the WAL object is not created successfully, so next time
calling getWAL(), the wal need to be created again, but it will check the dir
to see if a WAL already exists, causing the error stack I pasted above,
[~Apache9].
> WAL create fails during region open may cause region assign forever fail
>
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748309#comment-16748309
]
Allan Yang commented on HBASE-21751:
{quote}
Be careful that the len you get from NN may not be the real length if the file
is not closed yet...
{quote}
[~Apache9],Yes, I know, I haven't come out a better solution for this case.
Since rollWriter is called during wal creation, and roll writer can fail... For
normal rollWriter, we will abort the RS. But I don't want to abort the server
when assigning...
> WAL create fails during region open may cause region assign forever fail
>
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748311#comment-16748311
]
Duo Zhang commented on HBASE-21751:
---
But I suppose that any time when we fail to roll a wal writer we will abort the
RS? No? What do you mean by wal creation? You are using multi WAL?
> WAL create fails during region open may cause region assign forever fail
>
>
> Key: HBASE-21751
> URL: https://issues.apache.org/jira/browse/HBASE-21751
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.1.2, 2.0.4
>Reporter: Allan Yang
>Assignee: Allan Yang
>Priority: Major
> Fix For: 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21751.patch
>
>
> During the first region opens on the RS, WALFactory will create a WAL file,
> but if the wal creation fails, in some cases, HDFS will leave a empty file in
> the dir(e.g. disk full, file is created succesfully but block allocation
> fails). We have a check in AbstractFSWAL that if WAL belong to the same
> factory exists, then a error will be throw. Thus, the region can never be
> open on this RS later.
> {code:java}
> 2019-01-17 02:15:53,320 ERROR [RS_OPEN_META-regionserver/server003:16020-0]
> handler.OpenRegionHandler(301): Failed open of region=hbase:meta,,1.1588230740
> java.io.IOException: Target WAL already exists within directory
> hdfs://cluster/hbase/WALs/server003.hbase.hostname.com,16020,1545269815888
> at
> org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.(AbstractFSWAL.java:382)
> at
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.(AsyncFSWAL.java:210)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:72)
> at
> org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:47)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
> at
> org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
> at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:264)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2085)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:284)
> at
> org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1147)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
> at java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748312#comment-16748312 ] Allan Yang commented on HBASE-21561: [~xucang] If it is just a getProcedures() method not exposed, you can add it easily, +1 for it. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-21750: -- Attachment: HBASE-21750.v1.patch > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch, HBASE-21750.v1.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748303#comment-16748303
]
Hadoop QA commented on HBASE-21561:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 20m
5s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
5s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
58s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
38s{color} | {color:red} hbase-server: The patch generated 5 new + 145
unchanged - 0 fixed = 150 total (was 145) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
9s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
2m 5s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}145m 27s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}187m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955699/HBASE-21561.branch-1.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname |
[jira] [Commented] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748300#comment-16748300 ] stack commented on HBASE-21750: --- +1 up on rb. Excellent. Let me resubmit. > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748299#comment-16748299 ] Abhishek Singh Chouhan commented on HBASE-21680: Sounds good. Thanks! > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21593) closing flags show be set false in HRegion
[
https://issues.apache.org/jira/browse/HBASE-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748295#comment-16748295
]
Hadoop QA commented on HBASE-21593:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 14m
27s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
31s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} branch-1 passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 33s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed with JDK v1.8.0_201 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}105m
51s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}138m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21593 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955703/HBASE-21593.branch-1.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux
[jira] [Commented] (HBASE-21585) Use ConnectionImplementation instead of ClusterConnection for sync client implementation
[ https://issues.apache.org/jira/browse/HBASE-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748283#comment-16748283 ] stack commented on HBASE-21585: --- Oh, there is a comment in plan asking for update on plan. Seems like you put some of the plan in the above.. .Could hoist it up into the parent. Thanks. > Use ConnectionImplementation instead of ClusterConnection for sync client > implementation > > > Key: HBASE-21585 > URL: https://issues.apache.org/jira/browse/HBASE-21585 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21585-HBASE-21512-v1.patch, > HBASE-21585-HBASE-21512-v2.patch, HBASE-21585-HBASE-21512.patch > > > So we can remove lots of method declarations in ClusterConnection, if they > are only used by sync client implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21585) Use ConnectionImplementation instead of ClusterConnection for sync client implementation
[ https://issues.apache.org/jira/browse/HBASE-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748281#comment-16748281 ] stack commented on HBASE-21585: --- bq. Maybe we should add default implementation to these two methods that throws UnsupportedOperation exception? Yes. For those who might have Connection impls already. bq. and there maybe no other stack trace deep in side the hbase code, as all things are event-driven now nod. Makes sense. Any chance of including the server and region we're going against when we throw an exception? bq. RegionLocations is IA.Private Change for hbase3? Thanks. > Use ConnectionImplementation instead of ClusterConnection for sync client > implementation > > > Key: HBASE-21585 > URL: https://issues.apache.org/jira/browse/HBASE-21585 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Attachments: HBASE-21585-HBASE-21512-v1.patch, > HBASE-21585-HBASE-21512-v2.patch, HBASE-21585-HBASE-21512.patch > > > So we can remove lots of method declarations in ClusterConnection, if they > are only used by sync client implementation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748278#comment-16748278 ] stack edited comment on HBASE-21512 at 1/21/19 11:59 PM: - Any chance of an update on where this project is at now [~Apache9] given your recent work? A high-level will help when doing the lower-level reviews. For instance, ClusterConnection has almost nothing in it now. Should it continue? Or intent is it is server-side only Thanks. was (Author: stack): Any chance of an update on where this project is at now [~Apache9] given your recent work? A high-level will help when doing the lower-level reviews. For instance, ClusterConnection has almost nothing in it now. Should it continue? Thanks. > Introduce an AsyncClusterConnection and replace the usage of ClusterConnection > -- > > Key: HBASE-21512 > URL: https://issues.apache.org/jira/browse/HBASE-21512 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > > At least for the RSProcedureDispatcher, with CompletableFuture we do not need > to set a delay and use a thread pool any more, which could reduce the > resource usage and also the latency. > Once this is done, I think we can remove the ClusterConnection completely, > and start to rewrite the old sync client based on the async client, which > could reduce the code base a lot for our client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748278#comment-16748278 ] stack edited comment on HBASE-21512 at 1/21/19 11:57 PM: - Any chance of an update on where this project is at now [~Apache9] given your recent work? A high-level will help when doing the lower-level reviews. For instance, ClusterConnection has almost nothing in it now. Should it continue? Thanks. was (Author: stack): Any chance of an update on where this project is at now [~Apache9] given your recent work? A high-level will help when doing the lower-level reviews. Thanks. > Introduce an AsyncClusterConnection and replace the usage of ClusterConnection > -- > > Key: HBASE-21512 > URL: https://issues.apache.org/jira/browse/HBASE-21512 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > > At least for the RSProcedureDispatcher, with CompletableFuture we do not need > to set a delay and use a thread pool any more, which could reduce the > resource usage and also the latency. > Once this is done, I think we can remove the ClusterConnection completely, > and start to rewrite the old sync client based on the async client, which > could reduce the code base a lot for our client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[ https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748278#comment-16748278 ] stack commented on HBASE-21512: --- Any chance of an update on where this project is at now [~Apache9] given your recent work? A high-level will help when doing the lower-level reviews. Thanks. > Introduce an AsyncClusterConnection and replace the usage of ClusterConnection > -- > > Key: HBASE-21512 > URL: https://issues.apache.org/jira/browse/HBASE-21512 > Project: HBase > Issue Type: Umbrella >Reporter: Duo Zhang >Priority: Major > Fix For: 3.0.0 > > > At least for the RSProcedureDispatcher, with CompletableFuture we do not need > to set a delay and use a thread pool any more, which could reduce the > resource usage and also the latency. > Once this is done, I think we can remove the ClusterConnection completely, > and start to rewrite the old sync client based on the async client, which > could reduce the code base a lot for our client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21576) master should proactively reassign meta when killing a RS with it
[
https://issues.apache.org/jira/browse/HBASE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748269#comment-16748269
]
stack commented on HBASE-21576:
---
Yeah, would be interested in why the RS was slow to go down given it is trying
to abort (Aborting, RS's skip everything but the vitals. They do not try to
tidy up regions). Why the slow death and the rectification by Master was also
slow seems to be the issue here (As per [~xucang], meta should get assigned on
RS exit. Meta is already favored ahead of all other Regions). Thanks.
> master should proactively reassign meta when killing a RS with it
> -
>
> Key: HBASE-21576
> URL: https://issues.apache.org/jira/browse/HBASE-21576
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Master has killed an RS that was hosting meta due to some HDFS issue (most
> likely; I've lost the RS logs due to HBASE-21575).
> RS took a very long time to die (again, might be a separate bug, I'll file if
> I see repro), and a long time to restart; meanwhile master never tried to
> reassign meta, and eventually killed itself not being able to update it.
> It seems like a RS on a bad machine would be especially prone to slow
> abort/startup, as well as to issues causing master to kill it, so it would
> make sense for master to immediately relocate meta once meta-hosting RS is
> dead after a kill; or even when killing the RS. In the former case (if the RS
> needs to die for meta to be reassigned safely), perhaps the RS hosting meta
> in particular should try to die fast in such circumstances, and not do any
> cleanup.
> {noformat}
> 2018-12-08 04:52:55,144 WARN
> [RpcServer.default.FPBQ.Fifo.handler=39,queue=4,port=17000]
> master.MasterRpcServices: ,17020,1544264858183 reported a fatal
> error:
> * ABORTING region server ,17020,1544264858183: Replay of WAL
> required. Forcing server shutdown *
> [aborting for ~7 minutes]
> 2018-12-08 04:53:44,190 INFO [PEWorker-7] client.RpcRetryingCallerImpl: Call
> exception, tries=6, retries=61, started=41190 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
> ,17020,1544264858183 aborting, details=row '...' on table
> 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=,17020,1544264858183, seqNum=-1
> ... [starting for ~5]
> 2018-12-08 04:59:58,574 INFO
> [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=17000]
> client.RpcRetryingCallerImpl: Call exception, tries=10, retries=61,
> started=392702 ms ago, cancelled=false, msg=Call to failed on
> connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: , details=row '...' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=,17020,1544264858183,
> seqNum=-1
> ... [re-initializing for at least ~7]
> 2018-12-08 05:04:17,271 INFO [hconnection-0x4d58bcd4-shared-pool3-t1877]
> client.RpcRetryingCallerImpl: Call exception, tries=6, retries=61,
> started=41137 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server
> ,17020,1544274145387 is not running yet
> ...
> 2018-12-08 05:11:18,470 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=38,queue=3,port=17000] master.HMaster:
> * ABORTING master ...,17000,1544230401860: FAILED persisting region=...
> state=OPEN *^M
> {noformat}
> There are no signs of meta assignment activity at all in master logs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748267#comment-16748267 ] Andrew Purtell commented on HBASE-21680: I'm planning on committing HBASE-21749 to branch-1 after application of this change [~abhishek.chouhan]. > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748268#comment-16748268 ] Andrew Purtell commented on HBASE-21680: So yes it will be fixed on all relevant branches > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21576) master should proactively reassign meta when killing a RS with it
[
https://issues.apache.org/jira/browse/HBASE-21576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748257#comment-16748257
]
Xu Cang commented on HBASE-21576:
-
just providing my limited knowledge about this here, please correct me if I
miss something.
When a RS is dead (its zk ephemeral node deleted), it will be added to dead
servers' list. Then ServerCrashProcedure will kick in to do housekeeping work
such as moving regions to other servers, which will handle reassigning meta.
So the question here is, why was your RS not added into dead servers list.
> master should proactively reassign meta when killing a RS with it
> -
>
> Key: HBASE-21576
> URL: https://issues.apache.org/jira/browse/HBASE-21576
> Project: HBase
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Master has killed an RS that was hosting meta due to some HDFS issue (most
> likely; I've lost the RS logs due to HBASE-21575).
> RS took a very long time to die (again, might be a separate bug, I'll file if
> I see repro), and a long time to restart; meanwhile master never tried to
> reassign meta, and eventually killed itself not being able to update it.
> It seems like a RS on a bad machine would be especially prone to slow
> abort/startup, as well as to issues causing master to kill it, so it would
> make sense for master to immediately relocate meta once meta-hosting RS is
> dead after a kill; or even when killing the RS. In the former case (if the RS
> needs to die for meta to be reassigned safely), perhaps the RS hosting meta
> in particular should try to die fast in such circumstances, and not do any
> cleanup.
> {noformat}
> 2018-12-08 04:52:55,144 WARN
> [RpcServer.default.FPBQ.Fifo.handler=39,queue=4,port=17000]
> master.MasterRpcServices: ,17020,1544264858183 reported a fatal
> error:
> * ABORTING region server ,17020,1544264858183: Replay of WAL
> required. Forcing server shutdown *
> [aborting for ~7 minutes]
> 2018-12-08 04:53:44,190 INFO [PEWorker-7] client.RpcRetryingCallerImpl: Call
> exception, tries=6, retries=61, started=41190 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
> ,17020,1544264858183 aborting, details=row '...' on table
> 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=,17020,1544264858183, seqNum=-1
> ... [starting for ~5]
> 2018-12-08 04:59:58,574 INFO
> [RpcServer.default.FPBQ.Fifo.handler=45,queue=0,port=17000]
> client.RpcRetryingCallerImpl: Call exception, tries=10, retries=61,
> started=392702 ms ago, cancelled=false, msg=Call to failed on
> connection exception:
> org.apache.hbase.thirdparty.io.netty.channel.ConnectTimeoutException:
> connection timed out: , details=row '...' on table 'hbase:meta' at
> region=hbase:meta,,1.1588230740, hostname=,17020,1544264858183,
> seqNum=-1
> ... [re-initializing for at least ~7]
> 2018-12-08 05:04:17,271 INFO [hconnection-0x4d58bcd4-shared-pool3-t1877]
> client.RpcRetryingCallerImpl: Call exception, tries=6, retries=61,
> started=41137 ms ago, cancelled=false,
> msg=org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server
> ,17020,1544274145387 is not running yet
> ...
> 2018-12-08 05:11:18,470 ERROR
> [RpcServer.default.FPBQ.Fifo.handler=38,queue=3,port=17000] master.HMaster:
> * ABORTING master ...,17000,1544230401860: FAILED persisting region=...
> state=OPEN *^M
> {noformat}
> There are no signs of meta assignment activity at all in master logs
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21720) metric to measure how actions are distributed to servers within a MultiAction
[ https://issues.apache.org/jira/browse/HBASE-21720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tommy Li updated HBASE-21720: - Attachment: HBASE-21720.master.004.patch > metric to measure how actions are distributed to servers within a MultiAction > - > > Key: HBASE-21720 > URL: https://issues.apache.org/jira/browse/HBASE-21720 > Project: HBase > Issue Type: Improvement > Components: Client, metrics, monitoring >Reporter: Tommy Li >Assignee: Tommy Li >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-21720.master.001.patch, > HBASE-21720.master.002.patch, HBASE-21720.master.003.patch, > HBASE-21720.master.004.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21593) closing flags show be set false in HRegion
[
https://issues.apache.org/jira/browse/HBASE-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang updated HBASE-21593:
Attachment: HBASE-21593.branch-1.001.patch
Status: Patch Available (was: Open)
> closing flags show be set false in HRegion
> --
>
> Key: HBASE-21593
> URL: https://issues.apache.org/jira/browse/HBASE-21593
> Project: HBase
> Issue Type: Bug
>Reporter: xiaolerzheng
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-21593.branch-1.001.patch,
> image-2018-12-13-16-04-51-892.png, image-2018-12-13-16-05-09-246.png,
> image-2018-12-13-16-05-36-404.png
>
>
> in HRegion.java
>
>
> 1429 // block waiting for the lock for closing
> 1430 lock.writeLock().lock();
> 1431 this.closing.set(true);
> 1432 status.setStatus("Disabling writes for close");
>
>
>
>
> 1557 } finally {
> {color:#FF} //should here add {color}
> {color:#FF} this.closing.set(false); {color}
> 1558 lock.writeLock().unlock();
> 1559 }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HBASE-21593) closing flags show be set false in HRegion
[
https://issues.apache.org/jira/browse/HBASE-21593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xu Cang reassigned HBASE-21593:
---
Assignee: Xu Cang
> closing flags show be set false in HRegion
> --
>
> Key: HBASE-21593
> URL: https://issues.apache.org/jira/browse/HBASE-21593
> Project: HBase
> Issue Type: Bug
>Reporter: xiaolerzheng
>Assignee: Xu Cang
>Priority: Minor
> Attachments: HBASE-21593.branch-1.001.patch,
> image-2018-12-13-16-04-51-892.png, image-2018-12-13-16-05-09-246.png,
> image-2018-12-13-16-05-36-404.png
>
>
> in HRegion.java
>
>
> 1429 // block waiting for the lock for closing
> 1430 lock.writeLock().lock();
> 1431 this.closing.set(true);
> 1432 status.setStatus("Disabling writes for close");
>
>
>
>
> 1557 } finally {
> {color:#FF} //should here add {color}
> {color:#FF} this.closing.set(false); {color}
> 1558 lock.writeLock().unlock();
> 1559 }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748243#comment-16748243
]
Hadoop QA commented on HBASE-21680:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} branch-1 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
48s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} branch-1 passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} branch-1 passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s{color}
| {color:red} hbase-server-jdk1.7.0_201 with JDK v1.7.0_201 generated 2 new + 4
unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
42s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
1m 37s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} the patch passed with JDK v1.8.0_192 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.7.0_201 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 55s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}128m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestAssignmentManagerOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:61288f8 |
| JIRA Issue | HBASE-21680 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955691/HBASE-21680-branch-1.patch
|
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-21561: Attachment: HBASE-21561.branch-1.001.patch > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch-1.001.patch, > HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748240#comment-16748240
]
Hadoop QA commented on HBASE-21561:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HBASE-21561 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955698/HBASE-21561.branch1.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15666/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Backport HBASE-21413 (Empty meta log doesn't get split when restart whole
> cluster) to branch-1
> --
>
> Key: HBASE-21561
> URL: https://issues.apache.org/jira/browse/HBASE-21561
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 1.5.0, 1.4.10
>
> Attachments: HBASE-21561.branch1.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748239#comment-16748239 ] Xu Cang commented on HBASE-21561: - One thing I have to mention is, during the backporting, I found "getProcedures()" is not implemented on branch-1. I traced the history about this on branch-1 a bit and found that was introduced with ProcedureV2. If I understand correctly, ProcedureV2 was backported to branch1. Is there any reason we don't have this method backported? [~apurtell][~allan163] So in my patch for the unit test, I use procedure's name to identify SCP instead of class type. It's less optimal but acceptable for me as for now. Thanks. > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch1-1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-21561: Attachment: (was: HBASE-21561.branch1-1.001.patch) > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-21561: Attachment: HBASE-21561.branch1.001.patch > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[
https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748238#comment-16748238
]
Hadoop QA commented on HBASE-21561:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} HBASE-21561 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.8.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HBASE-21561 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955696/HBASE-21561.branch1-1.001.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/15665/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Backport HBASE-21413 (Empty meta log doesn't get split when restart whole
> cluster) to branch-1
> --
>
> Key: HBASE-21561
> URL: https://issues.apache.org/jira/browse/HBASE-21561
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Xu Cang
>Priority: Minor
> Fix For: 1.5.0, 1.4.10
>
> Attachments: HBASE-21561.branch1-1.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-21561: Attachment: HBASE-21561.branch1-1.001.patch Status: Patch Available (was: Open) > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > Attachments: HBASE-21561.branch1-1.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748224#comment-16748224 ] Abhishek Singh Chouhan commented on HBASE-21680: Latest patch LGTM, +1. [~apurtell] Do we also want to fix the NPE issue over at HBASE-21749 with the patch here? (looks like we may hit the same thing here also) > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21667) Move to latest ASF Parent POM
[
https://issues.apache.org/jira/browse/HBASE-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748211#comment-16748211
]
Hadoop QA commented on HBASE-21667:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 3m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:orange}-0{color} | {color:orange} test4tests {color} | {color:orange}
0m 0s{color} | {color:orange} The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch. Also
please list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 14m
29s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
10s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
1s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
9s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 9s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 24m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 5m
6s{color} | {color:blue} patch has no errors when building the reference guide.
See footer for rendered docs, which you should manually inspect. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
14s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}245m 44s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}334m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestSplitTransactionOnCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21667 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955665/HBASE-21667.master.001.patch
|
| Optional Tests | dupname asflicense javac javadoc unit shadedjars
hadoopcheck xml compile refguide mvnsite |
| uname | Linux de153a5d3392 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 35df6147ee |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| refguide |
[jira] [Updated] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21680: --- Attachment: HBASE-21680-branch-1.patch > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21680) Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic Replication Web UI - Regionserver) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748191#comment-16748191 ] Andrew Purtell commented on HBASE-21680: Latest patch clears ageOfLastShippedOp map every metrics refresh per RB feedback. > Port HBASE-20194 (Basic Replication WebUI - Master) and HBASE-20193 (Basic > Replication Web UI - Regionserver) to branch-1 > - > > Key: HBASE-21680 > URL: https://issues.apache.org/jira/browse/HBASE-21680 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Major > Fix For: 1.5.0 > > Attachments: HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, > HBASE-21680-branch-1.patch, HBASE-21680-branch-1.patch, Screen Shot > 2019-01-16 at 3.20.00 PM.png, Screen Shot 2019-01-16 at 3.20.50 PM.png, > Screen Shot 2019-01-16 at 3.21.17 PM.png, Screen Shot 2019-01-17 at 5.25.21 > PM.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-21561: --- Parent Issue: HBASE-21673 (was: HBASE-21413) > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748190#comment-16748190 ] Andrew Purtell commented on HBASE-21561: Assigned to you [~xucang], go for it! > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell reassigned HBASE-21561: -- Assignee: Xu Cang > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Xu Cang >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21561) Backport HBASE-21413 (Empty meta log doesn't get split when restart whole cluster) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-21561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748169#comment-16748169 ] Xu Cang commented on HBASE-21561: - Is anyone working on this? If now I can take this up. [~allan163] ? > Backport HBASE-21413 (Empty meta log doesn't get split when restart whole > cluster) to branch-1 > -- > > Key: HBASE-21561 > URL: https://issues.apache.org/jira/browse/HBASE-21561 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Priority: Minor > Fix For: 1.5.0, 1.4.10 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21648) [rsgroup] hbase shell "move_servers_rsgroup" or "balance_rsgroup" will be failed when meet a split region.
[
https://issues.apache.org/jira/browse/HBASE-21648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748159#comment-16748159
]
Xu Cang commented on HBASE-21648:
-
{code:java}
+if(regionState == null || (regionState.isClosed() &&
region.isSplit() && region.isOffline())) {
{code}
I have question regarding this line. Should it be
{code:java}
regionState.isClosed() || region.isSplit() || region.isOffline()
{code}
?
> [rsgroup] hbase shell "move_servers_rsgroup" or "balance_rsgroup" will be
> failed when meet a split region.
> --
>
> Key: HBASE-21648
> URL: https://issues.apache.org/jira/browse/HBASE-21648
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.1.0
>Reporter: xuming
>Priority: Major
> Attachments: HBASE-21648.v1.patch
>
>
> A example：
> We have a table "A" which is in RSGroup "group1".
> "bd806f94a53be74e65bd76e1e6e16e5a" is a region of A and is opened on RS "rs1".
> Two steps will repeat this bug:
> step1: Split region bd806f94a53be74e65bd76e1e6e16e5a
> step2: Before the region is cleared by CatalogJanitor, client runs shell :
> move_server_rsgroup 'group2', ['rs1:60020'] or balance_rsgroup 'group1'
> Finally, client will have exceptions below and rest regions moving will be
> interrupted.
> {noformat}
> ERROR: org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> bd806f94a53be74e65bd76e1e6e16e5a is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:189)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:71)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:755)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.move(AssignmentManager.java:560)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.moveServers(RSGroupAdminServer.java:349)
> at
> org.apache.hadoop.hbase.rsgroup.FGRSGroupAdminServer.moveServers(FGRSGroupAdminServer.java:119)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.moveServers(RSGroupAdminEndpoint.java:209)
> at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13870)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:813)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> For usage try 'help "move_servers_rsgroup”'
> ERROR: org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> bd806f94a53be74e65bd76e1e6e16e5a is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:189)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:71)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:755)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.moveAsync(AssignmentManager.java:565)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminServer.balanceRSGroup(RSGroupAdminServer.java:516)
> at
> org.apache.hadoop.hbase.rsgroup.FGRSGroupAdminServer.balanceRSGroup(FGRSGroupAdminServer.java:164)
> at
> org.apache.hadoop.hbase.rsgroup.RSGroupAdminEndpoint$RSGroupAdminServiceImpl.balanceRSGroup(RSGroupAdminEndpoint.java:296)
> at
> org.apache.hadoop.hbase.protobuf.generated.RSGroupAdminProtos$RSGroupAdminService.callMethod(RSGroupAdminProtos.java:13890)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.execMasterService(MasterRpcServices.java:813)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:130)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> For usage try 'help "balance_rsgroup"'{noformat}
> Aflter splitting, this parent region will not be used anymore and will be
>
[jira] [Commented] (HBASE-21734) Some optimization in FilterListWithOR
[ https://issues.apache.org/jira/browse/HBASE-21734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748138#comment-16748138 ] Peter Somogyi commented on HBASE-21734: --- Thanks for the detailed explanation [~openinx]! +1 from my side. > Some optimization in FilterListWithOR > - > > Key: HBASE-21734 > URL: https://issues.apache.org/jira/browse/HBASE-21734 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Attachments: HBASE-21734.branch-1.v1.patch, HBASE-21734.v1.patch, > columnkey.txt, perf-ut.patch > > > In HBASE-21620, [~KarthickRam] and [~mohamed.meeran] complaind that their > performance of filter list has been degraded after that patch in here [1]. > I wrote a UT for this, and test under my host. It's true. I gussed there > may be two reasons: > 1. the comparator.compare(nextKV, cell) > 0 StoreScanner; > 2. the filter list concated by OR will choose the minimal forward step among > all sub-filters. in this patch, we have stricter restrictions on all sub > filters, include those sub-filter whose has non-null RC returned in > calculateReturnCodeByPrevCellAndRC (previously, we will skip to merge this > sub-filter's rc, but it's wrong in some case), and merge all of the > sub-filter's RC, this is also some time cost. > The former one seems not the main problem, because the UT still cost ~ 3s > even if I comment the compare. the second one has some impact indeed, > because after i skip to merge the sub-filters's RC if > calculateReturnCodeByPrevCellAndRC return a non-null rc, the UT cost ~ 1s, > it's improvement but the logic is not wrong. > 1. > https://issues.apache.org/jira/browse/HBASE-21620?focusedCommentId=16737100=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16737100 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21749) RS UI may throw NPE and make rs-status page inaccessible with multiwal and replication
[
https://issues.apache.org/jira/browse/HBASE-21749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748133#comment-16748133
]
Andrew Purtell commented on HBASE-21749:
lgtm, let me commit after some local checks. Thanks [~nihaljain.cs]
> RS UI may throw NPE and make rs-status page inaccessible with multiwal and
> replication
> --
>
> Key: HBASE-21749
> URL: https://issues.apache.org/jira/browse/HBASE-21749
> Project: HBase
> Issue Type: Bug
> Components: Replication, UI
>Affects Versions: 3.0.0, 2.1.0
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Major
> Attachments: HBASE-21749.master.001.patch
>
>
> Sometimes RS UI is unable to open as we get a NPE; This happens because
> {{shipper.getCurrentPath()}} may return null.
> We should have a null check @
> [ReplicationSource.java#L331|https://github.com/apache/hbase/blob/a2f6768acdc30b789c7cb8482b9f4352803f60a1/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java#L331]
>
> {code:java}
> Path currentPath = shipper.getCurrentPath();
> try {
> fileSize = getFileSize(currentPath);
> } catch (IOException e) {
> LOG.warn("Ignore the exception as the file size of HLog only affects
> the web ui", e);
> fileSize = -1;
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21585) Use ConnectionImplementation instead of ClusterConnection for sync client implementation
[
https://issues.apache.org/jira/browse/HBASE-21585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748104#comment-16748104
]
Hadoop QA commented on HBASE-21585:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 41 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-21512 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
25s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
44s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
45s{color} | {color:green} HBASE-21512 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
25s{color} | {color:green} HBASE-21512 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 52s{color}
| {color:red} hbase-client generated 2 new + 98 unchanged - 2 fixed = 100 total
(was 100) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} hbase-client: The patch generated 0 new + 243
unchanged - 36 fixed = 243 total (was 279) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
29s{color} | {color:red} hbase-server: The patch generated 2 new + 542
unchanged - 31 fixed = 544 total (was 573) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} hbase-mapreduce: The patch generated 0 new + 4
unchanged - 1 fixed = 4 total (was 5) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} hbase-thrift: The patch generated 2 new + 0 unchanged
- 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch passed checkstyle in hbase-endpoint
{color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} hbase-it: The patch generated 0 new + 3 unchanged -
3 fixed = 3 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
15s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 16s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
40s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}257m 46s{color}
| {color:red} hbase-server in the patch failed.
[jira] [Commented] (HBASE-21751) WAL create fails during region open may cause region assign forever fail
[
https://issues.apache.org/jira/browse/HBASE-21751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748094#comment-16748094
]
Hadoop QA commented on HBASE-21751:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
37s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
14s{color} | {color:red} hbase-server: The patch generated 2 new + 5 unchanged
- 0 fixed = 7 total (was 5) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 36s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
33s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-21751 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955658/HBASE-21751.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 2271cc2d05fd 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 35df6147ee |
| maven | version: Apache Maven 3.5.4
(1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T18:33:14Z) |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC3 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HBASE-Build/15661/artifact/patchprocess/diff-checkstyle-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/15661/testReport/ |
| Max. process+thread count | 4884 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
[jira] [Commented] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[
https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748073#comment-16748073
]
Hadoop QA commented on HBASE-21750:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
33s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} hbase-common: The patch generated 0 new + 27
unchanged - 1 fixed = 27 total (was 28) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
33s{color} | {color:red} hbase-client: The patch generated 1 new + 50 unchanged
- 0 fixed = 51 total (was 50) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} hbase-server: The patch generated 12 new + 311
unchanged - 1 fixed = 323 total (was 312) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
32s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 45s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
29s{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
5s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 24m 12s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 26s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.filter.TestFilterList |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue |
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748034#comment-16748034
]
Hudson commented on HBASE-21738:
Results for branch branch-2
[build #1625 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21512) Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
[
https://issues.apache.org/jira/browse/HBASE-21512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748011#comment-16748011
]
Hudson commented on HBASE-21512:
Results for branch HBASE-21512
[build #64 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/64/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/64//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/64//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-21512/64//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Introduce an AsyncClusterConnection and replace the usage of ClusterConnection
> --
>
> Key: HBASE-21512
> URL: https://issues.apache.org/jira/browse/HBASE-21512
> Project: HBase
> Issue Type: Umbrella
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 3.0.0
>
>
> At least for the RSProcedureDispatcher, with CompletableFuture we do not need
> to set a delay and use a thread pool any more, which could reduce the
> resource usage and also the latency.
> Once this is done, I think we can remove the ClusterConnection completely,
> and start to rewrite the old sync client based on the async client, which
> could reduce the code base a lot for our client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21667) Move to latest ASF Parent POM
[ https://issues.apache.org/jira/browse/HBASE-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-21667: -- Status: Patch Available (was: Open) > Move to latest ASF Parent POM > - > > Key: HBASE-21667 > URL: https://issues.apache.org/jira/browse/HBASE-21667 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21667.master.001.patch > > > Currently HBase depends on version 18 which was released on 2016-05-18. > Version 21 was released in August 2018. > Relevant dependency upgrades > > ||Name||Currently used version||New version||Notes|| > |surefire.version|2.21.0|2.22.0| | > |maven-compiler-plugin|3.6.1|3.7| | > |maven-dependency-plugin|3.0.1|3.1.1| | > |maven-jar-plugin|3.0.0|3.0.2| | > |maven-javadoc-plugin|3.0.0|3.0.1| | > |maven-resources-plugin|2.7|3.1.0| | > |maven-site-plugin|3.4|3.7.1|Currently not relying on ASF version. See: > HBASE-18333| > |maven-source-plugin|3.0.0|3.0.1| | > |maven-shade-plugin|3.0.0|3.1.1|Newly added to ASF pom| > |maven-clean-plugin|3.0.0|3.1.0| | > |maven-project-info-reports-plugin |2.9|3.0.0| | > Version 21 added net.nicoulaj.maven.plugins:checksum-maven-plugin which > introduced SHA512 checksum instead of SHA1. Should verify if we can rely on > that for releases or breaks our current processes. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21667) Move to latest ASF Parent POM
[ https://issues.apache.org/jira/browse/HBASE-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi updated HBASE-21667: -- Attachment: HBASE-21667.master.001.patch > Move to latest ASF Parent POM > - > > Key: HBASE-21667 > URL: https://issues.apache.org/jira/browse/HBASE-21667 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Minor > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21667.master.001.patch > > > Currently HBase depends on version 18 which was released on 2016-05-18. > Version 21 was released in August 2018. > Relevant dependency upgrades > > ||Name||Currently used version||New version||Notes|| > |surefire.version|2.21.0|2.22.0| | > |maven-compiler-plugin|3.6.1|3.7| | > |maven-dependency-plugin|3.0.1|3.1.1| | > |maven-jar-plugin|3.0.0|3.0.2| | > |maven-javadoc-plugin|3.0.0|3.0.1| | > |maven-resources-plugin|2.7|3.1.0| | > |maven-site-plugin|3.4|3.7.1|Currently not relying on ASF version. See: > HBASE-18333| > |maven-source-plugin|3.0.0|3.0.1| | > |maven-shade-plugin|3.0.0|3.1.1|Newly added to ASF pom| > |maven-clean-plugin|3.0.0|3.1.0| | > |maven-project-info-reports-plugin |2.9|3.0.0| | > Version 21 added net.nicoulaj.maven.plugins:checksum-maven-plugin which > introduced SHA512 checksum instead of SHA1. Should verify if we can rely on > that for releases or breaks our current processes. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748005#comment-16748005
]
Hudson commented on HBASE-21738:
Results for branch branch-2.0
[build #1275 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21750: - Status: Patch Available (was: Open) > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-21750) Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for better performance because the latter one has been optimized
[ https://issues.apache.org/jira/browse/HBASE-21750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu updated HBASE-21750: - Attachment: HBASE-21750.v1.patch > Most of KeyValueUtil#length can be replaced by cell#getSerializedSize for > better performance because the latter one has been optimized > -- > > Key: HBASE-21750 > URL: https://issues.apache.org/jira/browse/HBASE-21750 > Project: HBase > Issue Type: Sub-task >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: HBASE-21750.v1.patch > > > After HBASE-21657, Most subclass of Cell has a cached serialized size (Except > those cells with tags), so I think most of the KeyValueUtil#length can be > replaced by cell#getSerializedSize. Such as: > - KeyValueUtil.length in StoreFlusher#performFlush; > - KeyValueUtil.length in Compactor#performCompaction ; > and so on.. > Will prepare a patch for this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21716) Add toStringCustomizedValues to TableDescriptor
[ https://issues.apache.org/jira/browse/HBASE-21716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747993#comment-16747993 ] kevin su commented on HBASE-21716: -- I double check my latest patch, but i didn't see any *indentation* error. Could anyone help me. Thanks for advanced. > Add toStringCustomizedValues to TableDescriptor > --- > > Key: HBASE-21716 > URL: https://issues.apache.org/jira/browse/HBASE-21716 > Project: HBase > Issue Type: Task >Reporter: Duo Zhang >Assignee: kevin su >Priority: Major > Labels: beginner > Attachments: HBASE-21716.patch01, HBASE-21716.v1.patch > > > HTableDescriptor.toStringCustomizedValues is used in our code base,especially > that on the web page, we should add this method to TableDescriptor if we want > to remove HTableDescriptor completely -- This message was sent by Atlassian JIRA (v7.6.3#76005)
