[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509217#comment-16509217
]
Guanghao Zhang commented on HBASE-20697:
bq. for(RegionLocations regionLocations:listRegionLocations())
No need to scan meta again... Use a map which key is region's start key and end
key?
Please add patch for master first, then add patch for other branchs which need
backport. You can use dev-support/submit-patch.py to submit patch. The patch
file should have version number, etc. v1, v2, v3. Take a look about hbase
document http://hbase.apache.org/book.html#submitting.patches.create
Thanks.
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Minor
> Fix For: 1.2.7, 1.3.3
>
> Attachments: HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HBASE-20682) MoveProcedure can be subtask of ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher.
[ https://issues.apache.org/jira/browse/HBASE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-20682 started by stack. - > MoveProcedure can be subtask of > ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher. > > > Key: HBASE-20682 > URL: https://issues.apache.org/jira/browse/HBASE-20682 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.1 > > > MoveProcedure is used in at least two places as a means of 'reopening' a > region after a config has changed. This issue is about review of MP so this > usage is first-class (and that MP is good procedure citizen running as a > subprocedure. In question is what should happen if the source server or the > target servers are offline when we go to run. As of the parent issue, we'll > skip to the end. Should we instead go ahead with the move (internally, if we > are asked to assign to an offline server, we'll pick another -- do same for > unassign? Would help in this reopen use case). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20682) MoveProcedure can be subtask of ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher.
[ https://issues.apache.org/jira/browse/HBASE-20682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509213#comment-16509213 ] stack commented on HBASE-20682: --- Started. Its taking a while... > MoveProcedure can be subtask of > ModifyTableProcedure/ReopenTableRegionsProcedure; ensure all kosher. > > > Key: HBASE-20682 > URL: https://issues.apache.org/jira/browse/HBASE-20682 > Project: HBase > Issue Type: Sub-task > Components: amv2 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.1 > > > MoveProcedure is used in at least two places as a means of 'reopening' a > region after a config has changed. This issue is about review of MP so this > usage is first-class (and that MP is good procedure citizen running as a > subprocedure. In question is what should happen if the source server or the > target servers are offline when we go to run. As of the parent issue, we'll > skip to the end. Should we instead go ahead with the move (internally, if we > are asked to assign to an offline server, we'll pick another -- do same for > unassign? Would help in this reopen use case). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509198#comment-16509198
]
stack commented on HBASE-19121:
---
I just noticed that we reference hbck in some of our shell help doc:
{code}
hbase(main):022:0> help 'unassign'
Unassign a region. Unassign will close region in current location and then
reopen it again. Pass 'true' to force the unassignment ('force' will clear
all in-memory state in master before the reassign. If results in
double assignment use hbck -fix to resolve. To be used by experts).
Use with caution. For expert use only. Examples:
hbase> unassign 'REGIONNAME'
hbase> unassign 'REGIONNAME', true
hbase> unassign 'ENCODED_REGIONNAME'
hbase> unassign 'ENCODED_REGIONNAME', true
{code}
We should clean these up too as part of move to hbck2.
> HBCK for AMv2 (A.K.A HBCK2)
> ---
>
> Key: HBASE-19121
> URL: https://issues.apache.org/jira/browse/HBASE-19121
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Reporter: stack
>Assignee: Umesh Agashe
>Priority: Major
> Attachments: hbase-19121.master.001.patch
>
>
> We don't have an hbck for the new AM. Old hbck may actually do damage going
> against AMv2.
> Fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20625) refactor some WALCellCodec related code
[
https://issues.apache.org/jira/browse/HBASE-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509193#comment-16509193
]
Hadoop QA commented on HBASE-20625:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
53s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hbase-server: The patch generated 0 new + 72
unchanged - 3 fixed = 72 total (was 75) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
55s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 47s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
14s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}173m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20625 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927417/HBASE-20625.master.005.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 0105409ca2ff 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / edf60b965b |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13202/testReport/ |
| Max. process+thread count | 4354 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13202/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This
[jira] [Commented] (HBASE-20704) Sometimes some compacted storefiles are not archived on region close
[ https://issues.apache.org/jira/browse/HBASE-20704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509190#comment-16509190 ] Anoop Sam John commented on HBASE-20704: Makes sense.. We can handle in another jira. That is better. > Sometimes some compacted storefiles are not archived on region close > > > Key: HBASE-20704 > URL: https://issues.apache.org/jira/browse/HBASE-20704 > Project: HBase > Issue Type: Bug > Components: Compaction >Affects Versions: 3.0.0, 1.3.0, 1.4.0, 1.5.0, 2.0.0 >Reporter: Francis Liu >Assignee: Francis Liu >Priority: Critical > Attachments: HBASE-20704.001.patch, HBASE-20704.002.patch > > > During region close compacted files which have not yet been archived by the > discharger are archived as part of the region closing process. It is > important that these files are wholly archived to insure data consistency. ie > a storefile containing delete tombstones can be archived while older > storefiles containing cells that were supposed to be deleted are left > unarchived thereby undeleting those cells. > On region close a compacted storefile is skipped from archiving if it has > read references (ie open scanners). This behavior is correct for when the > discharger chore runs but on region close consistency is of course more > important so we should add a special case to ignore any references on the > storefile and go ahead and archive it. > Attached patch contains a unit test that reproduces the problem and the > proposed fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20700) Move meta region when server crash can cause the procedure to be stuck
[
https://issues.apache.org/jira/browse/HBASE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509185#comment-16509185
]
Hadoop QA commented on HBASE-20700:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.0 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
40s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
12s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s{color} | {color:green} branch-2.0 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} branch-2.0 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hbase-procedure passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} hbase-server: The patch generated 0 new + 274
unchanged - 4 fixed = 274 total (was 278) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 38s{color} | {color:green} Patch does not cause any errors with Hadoop
2.6.5 2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
45s{color} | {color:green} hbase-procedure in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}124m
16s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}171m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:6f01af0 |
| JIRA Issue | HBASE-20700 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927415/HBASE-20700-branch-2.0-v1.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux c20df5b21f39 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[ https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509162#comment-16509162 ] stack commented on HBASE-20708: --- Can we do up an outline doc [~Apache9]? I'm having a bit of trouble following along. The RMP was added so there was one place for doing hbase:meta checking and assign. Previous this stuff was spread about the place. RMP collected it all up into one location. It is run on clean startup and deploying to new location on server crash. SCP was turned off during startup because with it running concurrent to startup, it was too hard to reason around all the moving parts. bq. then we could let it go until it reaches the SERVER_CRASH_GET_REGIONS state. Remove the guard and just let SCPs block on SERVER_CRASH_GET_REGIONS instead? That could work. bq. There is no way to make sure that the meta region is online, as a server crash can happen at any time. So we should not make assumption that the meta region is online when designing or writing code. If a crash of the hbase:meta server during statup, we recover, no? bq. I think we could make the logic a little simpler here. Sure. The SCP that was carrying hbase:meta is marked specially. Would be an improvement if the queuing scheduled it first. Would be an improvement if could remove checks for meta deployed that are in various places in code. bq. And when reading the code, I found something strange, when updating meta location, we always mark it as OPEN. IIRC, thinking was probably that hbase:meta would never be CLOSED. > Remove the usage of RecoverMetaProcedure in master startup > -- > > Key: HBASE-20708 > URL: https://issues.apache.org/jira/browse/HBASE-20708 > Project: HBase > Issue Type: Bug > Components: proc-v2, Region Assignment >Reporter: Duo Zhang >Priority: Blocker > Fix For: 3.0.0, 2.1.0 > > > In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only > used by RMP to avoid dead lock with MoveRegionProcedure. But we will always > schedule a RMP when master starting up, so we still need to make sure that > there is no race between this RMP and other RMPs and SCPs scheduled before > the master restarts. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509144#comment-16509144
]
zhaoyuan edited comment on HBASE-20697 at 6/12/18 4:22 AM:
---
Hi [~zghaobac] I checked out the failed test TestAdmin1 and ran it locally in
my mac, well it passed，and this test has nothing to to with
RegionLocator.getAllRegionLocations(),So I think it is false positive .For
branch-1.2 I think this simple patch works.Please check the QA results and
tell me If I miss anything, thanks!
was (Author: smartzy):
Hi [~zghaobac] I checked out the failed test TestAdmin1 and ran it locally in
my mac, well it passed. So I think it is false positive ,and for branch-1.2 I
think this simple patch works.Please check the QA results and tell me If I
miss anything, thanks!
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Minor
> Fix For: 1.2.7, 1.3.3
>
> Attachments: HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be
[jira] [Commented] (HBASE-20635) Support to convert the shaded user permission proto to client user permission object
[
https://issues.apache.org/jira/browse/HBASE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509156#comment-16509156
]
Hudson commented on HBASE-20635:
Results for branch branch-2
[build #852 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/852/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/852//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/852//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/852//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Support to convert the shaded user permission proto to client user permission
> object
>
>
> Key: HBASE-20635
> URL: https://issues.apache.org/jira/browse/HBASE-20635
> Project: HBase
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20635.patch, HBASE-20635_v2.patch,
> PHOENIX-4528_5.x-HBase-2.0_v2.patch
>
>
> Currently we have API to build the protobuf UserPermission to client user
> permission in AccessControlUtil but we cannot do the same when we use shaded
> protobufs.
> {noformat}
> /**
>* Converts a user permission proto to a client user permission object.
>*
>* @param proto the protobuf UserPermission
>* @return the converted UserPermission
>*/
> public static UserPermission
> toUserPermission(AccessControlProtos.UserPermission proto) {
> return new UserPermission(proto.getUser().toByteArray(),
> toTablePermission(proto.getPermission()));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509144#comment-16509144
]
zhaoyuan commented on HBASE-20697:
--
Hi [~zghaobac] I checked out the failed test TestAdmin1 and ran it locally in
my mac, well it passed. So I think it is false positive ,and for branch-1.2 I
think this simple patch works.Please check the QA results and tell me If I
miss anything, thanks!
> Can't cache All region locations of the specify table by calling
> table.getRegionLocator().getAllRegionLocations()
> -
>
> Key: HBASE-20697
> URL: https://issues.apache.org/jira/browse/HBASE-20697
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.3.1, 1.2.6
>Reporter: zhaoyuan
>Assignee: zhaoyuan
>Priority: Minor
> Fix For: 1.2.7, 1.3.3
>
> Attachments: HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch,
> HBASE-20697-branch-1.2.patch, HBASE-20697-branch-1.2.patch
>
>
> When we upgrade and restart a new version application which will read and
> write to HBase, we will get some operation timeout. The time out is expected
> because when the application restarts，It will not hold any region locations
> cache and do communication with zk and meta regionserver to get region
> locations.
> We want to avoid these timeouts so we do warmup work and as far as I am
> concerned,the method table.getRegionLocator().getAllRegionLocations() will
> fetch all region locations and cache them. However, it didn't work good.
> There are still a lot of time outs,so it confused me.
> I dig into the source code and find something below
> {code:java}
> // code placeholder
> public List getAllRegionLocations() throws IOException {
> TableName tableName = getName();
> NavigableMap locations =
> MetaScanner.allTableRegions(this.connection, tableName);
> ArrayList regions = new ArrayList<>(locations.size());
> for (Entry entry : locations.entrySet()) {
> regions.add(new HRegionLocation(entry.getKey(), entry.getValue()));
> }
> if (regions.size() > 0) {
> connection.cacheLocation(tableName, new RegionLocations(regions));
> }
> return regions;
> }
> In MetaCache
> public void cacheLocation(final TableName tableName, final RegionLocations
> locations) {
> byte [] startKey =
> locations.getRegionLocation().getRegionInfo().getStartKey();
> ConcurrentMap tableLocations =
> getTableLocations(tableName);
> RegionLocations oldLocation = tableLocations.putIfAbsent(startKey,
> locations);
> boolean isNewCacheEntry = (oldLocation == null);
> if (isNewCacheEntry) {
> if (LOG.isTraceEnabled()) {
> LOG.trace("Cached location: " + locations);
> }
> addToCachedServers(locations);
> return;
> }
> {code}
> It will collect all regions into one RegionLocations object and only cache
> the first not null region location and then when we put or get to hbase, we
> do getCacheLocation()
> {code:java}
> // code placeholder
> public RegionLocations getCachedLocation(final TableName tableName, final
> byte [] row) {
> ConcurrentNavigableMap tableLocations =
> getTableLocations(tableName);
> Entry e = tableLocations.floorEntry(row);
> if (e == null) {
> if (metrics!= null) metrics.incrMetaCacheMiss();
> return null;
> }
> RegionLocations possibleRegion = e.getValue();
> // make sure that the end key is greater than the row we're looking
> // for, otherwise the row actually belongs in the next region, not
> // this one. the exception case is when the endkey is
> // HConstants.EMPTY_END_ROW, signifying that the region we're
> // checking is actually the last region in the table.
> byte[] endKey =
> possibleRegion.getRegionLocation().getRegionInfo().getEndKey();
> if (Bytes.equals(endKey, HConstants.EMPTY_END_ROW) ||
> getRowComparator(tableName).compareRows(
> endKey, 0, endKey.length, row, 0, row.length) > 0) {
> if (metrics != null) metrics.incrMetaCacheHit();
> return possibleRegion;
> }
> // Passed all the way through, so we got nothing - complete cache miss
> if (metrics != null) metrics.incrMetaCacheMiss();
> return null;
> }
> {code}
> It will choose the first location to be possibleRegion and possibly it will
> miss match.
> So did I forget something or may be wrong somewhere? If this is indeed a bug
> I think it can be fixed not very hard.
> Hope commiters and PMC review this !
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20691) Storage policy should allow deferring to HDFS
[ https://issues.apache.org/jira/browse/HBASE-20691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509138#comment-16509138 ] Yu Li commented on HBASE-20691: --- Any more comments or are we good to move on gentlemen? [~mdrob] [~busbey] Thanks. :-) > Storage policy should allow deferring to HDFS > - > > Key: HBASE-20691 > URL: https://issues.apache.org/jira/browse/HBASE-20691 > Project: HBase > Issue Type: Bug > Components: Filesystem Integration, wal >Affects Versions: 1.5.0, 2.0.0 >Reporter: Sean Busbey >Assignee: Yu Li >Priority: Minor > Fix For: 2.1.0, 1.5.0 > > Attachments: HBASE-20691.patch, HBASE-20691.v2.patch, > HBASE-20691.v3.patch > > > In HBase 1.1 - 1.4 we can defer storage policy decisions to HDFS by using > "NONE" as the storage policy in hbase configs. > As described on this [dev@hbase thread "WAL storage policies and interactions > with Hadoop admin > tools."|https://lists.apache.org/thread.html/d220726fab4bb4c9e117ecc8f44246402dd97bfc986a57eb2237@%3Cdev.hbase.apache.org%3E] > we no longer have that option in 2.0.0 and 1.5.0 (as the branch is now). > Additionally, we can't set the policy to HOT in the event that HDFS has > changed the policy for a parent directory of our WALs. > We should put back that ability. Presuming this is done by re-adopting the > "NONE" placeholder variable, we need to ensure that value doesn't get passed > to HDFS APIs. Since it isn't a valid storage policy attempting to use it will > cause a bunch of logging churn (which will be a regression of the problem > HBASE-18118 sought to fix). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20708) Remove the usage of RecoverMetaProcedure in master startup
[
https://issues.apache.org/jira/browse/HBASE-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509129#comment-16509129
]
Duo Zhang commented on HBASE-20708:
---
And there is another problem for master restart is that, we need to find out
the RSes crashed after the old master quits and before the new master starts.
For now this is done at two places, first we will do it in
MasterMetaBootstrap.processDeadServers, and second in AssignmentManager after
we finish loading meta. And now we have two guards for SCP. After master
bootstrap we will enable server crash processing, but if the SCP is not for a
RS with meta region, then it will need to wait for the assignment to finish
loading the meta.
I think we could make the logic a little simpler here. When master starts, we
load all the procedures first but do not start procedure workers, initialize
RegionServerTracker to get the current online server lists, and scan the wal
directory to get RSes which have been alive for sometime, and finally we can
use these informations to find out the crashed RSes. And we can use the loaded
procedures to filter out the RSes which have not been processed, i.e, do not
have a SCP yet.
And I think we can remove the enable server crash processing guard, if a SCP is
for a RS with meta, then we could let it go until it reaches the
SERVER_CRASH_GET_REGIONS state.
And when reading the code, I found something strange, when updating meta
location, we always mark it as OPEN.
{code}
private void updateMetaLocation(final RegionInfo regionInfo, final ServerName
serverName)
throws IOException {
try {
MetaTableLocator.setMetaLocation(master.getZooKeeper(), serverName,
regionInfo.getReplicaId(), State.OPEN);
} catch (KeeperException e) {
throw new IOException(e);
}
}
{code}
Any reason why we do this? [~stack] Thanks.
> Remove the usage of RecoverMetaProcedure in master startup
> --
>
> Key: HBASE-20708
> URL: https://issues.apache.org/jira/browse/HBASE-20708
> Project: HBase
> Issue Type: Bug
> Components: proc-v2, Region Assignment
>Reporter: Duo Zhang
>Priority: Blocker
> Fix For: 3.0.0, 2.1.0
>
>
> In HBASE-20700, we make RecoverMetaProcedure use a special lock which is only
> used by RMP to avoid dead lock with MoveRegionProcedure. But we will always
> schedule a RMP when master starting up, so we still need to make sure that
> there is no race between this RMP and other RMPs and SCPs scheduled before
> the master restarts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20717) Implement CCSMap - a better concurrent map with compacted data structure
[ https://issues.apache.org/jira/browse/HBASE-20717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509119#comment-16509119 ] Yu Li commented on HBASE-20717: --- Thanks for the reminder. We plan to abandon the review in the parent JIRA and create new ones for these sub-tasks, so let us record the left comments and address them in new patches separately, unless objections. > Implement CCSMap - a better concurrent map with compacted data structure > > > Key: HBASE-20717 > URL: https://issues.apache.org/jira/browse/HBASE-20717 > Project: HBase > Issue Type: Sub-task >Reporter: Yu Li >Assignee: Chance Li >Priority: Major > > As described in HBASE-20312, we will implement the base data structure of > CCSMap and relative unit tests in this task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509116#comment-16509116 ] Yu Li commented on HBASE-20312: --- [~stack] [~apurtell] [~mdrob] [~yuzhih...@gmail.com] For your information gentlemen and it would be great if we could get some review comments on the design doc from you first (smile) > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20718) Implement a MemStore with CCSMap for better performance
Yu Li created HBASE-20718: - Summary: Implement a MemStore with CCSMap for better performance Key: HBASE-20718 URL: https://issues.apache.org/jira/browse/HBASE-20718 Project: HBase Issue Type: Sub-task Reporter: Yu Li Assignee: Chance Li As described in HBASE-20312, this task targets at adopting CCSMap into hbase: implementing the CCSLMMemStore with CCSMap for better performance -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20717) Implement CCSMap - a better concurrent map with compacted data structure
[ https://issues.apache.org/jira/browse/HBASE-20717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509108#comment-16509108 ] Ted Yu commented on HBASE-20717: There were some remaining review comments on previous reviewboard which should be addressed. > Implement CCSMap - a better concurrent map with compacted data structure > > > Key: HBASE-20717 > URL: https://issues.apache.org/jira/browse/HBASE-20717 > Project: HBase > Issue Type: Sub-task >Reporter: Yu Li >Assignee: Chance Li >Priority: Major > > As described in HBASE-20312, we will implement the base data structure of > CCSMap and relative unit tests in this task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20717) Implement CCSMap - a better concurrent map with compacted data structure
Yu Li created HBASE-20717: - Summary: Implement CCSMap - a better concurrent map with compacted data structure Key: HBASE-20717 URL: https://issues.apache.org/jira/browse/HBASE-20717 Project: HBase Issue Type: Sub-task Reporter: Yu Li Assignee: Chance Li As described in HBASE-20312, we will implement the base data structure of CCSMap and relative unit tests in this task. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509105#comment-16509105 ] Yu Li edited comment on HBASE-20312 at 6/12/18 2:56 AM: Uploaded the design doc for better understanding and easier review of code (Thanks [~chancelq] as co-author). In theory we have below conclusion and more detailed analysis please refer to the doc: * With JDK's CSLM we need 62.5% extra objects and 100% extra memory * With CCSMap (almost) no extra objects and only 36% extra memory required Please let us know if any comments on the design, and as per suggested we will split the job into two sub-tasks for CCSMap implementation and adoption into HBase separately. was (Author: carp84): Uploaded the design doc for better understanding and easier review of code (Thanks [~chancelq] as co-author). In theory we have below conclusion and more detailed analysis please refer to the doc: * With JDK's CSLM we need 62.5% extra objects and 100% extra memory * With CCSMap (almost) no extra objects and only 36% extra memory required Please let us know if any comments on the design, and as per suggested we will split the job into two sub-tasks for CCSMap implementation and adoption into HBase separately. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20312) CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore
[ https://issues.apache.org/jira/browse/HBASE-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509105#comment-16509105 ] Yu Li commented on HBASE-20312: --- Uploaded the design doc for better understanding and easier review of code (Thanks [~chancelq] as co-author). In theory we have below conclusion and more detailed analysis please refer to the doc: * With JDK's CSLM we need 62.5% extra objects and 100% extra memory * With CCSMap (almost) no extra objects and only 36% extra memory required Please let us know if any comments on the design, and as per suggested we will split the job into two sub-tasks for CCSMap implementation and adoption into HBase separately. > CCSMap: A faster, GC-friendly, less memory Concurrent Map for memstore > -- > > Key: HBASE-20312 > URL: https://issues.apache.org/jira/browse/HBASE-20312 > Project: HBase > Issue Type: New Feature > Components: regionserver >Reporter: Xiang Wang >Assignee: Chance Li >Priority: Major > Fix For: 3.0.0 > > Attachments: 1.1.2-ccsmap-number.png, HBASE-20312-1.3.2.patch, > HBASE-20312-master.v1.patch, HBASE-20312-master.v2.patch, > HBASE-20312-master.v3.patch, HBASE-20312-master.v4.patch, > HBASE-20312-master.v5.patch, HBASE-20312-master.v6.patch, > HBASE-20312-master.v7.patch, HBASE-20312-master.v8.patch, > HBASE-20312-master.v9.patch, ccsmap-branch-1.1.patch, hits.png, jira1.png, > jira2.png, jira3.png, off-heap-test-put-master.png, > on-heap-test-put-master.png > > > Now hbase use ConcurrentSkipListMap as memstore's data structure. > Although MemSLAB reduces memory fragment brought by key-value pairs. > Hundred of millions key-value pairs still make young generation > garbage-collection(gc) stop time long. > > These are 2 gc problems of ConcurrentSkipListMap: > 1. HBase needs 3 objects to store one key-value on expectation. One > Index(skiplist's average node height is 1), one Node, and one KeyValue. Too > many objects are created for memstore. > 2. Recent inserted KeyValue and its map structure(Index, Node) are assigned > on young generation.The card table (for CMS gc algorithm) or RSet(for G1 gc > algorithm) will change frequently on high writing throughput, which makes YGC > slow. > > We devleoped a new skip-list map called CompactdConcurrentSkipListMap(CCSMap > for short), > which provides similary features like ConcurrentSkipListMap but get rid of > Objects for every key-value pairs. > CCSMap's memory structure is like this picture: > !jira1.png! > > One CCSMap consists of a certain number of Chunks. One Chunk consists of a > certain number of nodes. One node is corresspding one element. This element's > all information and its key-value is encoded on a continuous memory segment > without any objects. > Features: > 1. all insert,update,delete operations is lock-free on CCSMap. > 2. Consume less memory, it brings 40% memory saving for 50Byte key-value. > 3. Faster on small key-value because of better cacheline usage. 20~30% better > read/write troughput than ConcurrentSkipListMap for 50Byte key-value. > CCSMap do not support recyle space when deleting element. But it doesn't > matter for hbase because of region flush. > CCSMap has been running on Alibaba's hbase clusters over 17 months, it cuts > down YGC time significantly. here are 2 graph of before and after. > !jira2.png! > !jira3.png! > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509084#comment-16509084 ] stack commented on HBASE-20716: --- Agree [~chia7712]. Thanks [~busbey] > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509081#comment-16509081 ] Sean Busbey commented on HBASE-20716: - Agreed that this is Improvement and made the switch. > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20716: Issue Type: Improvement (was: Bug) > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Improvement > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20674) clean up short circuit read logic and docs
[
https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509079#comment-16509079
]
Sean Busbey commented on HBASE-20674:
-
looks like Surefire recognized that it failed ([log from running unit
tests|https://builds.apache.org/job/PreCommit-HBASE-Build/13195/artifact/patchprocess/patch-unit-root.txt]):
{code}
[ERROR] Errors:
[ERROR]
TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL>TestReplicationKillMasterRS.killOneMasterRS:40->TestReplicationKillRS.loadTableAndKillRS:70
» RetriesExhausted
[INFO]
{code}
But as you mentioned [Jenkins JUnit reporter couldn't parse the
results|https://builds.apache.org/job/PreCommit-HBASE-Build/13195/testReport/TEST-org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL/xml/_failed_to_read_/]:
{code}
Failed to read test report file
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/hbase-server/target/surefire-reports/TEST-org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL.xml
org.dom4j.DocumentException: Error on line 80 of document : XML document
structures must start and end within the same entity. Nested exception: XML
document structures must start and end within the same entity.
{code}
If you browse to [the {{artifact/patchprocess}} directory for the
build|https://builds.apache.org/job/PreCommit-HBASE-Build/13195/artifact/patchprocess/]
there'll be a [file with all the surefire results zipped up named
'test_logs.zip'|https://builds.apache.org/job/PreCommit-HBASE-Build/13195/artifact/patchprocess/test_logs.zip].
{code}
$ find . -name *TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL*
./component/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL-output.txt
./component/hbase-server/target/surefire-reports/TEST-org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL.xml
./component/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.replication.multiwal.TestReplicationKillMasterRSCompressedWithMultipleAsyncWAL.txt
{code}
The xml file in question appears truncated. It just ends after opening the
CDATA block for stderr of the test.
The other files contain stuff, but glancing at them didn't reveal anything
obvious.
Hopefully that helps for pointers? It doesn't look like a bug in precommit,
FWIW. maybe surefire.
> clean up short circuit read logic and docs
> --
>
> Key: HBASE-20674
> URL: https://issues.apache.org/jira/browse/HBASE-20674
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 2.0.0
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Attachments: HBASE-20674.patch, HBASE-20674.v2.patch,
> HBASE-20674.v3.patch, HBASE-20674.v4.patch, HBASE-20674.v5.patch
>
>
> Mailing list discussion at
> https://lists.apache.org/thread.html/f6f73df0ceae29f762f9b9088e3ffd0bf8f109d3dd692df100bf4fd6@%3Cdev.hbase.apache.org%3E
> There are several inconsistencies between how our docs claim we do things and
> how we actually do things.
> There are two docs sections that attempt to address how SCR should work.
> dfs.client.read.shortcircuit.skip.checksum is advised to set to true, but our
> code in separate places ignores it and then later sets it to true anyway.
> CommonFSUtils and FSUtils duplicate code related to SCR setup.
> There is a workaround in HFileSystem for a bug that's been fixed in all
> versions of hadoop that we support. (HADOOP-9307)
> We suggest setting dfs.client.read.shortcircuit.buffer.size to a value that
> is very close to what we'd set it to anyway, without clearly explaining why
> this is important.
> There are other properties that we claim are important, but we don't offer
> any suggestions or explanations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20625) refactor some WALCellCodec related code
[
https://issues.apache.org/jira/browse/HBASE-20625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingyun Tian updated HBASE-20625:
-
Attachment: HBASE-20625.master.005.patch
> refactor some WALCellCodec related code
> ---
>
> Key: HBASE-20625
> URL: https://issues.apache.org/jira/browse/HBASE-20625
> Project: HBase
> Issue Type: Improvement
> Components: wal
>Affects Versions: 2.0.0
>Reporter: Jingyun Tian
>Assignee: Jingyun Tian
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HBASE-20625.master.001.patch,
> HBASE-20625.master.002.patch, HBASE-20625.master.002.patch,
> HBASE-20625.master.003.patch, HBASE-20625.master.004.patch,
> HBASE-20625.master.005.patch
>
>
> Currently I'm working on export HLog to another FileSystem, then I found the
> code of WALCellCodec and its related classes is not that clean. And there
> are several TODOs. Thus I tried to refactor the code based one these TODOs.
> e.g.
> {code}
> // TODO: it sucks that compression context is in WAL.Entry. It'd be nice if
> it was here.
> // Dictionary could be gotten by enum; initially, based on enum,
> context would create
> // an array of dictionaries.
> static class BaosAndCompressor extends ByteArrayOutputStream implements
> ByteStringCompressor {
> public ByteString toByteString() {
> // We need this copy to create the ByteString as the byte[] 'buf' is
> not immutable. We reuse
> // them.
> return ByteString.copyFrom(this.buf, 0, this.count);
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20637) Polish the WAL switching when transiting from A to S
[ https://issues.apache.org/jira/browse/HBASE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20637: -- Component/s: Replication > Polish the WAL switching when transiting from A to S > > > Key: HBASE-20637 > URL: https://issues.apache.org/jira/browse/HBASE-20637 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20637-HBASE-19064-v1.patch, > HBASE-20637-HBASE-19064-v2.patch, HBASE-20637-HBASE-19064-v3.patch, > HBASE-20637-HBASE-19064.patch > > > The design in HBASE-20424 implies that, when transiting from A to S, the RS > will abort and the WAL file will be split. > But in fact, there could be races that make a RS still alive after the > transiting which causes data inconsistency. And also, a full cluster restart > is expensive, we need to find a more smooth way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20637) Polish the WAL switching when transiting from A to S
[ https://issues.apache.org/jira/browse/HBASE-20637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20637: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Polish the WAL switching when transiting from A to S > > > Key: HBASE-20637 > URL: https://issues.apache.org/jira/browse/HBASE-20637 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: HBASE-19064 > > Attachments: HBASE-20637-HBASE-19064-v1.patch, > HBASE-20637-HBASE-19064-v2.patch, HBASE-20637-HBASE-19064-v3.patch, > HBASE-20637-HBASE-19064.patch > > > The design in HBASE-20424 implies that, when transiting from A to S, the RS > will abort and the WAL file will be split. > But in fact, there could be races that make a RS still alive after the > transiting which causes data inconsistency. And also, a full cluster restart > is expensive, we need to find a more smooth way. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20569) NPE in RecoverStandbyProcedure.execute
[ https://issues.apache.org/jira/browse/HBASE-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-20569: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to branch HBASE-19064. Thanks [~Apache9] for reviewing. > NPE in RecoverStandbyProcedure.execute > -- > > Key: HBASE-20569 > URL: https://issues.apache.org/jira/browse/HBASE-20569 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-20569.HBASE-19064.001.patch, > HBASE-20569.HBASE-19064.002.patch, HBASE-20569.HBASE-19064.003.patch, > HBASE-20569.HBASE-19064.004.patch, HBASE-20569.HBASE-19064.005.patch, > HBASE-20569.HBASE-19064.006.patch, HBASE-20569.HBASE-19064.007.patch, > HBASE-20569.HBASE-19064.008.patch, HBASE-20569.HBASE-19064.009.patch, > HBASE-20569.HBASE-19064.010.patch, HBASE-20569.HBASE-19064.011.patch, > HBASE-20569.HBASE-19064.012.patch, HBASE-20569.HBASE-19064.013.patch, > HBASE-20569.HBASE-19064.013.patch, HBASE-20569.HBASE-19064.014.patch, > HBASE-20569.HBASE-19064.015.patch > > > We call ReplaySyncReplicationWALManager.initPeerWorkers in INIT_WORKERS state > and then use it in DISPATCH_TASKS. But if we restart the master and the > procedure is restarted from state DISPATCH_TASKS, no one will call the > initPeerWorkers method and we will get NPE. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[
https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509069#comment-16509069
]
Sean Busbey commented on HBASE-20615:
-
{quote}
-1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace.
Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
{quote}
This is an issue in the commit for HBASE-20333 and I'll fix it either on commit
or if a new patch is needed on that jira.
> emphasize use of shaded client jars when they're present in an install
> --
>
> Key: HBASE-20615
> URL: https://issues.apache.org/jira/browse/HBASE-20615
> Project: HBase
> Issue Type: Sub-task
> Components: build, Client, Usability
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20615.0.patch, HBASE-20615.1.patch,
> HBASE-20615.2.patch
>
>
> Working through setting up an IT for our shaded artifacts in HBASE-20334
> makes our lack of packaging seem like an oversight. While I could work around
> by pulling the shaded clients out of whatever build process built the
> convenience binary that we're trying to test, it seems v awkward.
> After reflecting on it more, it makes more sense to me for there to be a
> common place in the install that folks running jobs against the cluster can
> rely on. If they need to run without a full hbase install, that should still
> work fine via e.g. grabbing from the maven repo.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509068#comment-16509068
]
Sean Busbey commented on HBASE-20334:
-
the unit test failure was:
{code}
[ERROR]
org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer.org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer
[ERROR] Run 1:
TestRemoveFromSerialReplicationPeer.testRemoveSerialFlag:103->SerialReplicationTestBase.addPeer:219
» TestTimedOut
[ERROR] Run 2:
TestRemoveFromSerialReplicationPeer.org.apache.hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer
»
[ERROR] Run 3:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.setUpBeforeClass:123
» IllegalState
[ERROR] Run 4:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.tearDownAfterClass:133
» IndexOutOfBounds
[ERROR] Run 5:
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.setUpBeforeClass:123
» IllegalState
[INFO]
[ERROR]
TestRemoveFromSerialReplicationPeer.testRemoveSerialFlag:103->SerialReplicationTestBase.addPeer:219
» InterruptedIO
[ERROR]
TestRemoveFromSerialReplicationPeer>SerialReplicationTestBase.tearDown:140 »
TimeoutIO
[INFO]
[ERROR] Tests run: 3496, Failures: 0, Errors: 3, Skipped: 45
{code}
This change doesn't touch anything outside of {{dev-support}} so it couldn't
have caused this failure.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509065#comment-16509065
]
Sean Busbey commented on HBASE-20334:
-
the shellcheck warning is this:
{code}
dev-support/hbase_nightly_pseudo-distributed-test.sh:202:18: warning: hbase is
referenced but not assigned. [SC2154]
{code}
It's a false positive hitting something in the HEREDOC for the test's
log4j.properties file.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509064#comment-16509064
]
Sean Busbey edited comment on HBASE-20334 at 6/12/18 1:54 AM:
--
{quote}
-1 whitespace 0m 0s The patch has 29 line(s) that end in
whitespace. Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
-1 whitespace 0m 1s The patch 48 line(s) with tabs.
{quote}
The tabs are all in the TSV file used in the test. all but one of the
whitespace instances is also. the last one is an error in the commit for
HBASE-20333 that I'll fix either on commit or if that jira needs another patch.
was (Author: busbey):
{quote}
-1 whitespace 0m 0s The patch has 29 line(s) that end in
whitespace. Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
-1 whitespace 0m 1s The patch 48 line(s) with tabs.
{quote}
The tabas are all in the TSV file used in the test. all but one of the
whitespace instances is also. the last one is an error in the commit for
HBASE-20333 that I'll fix either on commit or if that jira needs another patch.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509064#comment-16509064
]
Sean Busbey commented on HBASE-20334:
-
{quote}
-1 whitespace 0m 0s The patch has 29 line(s) that end in
whitespace. Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
-1 whitespace 0m 1s The patch 48 line(s) with tabs.
{quote}
The tabas are all in the TSV file used in the test. all but one of the
whitespace instances is also. the last one is an error in the commit for
HBASE-20333 that I'll fix either on commit or if that jira needs another patch.
> add a test that expressly uses both our shaded client and the one from hadoop
> 3
> ---
>
> Key: HBASE-20334
> URL: https://issues.apache.org/jira/browse/HBASE-20334
> Project: HBase
> Issue Type: Sub-task
> Components: hadoop3, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Major
> Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch
>
>
> Since we're making a shaded client that bleed out of our namespace and into
> Hadoop's, we should ensure that we can show our clients coexisting. Even if
> it's just an IT that successfully talks to both us and HDFS via our
> respective shaded clients, that'd be a big help in keeping us proactive.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20700) Move meta region when server crash can cause the procedure to be stuck
[ https://issues.apache.org/jira/browse/HBASE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HBASE-20700: -- Attachment: HBASE-20700-branch-2.0-v1.patch > Move meta region when server crash can cause the procedure to be stuck > -- > > Key: HBASE-20700 > URL: https://issues.apache.org/jira/browse/HBASE-20700 > Project: HBase > Issue Type: Sub-task > Components: master, proc-v2, Region Assignment >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.1.0, 2.0.1 > > Attachments: HBASE-20700-UT.patch, HBASE-20700-branch-2.0-v1.patch, > HBASE-20700-branch-2.0-v1.patch, HBASE-20700-branch-2.0.patch, > HBASE-20700-v1.patch, HBASE-20700-v2.patch, HBASE-20700.patch > > > As said in HBASE-20682. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509063#comment-16509063 ] Sean Busbey commented on HBASE-20334: - Here's v1 running in a nightly branch: https://builds.apache.org/blue/organizations/jenkins/HBase%20Nightly/detail/HBASE-20331/43/pipeline/100 and here's the output logs of each of the three configurations that are tested: https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43/artifact/output-integration/ > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509060#comment-16509060 ] Chia-Ping Tsai commented on HBASE-20716: Perhaps the type of this issue should be "Improvement" rather than "Bug"? > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509055#comment-16509055
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #43 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/43//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[
https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509049#comment-16509049
]
Hadoop QA commented on HBASE-20334:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
5s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} shellcheck {color} | {color:red} 0m
3s{color} | {color:red} The patch generated 1 new + 33 unchanged - 11 fixed =
34 total (was 44) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 29 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch 48 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
43s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
27s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 53s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}131m 4s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 48s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.replication.TestRemoveFromSerialReplicationPeer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20334 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927387/HBASE-20334.1.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
shadedjars hadoopcheck xml compile |
| uname | Linux 214d3635f4e5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 40f0a43462 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| shellcheck |
[jira] [Commented] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[
https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509033#comment-16509033
]
Hadoop QA commented on HBASE-20615:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
3s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
21s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} The patch generated 0 new + 33 unchanged - 11 fixed
= 33 total (was 44) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
40s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
0s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 51s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}179m
31s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 7m
40s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}245m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20615 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927375/HBASE-20615.2.patch |
| Optional Tests | asflicense shellcheck shelldocs javac javadoc unit
shadedjars hadoopcheck xml compile |
| uname | Linux fb02b942fcae 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 40f0a43462 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/13199/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13199/testReport/ |
| Max. process+thread count | 5513 (vs. ulimit of 1) |
| modules | C:
[jira] [Commented] (HBASE-20674) clean up short circuit read logic and docs
[ https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509011#comment-16509011 ] Mike Drob commented on HBASE-20674: --- I'm not sure what's going on with the precommit results, I click through to testReports, but then can't get info on the failure. Not sure if it's related to my changes or not. [~busbey] is this a precommit bug? > clean up short circuit read logic and docs > -- > > Key: HBASE-20674 > URL: https://issues.apache.org/jira/browse/HBASE-20674 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Mike Drob >Assignee: Mike Drob >Priority: Major > Attachments: HBASE-20674.patch, HBASE-20674.v2.patch, > HBASE-20674.v3.patch, HBASE-20674.v4.patch, HBASE-20674.v5.patch > > > Mailing list discussion at > https://lists.apache.org/thread.html/f6f73df0ceae29f762f9b9088e3ffd0bf8f109d3dd692df100bf4fd6@%3Cdev.hbase.apache.org%3E > There are several inconsistencies between how our docs claim we do things and > how we actually do things. > There are two docs sections that attempt to address how SCR should work. > dfs.client.read.shortcircuit.skip.checksum is advised to set to true, but our > code in separate places ignores it and then later sets it to true anyway. > CommonFSUtils and FSUtils duplicate code related to SCR setup. > There is a workaround in HFileSystem for a bug that's been fixed in all > versions of hadoop that we support. (HADOOP-9307) > We suggest setting dfs.client.read.shortcircuit.buffer.size to a value that > is very close to what we'd set it to anyway, without clearly explaining why > this is important. > There are other properties that we claim are important, but we don't offer > any suggestions or explanations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20569) NPE in RecoverStandbyProcedure.execute
[ https://issues.apache.org/jira/browse/HBASE-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509006#comment-16509006 ] Duo Zhang commented on HBASE-20569: --- +1. Please fix the checkstyle issues on commit. > NPE in RecoverStandbyProcedure.execute > -- > > Key: HBASE-20569 > URL: https://issues.apache.org/jira/browse/HBASE-20569 > Project: HBase > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Guanghao Zhang >Priority: Major > Attachments: HBASE-20569.HBASE-19064.001.patch, > HBASE-20569.HBASE-19064.002.patch, HBASE-20569.HBASE-19064.003.patch, > HBASE-20569.HBASE-19064.004.patch, HBASE-20569.HBASE-19064.005.patch, > HBASE-20569.HBASE-19064.006.patch, HBASE-20569.HBASE-19064.007.patch, > HBASE-20569.HBASE-19064.008.patch, HBASE-20569.HBASE-19064.009.patch, > HBASE-20569.HBASE-19064.010.patch, HBASE-20569.HBASE-19064.011.patch, > HBASE-20569.HBASE-19064.012.patch, HBASE-20569.HBASE-19064.013.patch, > HBASE-20569.HBASE-19064.013.patch, HBASE-20569.HBASE-19064.014.patch, > HBASE-20569.HBASE-19064.015.patch > > > We call ReplaySyncReplicationWALManager.initPeerWorkers in INIT_WORKERS state > and then use it in DISPATCH_TASKS. But if we restart the master and the > procedure is restarted from state DISPATCH_TASKS, no one will call the > initPeerWorkers method and we will get NPE. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16509001#comment-16509001
]
Duo Zhang commented on HBASE-20706:
---
I'm working on HBASE-20708 which may effect the high level design of pv2 so do
not have much time on this issue.
But I think we can do something in the ReopenTableRegionsProcedure and let
other procedures use this procedure to reopen regions. In RTRP we will only
include regions in OPEN and OPENING state, and for region in OPEN state, we
will record the openSeqNum and schedule a reopen procedure for it(maybe not MRP
since it could fail and then abort the parent procedure). After the reopen
finished, we will check if the openSeqNum is changed, if not we will schedule a
new reopen procedure again. And for region in OPENING state, we need to wait
until it changed to other states, if it changed to OPEN finally then we do the
same thing above, otherwise we can just drop it.
Just a rough design, may still have some problems, or some difficulties when
implementing, for example, how to wait for a region state change. But I think
we need to make sure that the region has loaded the new configurations. MRP is
not designed to be critical as it is designed to be used by LoadBalancer...
Thanks.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508997#comment-16508997
]
Duo Zhang commented on HBASE-20706:
---
Only include region in OPEN state for reopening in MTP is not correct. As said
above at least you also need to include the region in OPENING state, and we
also need to check other states to see if there are some states which the
region may have already been initialized at RS side.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20658) Release latch early for ModifyTable and TruncateTable procedure.
[ https://issues.apache.org/jira/browse/HBASE-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508988#comment-16508988 ] Ankit Singhal commented on HBASE-20658: --- bq. Truncate was a synchronous operation Yes, just checked that 1.x expects it to be synchronous. bq. However, modify table was an async operation so that reason doesn't hold for it. However, as you see, we added the synchronous lock for HBASE-19953. I did leave a comment which might be relevant. The comment actually talks about releasing a latch after performing some pre-check which is true for all other procedures except these two(where the latch is released at the end), so was not sure that it was an oversight or done purposely? > Release latch early for ModifyTable and TruncateTable procedure. > > > Key: HBASE-20658 > URL: https://issues.apache.org/jira/browse/HBASE-20658 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > > It seems we are releasing latch in Modify and Truncate table procedures > during completion cleanup which make the whole procedure to execute > synchronously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[
https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508976#comment-16508976
]
Hadoop QA commented on HBASE-19121:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
0s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
16s{color} | {color:red} hbase-server: The patch generated 55 new + 202
unchanged - 7 fixed = 257 total (was 209) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} shadedjars {color} | {color:red} 3m
40s{color} | {color:red} patch has 10 errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 11s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m
26s{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed =
1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}125m 36s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
20s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}167m 19s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Boxing/unboxing to parse a primitive
org.apache.hadoop.hbase.util.HBaseFsck2.processOptions(CommandLine) At
HBaseFsck2.java:org.apache.hadoop.hbase.util.HBaseFsck2.processOptions(CommandLine)
At HBaseFsck2.java:[line 332] |
| Failed junit tests | hadoop.hbase.util.TestHBaseFsck2 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-19121 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927372/hbase-19121.master.001.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux dfbac7674ad7 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 40f0a43462 |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs |
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[
https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508899#comment-16508899
]
Hadoop QA commented on HBASE-20695:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 5m
20s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
47s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 56s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}109m
17s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}151m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20695 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927364/HBASE-20695.master.004.patch
|
| Optional Tests | asflicense javac javadoc unit findbugs shadedjars
hadoopcheck hbaseanti checkstyle compile |
| uname | Linux ff9b28872bc9 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / b25c782f0d |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC3 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13197/testReport/ |
| Max. process+thread count | 4576 (vs. ulimit of 1) |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/13197/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Implement table level
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508893#comment-16508893
]
stack commented on HBASE-20706:
---
Oh, yeah, after fixing unit tests. +1 for branch-2.0 too please.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508891#comment-16508891
]
stack commented on HBASE-20706:
---
+1 from me. This is a fix to MTP IMO.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20656) Validate pre-2.0 coprocessors against HBase 2.0+
[
https://issues.apache.org/jira/browse/HBASE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508884#comment-16508884
]
Hudson commented on HBASE-20656:
Results for branch branch-2.0
[build #416 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/416/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/416//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/416//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/416//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Validate pre-2.0 coprocessors against HBase 2.0+
>
>
> Key: HBASE-20656
> URL: https://issues.apache.org/jira/browse/HBASE-20656
> Project: HBase
> Issue Type: New Feature
> Components: tooling
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20656.master.001.patch,
> HBASE-20656.master.002.patch, HBASE-20656.master.003.patch,
> HBASE-20656.master.004.patch, HBASE-20656.master.005.patch,
> HBASE-20656.master.006.patch, HBASE-20656.master.007.patch,
> HBASE-20656.master.008.patch
>
>
> We have co-processors for a while, but the API has been changed recently. We
> should give some tooling for our users to determine if they can use the
> previous co-processors safely or not.
> The tool should:
> - try to load the co-processors on our current classpath for ensuring class
> references are on our classpath,
> - should check for previously removed co-processor methods.
> In this version we check only method signatures. Code references should be
> checked in further versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-6028 started by Mohit Goel. - > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508861#comment-16508861
]
Josh Elser commented on HBASE-20706:
Just to make sure we work forward, I'd suggest that we commit this one. Have
had our internal battery of tests run against it for a few days now. Haven't
noticed any fallout.
I still need to fix the two UT failures. Would appreciate thinking about this
fix in isolation of other MRP/MTP/etc fixes we know we want to make.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20707) Move MissingSwitchDefault check from checkstyle to error-prone
[
https://issues.apache.org/jira/browse/HBASE-20707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508851#comment-16508851
]
Hudson commented on HBASE-20707:
Results for branch branch-2
[build #851 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Move MissingSwitchDefault check from checkstyle to error-prone
> --
>
> Key: HBASE-20707
> URL: https://issues.apache.org/jira/browse/HBASE-20707
> Project: HBase
> Issue Type: Bug
> Components: build
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20707.patch
>
>
> Both checkstyle and error-prone can alert when a switch statement is missing
> a default. However, because checkstyle does it via static analysis and
> error-prone does it during compilation, e-p can detect when all cases of an
> enum have been covered, and will _not_ warn about the needed default case.
> In fact, checkstyle explicitly mentions in their docs that even if you cover
> all enum cases now, you should still have a default label because the enum
> could change in the future. Which seems silly to me, because your analysis
> tools should still be running in the future and would catch it then.
> Se we should migrate the check from checkstyle to a slightly smarter
> error-prone check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20656) Validate pre-2.0 coprocessors against HBase 2.0+
[
https://issues.apache.org/jira/browse/HBASE-20656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508852#comment-16508852
]
Hudson commented on HBASE-20656:
Results for branch branch-2
[build #851 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Validate pre-2.0 coprocessors against HBase 2.0+
>
>
> Key: HBASE-20656
> URL: https://issues.apache.org/jira/browse/HBASE-20656
> Project: HBase
> Issue Type: New Feature
> Components: tooling
>Affects Versions: 3.0.0
>Reporter: Balazs Meszaros
>Assignee: Balazs Meszaros
>Priority: Major
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20656.master.001.patch,
> HBASE-20656.master.002.patch, HBASE-20656.master.003.patch,
> HBASE-20656.master.004.patch, HBASE-20656.master.005.patch,
> HBASE-20656.master.006.patch, HBASE-20656.master.007.patch,
> HBASE-20656.master.008.patch
>
>
> We have co-processors for a while, but the API has been changed recently. We
> should give some tooling for our users to determine if they can use the
> previous co-processors safely or not.
> The tool should:
> - try to load the co-processors on our current classpath for ensuring class
> references are on our classpath,
> - should check for previously removed co-processor methods.
> In this version we check only method signatures. Code references should be
> checked in further versions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508853#comment-16508853
]
Hudson commented on HBASE-19852:
Results for branch branch-2
[build #851 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/851//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch,
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch,
> HBASE-19852.master.009.patch, HBASE-19852.master.010.patch,
> HBASE-19852.master.011.patch, HBASE-19852.master.012.patch,
> HBASE-19852.master.013.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Status: Patch Available (was: In Progress) v1 - rebased master + patches for HBASE-20332, HBASE-20333, HBASE-19735, and HBASE-20615 - adds a support script for automating caching apache project artifacts - updates Jenkinsfile to use said caching script ot get a copy of hadoop 2 and hadoop 3 - updates our "source tarball" test to be general "packaging and integration with Hadoop" - adds use of test script with hadoop 2, hadoop 3, and hadoop 3 w/hadoop's client artifacts - updated handling of args to the test script so that we work with relative paths - updated for feedback from v0: does file checks for passed jars. > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-13583) AOT compile our JRuby
[ https://issues.apache.org/jira/browse/HBASE-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508830#comment-16508830 ] Mike Drob commented on HBASE-13583: --- Chatted a bunch with the JRuby team on IRC, apparently jrubyc isn't currently as useful as we imagined it to be. bq. right now it doesn't _really_ even compile to bytecode; it's just a .class file containing a serialized version of our internal representation which means we'll get neither startup speedup or java method invocation validation. > AOT compile our JRuby > - > > Key: HBASE-13583 > URL: https://issues.apache.org/jira/browse/HBASE-13583 > Project: HBase > Issue Type: Improvement > Components: build, scripts, shell >Reporter: Nick Dimiduk >Priority: Major > Attachments: HBASE-13583.patch, HBASE-13583.v2.patch, > HBASE-13583.v3.patch, HBASE-13583.v4.patch > > > Our Jruby code seems to not keep up well with Java changes. We should > investigate adding a compilation step for our shell and the rb scripts in bin > to ensure they're calling methods that exist on classes that exist. This > looks like as good a starting point as any: > https://github.com/jruby/jruby/wiki/GeneratingJavaClasses -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20334) add a test that expressly uses both our shaded client and the one from hadoop 3
[ https://issues.apache.org/jira/browse/HBASE-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20334: Attachment: HBASE-20334.1.patch > add a test that expressly uses both our shaded client and the one from hadoop > 3 > --- > > Key: HBASE-20334 > URL: https://issues.apache.org/jira/browse/HBASE-20334 > Project: HBase > Issue Type: Sub-task > Components: hadoop3, shading >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Attachments: HBASE-20334.0.patch, HBASE-20334.1.patch > > > Since we're making a shaded client that bleed out of our namespace and into > Hadoop's, we should ensure that we can show our clients coexisting. Even if > it's just an IT that successfully talks to both us and HDFS via our > respective shaded clients, that'd be a big help in keeping us proactive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508604#comment-16508604 ] Ankit Jain edited comment on HBASE-20672 at 6/11/18 9:51 PM: - Hey [~stack], Thank you for your comments. The intention behind these new metrics is to get a more real time understanding about the amount of read and write operations done by region server. The current metrics ReadRequestCount and WriteRequestCount are essentially values incremented from the start of the service and hence it makes it difficult to analyze the current operation load on the region server. Regarding your concern about extra overhead involved, for this patch we have not added any extra counters and are reusing the already existing ones. For populating the values for the rates, we are performing basic arithmetic operations using the values returned by preexisting counters. Currently, metric calculation happens after every 5000 ms at region server level (DEFAULT_REGIONSERVER_METRICS_PERIOD = 5000. So that is what is currently being considered as monitoring interval. So I guess it is just the monitoring system thing and is not specifically related to Hbase. I hope my comments answers your questions. Do let me know if any other concerns. was (Author: jain.ankit): Hey [~stack], Thank you for your comments. The intention behind these new metrics is to get a more real time understanding about the amount of read and write operations done by region server. The current metrics ReadRequestCount and WriteRequestCount are essentially values incremented from the start of the service and hence it makes it difficult to analyze the current operation load on the region server. Regarding your concern about extra overhead involved, for this patch we have not added any extra counters and are reusing the already existing ones. For populating the values for the rates, we are performing basic arithmetic operations using the values returned by preexisting counters. Currently, metric calculation happens after every 5000 ms at region server level (DEFAULT_REGIONSERVER_METRICS_PERIOD = 5000;). So that is what is currently being considered as monitoring interval. So I guess it is just the monitoring system thing and is not specifically related to Hbase. I hope my comments answers your questions. Do let me know if any other concerns. > Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at > every monitoring interval > - > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-20672: > Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at > every monitoring interval > - > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20642) IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException
[
https://issues.apache.org/jira/browse/HBASE-20642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508792#comment-16508792
]
Josh Elser commented on HBASE-20642:
{quote}if we fix HBASE-20658 by releasing the latch after some pre-checks for
Modify and Truncate table , then probably we may not need to do nonce check as
retry mechanism will not kick in if the procedure is submitted successfully.
{quote}
I think in both cases, we don't want to release those latches early (would
break functionality of Truncate, and regress on HBASE-19953 for modify table).
However, using the same nonce from the client seems like the right fix to me.
We have _one_ client and this client is (unwittingly) submitting the same
procedure multiple times. I don't think I've seen any reason that this should
be using a new nonce – so let's make the nonce help us in this known-deficient
case :)
> IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException
> -
>
> Key: HBASE-20642
> URL: https://issues.apache.org/jira/browse/HBASE-20642
> Project: HBase
> Issue Type: Bug
>Reporter: Ankit Singhal
>Assignee: Ankit Singhal
>Priority: Major
> Attachments: HBASE-20642.patch
>
>
> [~romil.choksi] reported that IntegrationTestDDLMasterFailover is failing
> while adding column family during the time master is restarting.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20658) Release latch early for ModifyTable and TruncateTable procedure.
[ https://issues.apache.org/jira/browse/HBASE-20658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508784#comment-16508784 ] Josh Elser commented on HBASE-20658: Yeah, I think that's what Duo mean, [~an...@apache.org]. Truncate was a synchronous operation, so, if we released the latch early, I think a 1.x client would return too fast. However, modify table was an async operation so that reason doesn't hold for it. However, as you see, we added the synchronous lock for HBASE-19953. I did leave a [comment|https://issues.apache.org/jira/browse/HBASE-19953?focusedCommentId=16370671=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16370671] which might be relevant. > Release latch early for ModifyTable and TruncateTable procedure. > > > Key: HBASE-20658 > URL: https://issues.apache.org/jira/browse/HBASE-20658 > Project: HBase > Issue Type: Bug >Reporter: Ankit Singhal >Assignee: Ankit Singhal >Priority: Major > > It seems we are releasing latch in Modify and Truncate table procedures > during completion cleanup which make the whole procedure to execute > synchronously. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508759#comment-16508759 ] stack commented on HBASE-20672: --- bq, The current metrics ReadRequestCount and WriteRequestCount are essentially values incremented from the start of the service and hence it makes it difficult to analyze the current operation load on the region server. Why is that? What is wrong with taking the current count, letting time pass, and then taking measure again? What the interval to measure on is up to you (or your monitoring system). For example, What is your objective with this change? You do not seem to be showing the new metrics anywhere in hbase UI, is that so? So what is the change for? Thanks. Oh, in case it helps, here is why I am having trouble with this change... Here is a graph that shows hit rates using current counters... TSDB is doing the per second for me. !hits1vs2.4.40.400.png! > Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at > every monitoring interval > - > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-20672: -- Attachment: hits1vs2.4.40.400.png > Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at > every monitoring interval > - > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch, hits1vs2.4.40.400.png > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508743#comment-16508743
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #42 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/42/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/42//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/42//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/42//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/42//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone
[
https://issues.apache.org/jira/browse/HBASE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508737#comment-16508737
]
Sean Busbey commented on HBASE-20333:
-
{code}
-1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace.
Use git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply
{code}
I'll fix this on commit or if a new iteration needs to be posted.
> break up shaded client into one with no Hadoop and one that's standalone
>
>
> Key: HBASE-20333
> URL: https://issues.apache.org/jira/browse/HBASE-20333
> Project: HBase
> Issue Type: Sub-task
> Components: shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20333.1.patch, HBASE-20333.WIP.0.patch
>
>
> there are contexts where we want to stay out of our downstream users way wrt
> dependencies, but they need more Hadoop classes than we provide. i.e. any
> downstream client that wants to use both HBase and HDFS in their application,
> or any non-MR YARN application.
> Now that Hadoop also has shaded client artifacts for Hadoop 3, we're also
> providing less incremental benefit by including our own rewritten Hadoop
> classes to avoid downstream needing to pull in all of Hadoop's transitive
> dependencies.
> right now those users need to ensure that any jars from the Hadoop project
> are loaded in the classpath prior to our shaded client jar. This is brittle
> and prone to weird debugging trouble.
> instead, we should have two artifacts: one that just lists Hadoop as a
> prerequisite and one that still includes the rewritten-but-not-relocated
> Hadoop classes.
> We can then use docs to emphasize when each of these is appropriate to use.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-19121: - Status: Patch Available (was: Open) > HBCK for AMv2 (A.K.A HBCK2) > --- > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: stack >Assignee: Umesh Agashe >Priority: Major > Attachments: hbase-19121.master.001.patch > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe reassigned HBASE-19121: Assignee: Umesh Agashe > HBCK for AMv2 (A.K.A HBCK2) > --- > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: stack >Assignee: Umesh Agashe >Priority: Major > Attachments: hbase-19121.master.001.patch > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[ https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20615: Status: Patch Available (was: In Progress) v2 - same as v1 except we grab the shaded jars as they exist in the tarball rather than specifically looking for those from version 3.0.0-SNAPSHOT > emphasize use of shaded client jars when they're present in an install > -- > > Key: HBASE-20615 > URL: https://issues.apache.org/jira/browse/HBASE-20615 > Project: HBase > Issue Type: Sub-task > Components: build, Client, Usability >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20615.0.patch, HBASE-20615.1.patch, > HBASE-20615.2.patch > > > Working through setting up an IT for our shaded artifacts in HBASE-20334 > makes our lack of packaging seem like an oversight. While I could work around > by pulling the shaded clients out of whatever build process built the > convenience binary that we're trying to test, it seems v awkward. > After reflecting on it more, it makes more sense to me for there to be a > common place in the install that folks running jobs against the cluster can > rely on. If they need to run without a full hbase install, that should still > work fine via e.g. grabbing from the maven repo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[ https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-20615: Attachment: HBASE-20615.2.patch > emphasize use of shaded client jars when they're present in an install > -- > > Key: HBASE-20615 > URL: https://issues.apache.org/jira/browse/HBASE-20615 > Project: HBase > Issue Type: Sub-task > Components: build, Client, Usability >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20615.0.patch, HBASE-20615.1.patch, > HBASE-20615.2.patch > > > Working through setting up an IT for our shaded artifacts in HBASE-20334 > makes our lack of packaging seem like an oversight. While I could work around > by pulling the shaded clients out of whatever build process built the > convenience binary that we're trying to test, it seems v awkward. > After reflecting on it more, it makes more sense to me for there to be a > common place in the install that folks running jobs against the cluster can > rely on. If they need to run without a full hbase install, that should still > work fine via e.g. grabbing from the maven repo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508721#comment-16508721 ] Umesh Agashe commented on HBASE-19121: -- HBCK2 will evolve. First version with basic command line options and parsing is in 001 patch. It also has action to FixAssignments. > HBCK for AMv2 (A.K.A HBCK2) > --- > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: stack >Priority: Major > Attachments: hbase-19121.master.001.patch > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19121) HBCK for AMv2 (A.K.A HBCK2)
[ https://issues.apache.org/jira/browse/HBASE-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-19121: - Attachment: hbase-19121.master.001.patch > HBCK for AMv2 (A.K.A HBCK2) > --- > > Key: HBASE-19121 > URL: https://issues.apache.org/jira/browse/HBASE-19121 > Project: HBase > Issue Type: Bug > Components: hbck >Reporter: stack >Priority: Major > Attachments: hbase-19121.master.001.patch > > > We don't have an hbck for the new AM. Old hbck may actually do damage going > against AMv2. > Fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20715) REST curl command for create namespace with "Content-Type: text/xml" specified (mistakenly) results in a 400 Bad Request
[
https://issues.apache.org/jira/browse/HBASE-20715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508685#comment-16508685
]
Josh Elser commented on HBASE-20715:
[~nihaljain.cs], do you know what the default Content-Type is assumed to be
when you don't supply it? Or, how the code path differs?
I can appreciated the HTTP/400 being thrown since your request didn't provide
any XML.
Is there specific documentation somewhere that you think should be updated?
> REST curl command for create namespace with "Content-Type: text/xml"
> specified (mistakenly) results in a 400 Bad Request
>
>
> Key: HBASE-20715
> URL: https://issues.apache.org/jira/browse/HBASE-20715
> Project: HBase
> Issue Type: Task
> Components: documentation, REST
>Reporter: Nihal Jain
>Priority: Minor
>
> I have a habit of setting "Content-Type: text/xml" for curl commands. Today,
> I had a bad time debugging the 400 error for below command. Finally, this
> discussion(https://issues.apache.org/jira/browse/HBASE-14147?focusedCommentId=15205420=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15205420)
> helped me in resolving the problem.
> As pointed out by [~mwarhaftig] and [~misty] in their discussion in
> HBASE-14147, a curl request of the following form will throw a {{400 BAD
> REQUEST}}
> {code:java}
> curl -vi -X POST -H "Accept: text/xml" -H "Content-Type: text/xml"
> "http://my_server:20550/namespaces/new_ns2/;
> {code}
> I think we should document this in the description for "Create a new
> namespace" explicitly mentioning that users should take care "Content-Type"
> is not set for this particular command. Similarly for "Alter an existing
> namespace" command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20635) Support to convert the shaded user permission proto to client user permission object
[
https://issues.apache.org/jira/browse/HBASE-20635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-20635:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master for now. Thanks Rajeshbabu!
> Support to convert the shaded user permission proto to client user permission
> object
>
>
> Key: HBASE-20635
> URL: https://issues.apache.org/jira/browse/HBASE-20635
> Project: HBase
> Issue Type: Bug
>Reporter: Rajeshbabu Chintaguntla
>Assignee: Rajeshbabu Chintaguntla
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20635.patch, HBASE-20635_v2.patch,
> PHOENIX-4528_5.x-HBase-2.0_v2.patch
>
>
> Currently we have API to build the protobuf UserPermission to client user
> permission in AccessControlUtil but we cannot do the same when we use shaded
> protobufs.
> {noformat}
> /**
>* Converts a user permission proto to a client user permission object.
>*
>* @param proto the protobuf UserPermission
>* @return the converted UserPermission
>*/
> public static UserPermission
> toUserPermission(AccessControlProtos.UserPermission proto) {
> return new UserPermission(proto.getUser().toByteArray(),
> toTablePermission(proto.getPermission()));
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508662#comment-16508662 ] Xu Cang commented on HBASE-20695: - Addressed your review in new version. Thanks again for reviewing the patch. [~zghaobac] > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-20695) Implement table level RegionServer replication metrics
[ https://issues.apache.org/jira/browse/HBASE-20695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Cang updated HBASE-20695: Attachment: HBASE-20695.master.004.patch > Implement table level RegionServer replication metrics > --- > > Key: HBASE-20695 > URL: https://issues.apache.org/jira/browse/HBASE-20695 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Xu Cang >Assignee: Xu Cang >Priority: Minor > Attachments: HBASE-20695.master.001.patch, > HBASE-20695.master.002.patch, HBASE-20695.master.003.patch, > HBASE-20695.master.004.patch > > > Region server metrics now are mainly global metrics. It would be nice to have > table level metrics such as table level source.AgeOfLastShippedOp to indicate > operators which table's replication is lagging behind. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508648#comment-16508648 ] stack commented on HBASE-6028: -- I added you as a contributor and assigned you the issue [~mogoel] (presuming it ok w/ [~esteban]). > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-6028: Assignee: Mohit Goel > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Mohit Goel >Priority: Minor > Labels: beginner > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack reassigned HBASE-6028: Assignee: (was: Esteban Gutierrez) > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Priority: Minor > Labels: beginner > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20333) break up shaded client into one with no Hadoop and one that's standalone
[
https://issues.apache.org/jira/browse/HBASE-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508634#comment-16508634
]
Hadoop QA commented on HBASE-20333:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
1s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
44s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
28s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
22s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}186m
33s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}255m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20333 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927320/HBASE-20333.1.patch |
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile shellcheck shelldocs |
| uname | Linux e6b75353d1b5 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / eb13cdd7ed |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/13193/artifact/patchprocess/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13193/testReport/ |
| Max. process+thread count | 4420 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle hbase-common
[jira] [Commented] (HBASE-20331) clean up shaded packaging for 2.1
[
https://issues.apache.org/jira/browse/HBASE-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508631#comment-16508631
]
Hudson commented on HBASE-20331:
Results for branch HBASE-20331
[build #41 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/41/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/41//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/41//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/41//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
--Failed when running client tests on top of Hadoop 2. [see log for
details|https://builds.apache.org/job/HBase%20Nightly/job/HBASE-20331/41//artifact/output-integration/hadoop-2.log].
(note that this means we didn't run on Hadoop 3)
> clean up shaded packaging for 2.1
> -
>
> Key: HBASE-20331
> URL: https://issues.apache.org/jira/browse/HBASE-20331
> Project: HBase
> Issue Type: Umbrella
> Components: Client, mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
>
> polishing pass on shaded modules for 2.0 based on trying to use them in more
> contexts.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508629#comment-16508629
]
Hadoop QA commented on HBASE-20332:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
14s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m
0s{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
59s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
26s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 3m
53s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
9m 59s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}183m
54s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 8m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}253m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20332 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927319/HBASE-20332.4.patch |
| Optional Tests | asflicense javac javadoc unit shadedjars hadoopcheck
xml compile shellcheck shelldocs |
| uname | Linux 81436e2f74de 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / eb13cdd7ed |
| maven | version: Apache Maven 3.5.3
(3383c37e1f9e9b3bc3df5050c29c8aff9f295297; 2018-02-24T19:49:05Z) |
| Default Java | 1.8.0_171 |
| shellcheck | v0.4.4 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/13194/testReport/ |
| Max. process+thread count | 4557 (vs. ulimit of 1) |
| modules | C: hbase-checkstyle hbase-common hbase-hadoop2-compat hbase-client
hbase-replication hbase-server hbase-mapreduce hbase-testing-util hbase-thrift
hbase-rsgroup hbase-shell hbase-endpoint hbase-backup hbase-it hbase-rest
hbase-examples
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508625#comment-16508625 ] stack commented on HBASE-20716: --- bq. beginner? consider me dubious about that classification, Stack. Dunno. Just replacing one usage w/ another? Will I change it [~mdrob]? > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20716) Unsafe access cleanup
[ https://issues.apache.org/jira/browse/HBASE-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508617#comment-16508617 ] Mike Drob commented on HBASE-20716: --- beginner? consider me dubious about that classification, Stack. > Unsafe access cleanup > - > > Key: HBASE-20716 > URL: https://issues.apache.org/jira/browse/HBASE-20716 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Priority: Minor > Labels: beginner > > We have two means of getting at unsafe; UnsafeAccess and then internal to the > Bytes class. They are effectively doing the same thing. We should have one > avenue to Unsafe only. > Many of our paths to Unsafe via UnsafeAccess traverse flags to check if > access is available, if it is aligned and the order in which words are > written on the machine. Each check costs -- especially if done millions of > times a second -- and on occasion adds bloat in hot code paths. The unsafe > access inside Bytes checks on startup what the machine is capable off and > then does a static assign of the appropriate class-to-use from there on out. > UnsafeAccess does not do this running the checks everytime. Would be good to > have the Bytes behavior pervasive. > The benefit of one access to Unsafe only is plain. The benefits we gain > removing checks will be harder to measure though should be plain when you > disassemble a hot-path; in a (very) rare case, the saved byte codes could be > the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20672) Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at every monitoring interval
[ https://issues.apache.org/jira/browse/HBASE-20672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508604#comment-16508604 ] Ankit Jain commented on HBASE-20672: Hey [~stack], Thank you for your comments. The intention behind these new metrics is to get a more real time understanding about the amount of read and write operations done by region server. The current metrics ReadRequestCount and WriteRequestCount are essentially values incremented from the start of the service and hence it makes it difficult to analyze the current operation load on the region server. Regarding your concern about extra overhead involved, for this patch we have not added any extra counters and are reusing the already existing ones. For populating the values for the rates, we are performing basic arithmetic operations using the values returned by preexisting counters. Currently, metric calculation happens after every 5000 ms at region server level (DEFAULT_REGIONSERVER_METRICS_PERIOD = 5000;). So that is what is currently being considered as monitoring interval. So I guess it is just the monitoring system thing and is not specifically related to Hbase. I hope my comments answers your questions. Do let me know if any other concerns. > Create new HBase metrics ReadRequestRate and WriteRequestRate that reset at > every monitoring interval > - > > Key: HBASE-20672 > URL: https://issues.apache.org/jira/browse/HBASE-20672 > Project: HBase > Issue Type: Improvement > Components: metrics >Reporter: Ankit Jain >Assignee: Ankit Jain >Priority: Minor > Fix For: 3.0.0 > > Attachments: HBASE-20672.branch-1.001.patch, > HBASE-20672.master.001.patch, HBASE-20672.master.002.patch, > HBASE-20672.master.003.patch > > > Hbase currently provides counter read/write requests (ReadRequestCount, > WriteRequestCount). That said it is not easy to use counter that reset only > after a restart of the service, we would like to expose 2 new metrics in > HBase to provide ReadRequestRate and WriteRequestRate at region server level. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20716) Unsafe access cleanup
stack created HBASE-20716: - Summary: Unsafe access cleanup Key: HBASE-20716 URL: https://issues.apache.org/jira/browse/HBASE-20716 Project: HBase Issue Type: Bug Components: Performance Reporter: stack We have two means of getting at unsafe; UnsafeAccess and then internal to the Bytes class. They are effectively doing the same thing. We should have one avenue to Unsafe only. Many of our paths to Unsafe via UnsafeAccess traverse flags to check if access is available, if it is aligned and the order in which words are written on the machine. Each check costs -- especially if done millions of times a second -- and on occasion adds bloat in hot code paths. The unsafe access inside Bytes checks on startup what the machine is capable off and then does a static assign of the appropriate class-to-use from there on out. UnsafeAccess does not do this running the checks everytime. Would be good to have the Bytes behavior pervasive. The benefit of one access to Unsafe only is plain. The benefits we gain removing checks will be harder to measure though should be plain when you disassemble a hot-path; in a (very) rare case, the saved byte codes could be the difference between inlining or not. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HBASE-19323) Make netty engine default in hbase2
[ https://issues.apache.org/jira/browse/HBASE-19323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-19323: -- Release Note: NettyRpcServer is now our default RPC server replacing SimpleRpcServer. To set the rpc server back to SimpleRpcServer, set 'hbase.rpc.server.impl' to org.apache.hadoop.hbase.ipc.SimpleRpcServer was:NettyRpcServer is now our default RPC server replacing SimpleRpcServer. > Make netty engine default in hbase2 > --- > > Key: HBASE-19323 > URL: https://issues.apache.org/jira/browse/HBASE-19323 > Project: HBase > Issue Type: Task > Components: rpc >Reporter: stack >Assignee: stack >Priority: Major > Fix For: 2.0.0-beta-1, 2.0.0 > > Attachments: > 0001-HBASE-19323-Make-netty-engine-default-in-hbase2.patch, > HBASE-19323.master.001.patch > > > HBASE-17263 added netty rpc server. This issue is about making it default > given it has seen good service across two singles-days at scale. Netty > handles the scenario seen in HBASE-19320 (See tail of HBASE-19320 for > suggestion to netty the default) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[ https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508550#comment-16508550 ] Sean Busbey commented on HBASE-20615: - lemme get v2 up. I forgot to replace the placeholder versions for hbase jars. > emphasize use of shaded client jars when they're present in an install > -- > > Key: HBASE-20615 > URL: https://issues.apache.org/jira/browse/HBASE-20615 > Project: HBase > Issue Type: Sub-task > Components: build, Client, Usability >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20615.0.patch, HBASE-20615.1.patch > > > Working through setting up an IT for our shaded artifacts in HBASE-20334 > makes our lack of packaging seem like an oversight. While I could work around > by pulling the shaded clients out of whatever build process built the > convenience binary that we're trying to test, it seems v awkward. > After reflecting on it more, it makes more sense to me for there to be a > common place in the install that folks running jobs against the cluster can > rely on. If they need to run without a full hbase install, that should still > work fine via e.g. grabbing from the maven repo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20674) clean up short circuit read logic and docs
[
https://issues.apache.org/jira/browse/HBASE-20674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508547#comment-16508547
]
Hadoop QA commented on HBASE-20674:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
59s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
5s{color} | {color:blue} branch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
56s{color} | {color:green} root: The patch generated 0 new + 161 unchanged - 13
fixed = 161 total (was 174) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} refguide {color} | {color:blue} 4m
12s{color} | {color:blue} patch has no errors when building the reference
guide. See footer for rendered docs, which you should manually inspect. {color}
|
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
17s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
8m 50s{color} | {color:green} Patch does not cause any errors with Hadoop 2.7.4
or 3.0.0. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
52s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}123m 54s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue |
[jira] [Commented] (HBASE-6028) Implement a cancel for in-progress compactions
[ https://issues.apache.org/jira/browse/HBASE-6028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508499#comment-16508499 ] Mohit Goel commented on HBASE-6028: --- Hi [~esteban] , I started looking into it with [~stack]. Should I assign it to myself ? > Implement a cancel for in-progress compactions > -- > > Key: HBASE-6028 > URL: https://issues.apache.org/jira/browse/HBASE-6028 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Derek Wollenstein >Assignee: Esteban Gutierrez >Priority: Minor > Labels: beginner > > Depending on current server load, it can be extremely expensive to run > periodic minor / major compactions. It would be helpful to have a feature > where a user could use the shell or a client tool to explicitly cancel an > in-progress compactions. This would allow a system to recover when too many > regions became eligible for compactions at once -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508456#comment-16508456
]
Sean Busbey commented on HBASE-20332:
-
{quote}
Since we're removing the exclusion in a bunch of places, we need to make sure
folks don't inadvertently rely on it as an unlisted dependency. We want to make
sure that when folks reference HTrace they're referencing HTrace v4 classes and
not earlier classes. so for now we tell checkstyle to flag org.htrace and
org.apache.htrace (I don't know of a way to flag that use of
org.apache.htrace.core specifically is fine).
{quote}
In a prior version of this patch I did this as an "hbase antipattern" check
that was able to use a regex to say importing {{org.apache.htrace.core}} and
subpackages is fine. I could switch back to that if y'all prefer.
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20332) shaded mapreduce module shouldn't include hadoop
[
https://issues.apache.org/jira/browse/HBASE-20332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508452#comment-16508452
]
Sean Busbey commented on HBASE-20332:
-
removing hadoop from the shaded mapreduce module means removing the htrace v3
version that was included as a transitive dependency of it (since we only need
v4). Part of doing that properly required cleaning up how we handle htrace v3
coming in as a transitive of hadoop in other places. A big part of that is
removing exclusions so that we get it when Hadoop needs it for e.g. tests
(rather than, for example, expressly listing it as a compile scope dependency
of hbase-server that we don't actually need).
Since we're removing the exclusion in a bunch of places, we need to make sure
folks don't inadvertently rely on it as an unlisted dependency. We want to
make sure that when folks reference HTrace they're referencing HTrace v4
classes and not earlier classes. so for now we tell checkstyle to flag
{{org.htrace}} and {{org.apache.htrace}} (I don't know of a way to flag that
use of {{org.apache.htrace.core}} specifically is fine).
> shaded mapreduce module shouldn't include hadoop
>
>
> Key: HBASE-20332
> URL: https://issues.apache.org/jira/browse/HBASE-20332
> Project: HBase
> Issue Type: Sub-task
> Components: mapreduce, shading
>Affects Versions: 2.0.0
>Reporter: Sean Busbey
>Assignee: Sean Busbey
>Priority: Critical
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-20332.0.patch, HBASE-20332.1.WIP.patch,
> HBASE-20332.2.WIP.patch, HBASE-20332.3.patch, HBASE-20332.4.patch
>
>
> AFAICT, we should just entirely skip including hadoop in our shaded mapreduce
> module
> 1) Folks expect to run yarn / mr apps via {{hadoop jar}} / {{yarn jar}}
> 2) those commands include all the needed Hadoop jars in your classpath by
> default (both client side and in the containers)
> 3) If you try to use "user classpath first" for your job as a workaround
> (e.g. for some library your application needs that hadoop provides) then our
> inclusion of *some but not all* hadoop classes then causes everything to fall
> over because of mixing rewritten and non-rewritten hadoop classes
> 4) if you don't use "user classpath first" then all of our
> non-relocated-but-still-shaded hadoop classes are ignored anyways so we're
> just wasting space
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19852:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to branch-2 and master. Thanks for your diligent work, Kevin!
Doesn't look like it comes back to 1.x at all. We can consider that if you want
to make some patches for that.
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch,
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch,
> HBASE-19852.master.009.patch, HBASE-19852.master.010.patch,
> HBASE-19852.master.011.patch, HBASE-19852.master.012.patch,
> HBASE-19852.master.013.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508364#comment-16508364
]
Josh Elser commented on HBASE-19852:
I'm presently getting this into branch-2 and master (if not some 1.x stuff too)
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch,
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch,
> HBASE-19852.master.009.patch, HBASE-19852.master.010.patch,
> HBASE-19852.master.011.patch, HBASE-19852.master.012.patch,
> HBASE-19852.master.013.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-19852) HBase Thrift 1 server SPNEGO Improvements
[
https://issues.apache.org/jira/browse/HBASE-19852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-19852:
---
Fix Version/s: 2.1.0
3.0.0
> HBase Thrift 1 server SPNEGO Improvements
> -
>
> Key: HBASE-19852
> URL: https://issues.apache.org/jira/browse/HBASE-19852
> Project: HBase
> Issue Type: Improvement
> Components: Thrift
>Reporter: Kevin Risden
>Assignee: Kevin Risden
>Priority: Major
> Fix For: 3.0.0, 2.1.0
>
> Attachments: HBASE-19852.master.001.patch,
> HBASE-19852.master.002.patch, HBASE-19852.master.003.patch,
> HBASE-19852.master.004.patch, HBASE-19852.master.006.patch,
> HBASE-19852.master.007.patch.txt, HBASE-19852.master.008.patch,
> HBASE-19852.master.009.patch, HBASE-19852.master.010.patch,
> HBASE-19852.master.011.patch, HBASE-19852.master.012.patch,
> HBASE-19852.master.013.patch
>
>
> HBase Thrift1 server has some issues when trying to use SPNEGO.
> From mailing list:
> http://mail-archives.apache.org/mod_mbox/hbase-user/201801.mbox/%3CCAJU9nmh5YtZ%2BmAQSLo91yKm8pRVzAPNLBU9vdVMCcxHRtRqgoA%40mail.gmail.com%3E
> {quote}While setting up the HBase Thrift server with HTTP, there were a
> significant amount of 401 errors where the HBase Thrift wasn't able to
> handle the incoming Kerberos request. Documentation online is sparse when
> it comes to setting up the principal/keytab for HTTP Kerberos.
> I noticed that the HBase Thrift HTTP implementation was missing SPNEGO
> principal/keytab like other Thrift based servers (HiveServer2). It looks
> like HiveServer2 Thrift implementation and HBase Thrift v1 implementation
> were very close to the same at one point. I made the following changes to
> HBase Thrift v1 server implementation to make it work:
> * add SPNEGO principal/keytab if in HTTP mode
> * return 401 immediately if no authorization header instead of waiting for
> try/catch down in program flow{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20615) emphasize use of shaded client jars when they're present in an install
[ https://issues.apache.org/jira/browse/HBASE-20615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508358#comment-16508358 ] Josh Elser commented on HBASE-20615: bq. Here's the result of a long journey with a sad ending. I think what you've put together here is absolutely great. I can't adequately put my appreciation into words for pushing this the mile. I'm sure that this will be a moving target, but having this construct (with tests!) will be a huge boon in reducing those heavy dependencies we still have. +1 on qa. > emphasize use of shaded client jars when they're present in an install > -- > > Key: HBASE-20615 > URL: https://issues.apache.org/jira/browse/HBASE-20615 > Project: HBase > Issue Type: Sub-task > Components: build, Client, Usability >Affects Versions: 2.0.0 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Major > Fix For: 3.0.0, 2.1.0 > > Attachments: HBASE-20615.0.patch, HBASE-20615.1.patch > > > Working through setting up an IT for our shaded artifacts in HBASE-20334 > makes our lack of packaging seem like an oversight. While I could work around > by pulling the shaded clients out of whatever build process built the > convenience binary that we're trying to test, it seems v awkward. > After reflecting on it more, it makes more sense to me for there to be a > common place in the install that folks running jobs against the cluster can > rely on. If they need to run without a full hbase install, that should still > work fine via e.g. grabbing from the maven repo. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-20697) Can't cache All region locations of the specify table by calling table.getRegionLocator().getAllRegionLocations()
[
https://issues.apache.org/jira/browse/HBASE-20697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508345#comment-16508345
]
Hadoop QA commented on HBASE-20697:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} branch-1.2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
18s{color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} branch-1.2 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} branch-1.2 passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s{color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
11s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} branch-1.2 passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} branch-1.2 passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 2m
11s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
7m 48s{color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.5 2.7.4. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed with JDK v1.8.0_172 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed with JDK v1.7.0_181 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
1s{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}106m 45s{color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAdmin1 |
\\
\\
||
[jira] [Commented] (HBASE-20569) NPE in RecoverStandbyProcedure.execute
[
https://issues.apache.org/jira/browse/HBASE-20569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508339#comment-16508339
]
Hadoop QA commented on HBASE-20569:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s{color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} HBASE-19064 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
26s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
23s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
10s{color} | {color:green} branch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} HBASE-19064 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} HBASE-19064 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
57s{color} | {color:red} hbase-server: The patch generated 1 new + 220
unchanged - 0 fixed = 221 total (was 220) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedjars {color} | {color:green} 4m
7s{color} | {color:green} patch has no errors when building our shaded
downstream artifacts. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
10m 6s{color} | {color:green} Patch does not cause any errors with Hadoop
2.7.4 or 3.0.0. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green}
1m 7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
29s{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}116m
7s{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}163m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hbase:b002b0b |
| JIRA Issue | HBASE-20569 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12927310/HBASE-20569.HBASE-19064.015.patch
|
| Optional Tests | asflicense cc unit hbaseprotoc javac javadoc findbugs
shadedjars hadoopcheck hbaseanti checkstyle compile |
| uname | Linux 6b166c293639 4.4.0-43-generic #63-Ubuntu SMP
[jira] [Commented] (HBASE-20700) Move meta region when server crash can cause the procedure to be stuck
[
https://issues.apache.org/jira/browse/HBASE-20700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508321#comment-16508321
]
Hudson commented on HBASE-20700:
Results for branch branch-2
[build #850 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/850/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/850//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/850//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/850//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Move meta region when server crash can cause the procedure to be stuck
> --
>
> Key: HBASE-20700
> URL: https://issues.apache.org/jira/browse/HBASE-20700
> Project: HBase
> Issue Type: Sub-task
> Components: master, proc-v2, Region Assignment
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20700-UT.patch, HBASE-20700-branch-2.0-v1.patch,
> HBASE-20700-branch-2.0.patch, HBASE-20700-v1.patch, HBASE-20700-v2.patch,
> HBASE-20700.patch
>
>
> As said in HBASE-20682.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-20706) [hack] Don't add known not-OPEN regions in reopen phase of MTP
[
https://issues.apache.org/jira/browse/HBASE-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16508310#comment-16508310
]
Josh Elser commented on HBASE-20706:
bq. I'm saying that the change to MTP looks like a bug fix, not a hack.
Gotcha. I was being pedantic -- I wouldn't be surprised if some other changes
in-flight would invalidate the need for this bug-fix. That's what I meant by
calling this a hack.
bq. Does MTP get table lock for the life of the procedure? I thought it didn't.
You're correct, it does not grab a lock.
bq. I guess this is not safe for reopening a region. If a region is in OPENING
state, then it may have already loaded everything at the RS side, so if we do
not reopen it then it will miss the new configurations which have just
modified...
[~Apache9], thanks for looking.
Agreed that this fundamentally doesn't work for how we're using it and that we
want to do some kind of ReopenRegionProcedure for ModifyTableProcedure and
others.
I'm not sure if you agree with the fix I've made specifically for how MRP works
now to avoid this problem.
> [hack] Don't add known not-OPEN regions in reopen phase of MTP
> --
>
> Key: HBASE-20706
> URL: https://issues.apache.org/jira/browse/HBASE-20706
> Project: HBase
> Issue Type: Sub-task
> Components: amv2
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 3.0.0, 2.1.0, 2.0.1
>
> Attachments: HBASE-20706.001.branch-2.0.patch
>
>
> Shake-down of ModifyTableProcedure, talked this one out with Stack – "proper"
> fix is likely pending in HBASE-20682. Using MoveRegionProcedure is likely the
> wrong construct, we would want something specific to reopen (e.g. a
> ReopenProcedure).
> However, we're in a really bad state right now. If there are non-open regions
> for a table which has a modify submitted against it, the entire system locks
> up in a fast-spin while holding the table's lock. This fills up HDFS with PV2
> wals, and prevents you from doing anything in the hbase shell to try to fix
> those unassigned regions. You'll see spam in the master log like:
> {noformat}
> 2018-06-07 03:21:29,448 WARN [PEWorker-1] procedure.ModifyTableProcedure:
> Retriable error trying to modify table=METRIC_RECORD_HOURLY_UUID (in
> state=MODIFY_TABLE_REOPEN_ALL_REGIONS)
> org.apache.hadoop.hbase.client.DoNotRetryRegionException:
> a3dc333606d38aeb6e2ab4b94233cfbc is not OPEN
> at
> org.apache.hadoop.hbase.master.procedure.AbstractStateMachineTableProcedure.checkOnline(AbstractStateMachineTableProcedure.java:193)
> at
> org.apache.hadoop.hbase.master.assignment.MoveRegionProcedure.(MoveRegionProcedure.java:67)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createMoveRegionProcedure(AssignmentManager.java:767)
> at
> org.apache.hadoop.hbase.master.assignment.AssignmentManager.createReopenProcedures(AssignmentManager.java:705)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:128)
> at
> org.apache.hadoop.hbase.master.procedure.ModifyTableProcedure.executeFromState(ModifyTableProcedure.java:50)
> at
> org.apache.hadoop.hbase.procedure2.StateMachineProcedure.execute(StateMachineProcedure.java:184)
> at
> org.apache.hadoop.hbase.procedure2.Procedure.doExecute(Procedure.java:850)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.execProcedure(ProcedureExecutor.java:1472)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.executeProcedure(ProcedureExecutor.java:1240)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor.access$800(ProcedureExecutor.java:75)
> at
> org.apache.hadoop.hbase.procedure2.ProcedureExecutor$WorkerThread.run(ProcedureExecutor.java:1760)
> {noformat}
> We unstuck out internal test cluster giving the following change on top of
> Sergey's HBASE-20657. When choosing the regions to reopen, if we filter out a
> table's regions to only be those which are currently OPEN. There may be some
> transient failures here as well, but a subsequent retry of the reopen step
> should filter out that change.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HBASE-20715) REST curl command for create namespace with "Content-Type: text/xml" specified (mistakenly) results in a 400 Bad Request
[
https://issues.apache.org/jira/browse/HBASE-20715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nihal Jain updated HBASE-20715:
---
Description:
I have a habit of setting "Content-Type: text/xml" for curl commands. Today, I
had a bad time debugging the 400 error for below command. Finally, this
discussion(https://issues.apache.org/jira/browse/HBASE-14147?focusedCommentId=15205420=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15205420)
helped me in resolving the problem.
As pointed out by [~mwarhaftig] and [~misty] in their discussion in
HBASE-14147, a curl request of the following form will throw a {{400 BAD
REQUEST}}
{code:java}
curl -vi -X POST -H "Accept: text/xml" -H "Content-Type: text/xml"
"http://my_server:20550/namespaces/new_ns2/;
{code}
I think we should document this in the description for "Create a new namespace"
explicitly mentioning that users should take care "Content-Type" is not set for
this particular command. Similarly for "Alter an existing namespace" command.
was:
I have a habit of setting "Content-Type: text/xml" for curl commands. Today, I
had a bad time debugging the 400 error for below command. Finally, this
discussion(https://issues.apache.org/jira/browse/HBASE-14147?focusedCommentId=15205420=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15205420)
helped me in resolving the problem.
As pointed out by [~mwarhaftig] and [~misty] in their discussion in
HBASE-14147, a curl request of the following form will throw a {{400 BAD
REQUEST}}
{code:java}
curl -vi -X POST -H "Accept: text/xml" -H "Content-Type: text/xml"
"http://my_server:20550/namespaces/new_ns2/;
{code}
I think we should document this in the description for "Create a new namespace"
explicitly mentioning that users should take care "Content-Type" is not set for
this particular command.
> REST curl command for create namespace with "Content-Type: text/xml"
> specified (mistakenly) results in a 400 Bad Request
>
>
> Key: HBASE-20715
> URL: https://issues.apache.org/jira/browse/HBASE-20715
> Project: HBase
> Issue Type: Task
> Components: documentation, REST
>Reporter: Nihal Jain
>Priority: Minor
>
> I have a habit of setting "Content-Type: text/xml" for curl commands. Today,
> I had a bad time debugging the 400 error for below command. Finally, this
> discussion(https://issues.apache.org/jira/browse/HBASE-14147?focusedCommentId=15205420=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15205420)
> helped me in resolving the problem.
> As pointed out by [~mwarhaftig] and [~misty] in their discussion in
> HBASE-14147, a curl request of the following form will throw a {{400 BAD
> REQUEST}}
> {code:java}
> curl -vi -X POST -H "Accept: text/xml" -H "Content-Type: text/xml"
> "http://my_server:20550/namespaces/new_ns2/;
> {code}
> I think we should document this in the description for "Create a new
> namespace" explicitly mentioning that users should take care "Content-Type"
> is not set for this particular command. Similarly for "Alter an existing
> namespace" command.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
