Re: Updated project lead on JIRA

[Stack]

Thanks Andrew,
s

On Thu, Oct 6, 2022 at 10:22 AM Andrew Purtell  wrote:

> I realized today that I could update the user that JIRA considers project
> lead, so I updated that role on our JIRA project to our current Chair, Duo
> Zhang. (The previous setting was Michael Stack.)
>
> Sure, this is pretty random. :-) But why not. Please let me know if you
> have any concerns.
>
> --
> Best regards,
> Andrew
>

[jira] [Created] (HBASE-27396) Purge stale pom comment: ""

[Michael Stack (Jira)]

Michael Stack created HBASE-27396:
-

 Summary: Purge stale pom comment: ""
 Key: HBASE-27396
 URL: https://issues.apache.org/jira/browse/HBASE-27396
 Project: HBase
  Issue Type: Task
Reporter: Michael Stack


Any pom that has a hadoop-2.0 profile in it – all but the master branch – has 
this comment in the activation clause:

 

{{

[jira] [Created] (HBASE-27340) Artifacts with resolved profiles

[Michael Stack (Jira)]

Michael Stack created HBASE-27340:
-

 Summary: Artifacts with resolved profiles
 Key: HBASE-27340
 URL: https://issues.apache.org/jira/browse/HBASE-27340
 Project: HBase
  Issue Type: Brainstorming
Reporter: Michael Stack


Brainstorming/Discussion. The maven-flatten-plugin makes it so published poms 
are 'flattened'. The poms contain the runtime-necessary dependencies only, 
'build' and 'test' dependencies and plugins are dropped, versions are resolved 
out of properties, and so on. The published poms are the barebones minimum 
needed to run.

With a switch, the plugin can also make it so the produced poms have all 
profiles 'resolved' – making it so the produced poms have all resolved hadoop2 
or hadoop3 dependencies baked-in – based off which profile we used building.

(I've been interested in this flattening technique since I ran into a 
downstreamer using hbase from a gradle build. Gradle does not respect profiles. 
You can't specify that the gradle build pull in hbase with hadoop3 dependencies 
using 'profiles'. I notice too our [~gjacoby] , [~apurtell] et al. up on the 
dev list talking about making a hadoop3 set of artifacts...who might be 
interested in this direction).

The attached patch adds the flatten plugin so folks can take a look-see. It 
uncovers some locations where our versioning on dependencies is not explicit. 
The workaround practiced here was adding hadoop2/hadoop3 profiles into 
sub-modules that were missing them or moving problematic dependencies that were 
outside of profiles under profiles in sub-modules that had them already. For 
the latter, if the dependency specified excludes, the excludes were moved up to 
the parent pom profile (parent pom profiles have dependencyManagement 
sections... sub-modules have explicit dependency mentions... checks with 
dependency:tree seem to show excludes continue to be effective).

This is the switch that flattens profiles:   
true

This is the sort of complaint we had when the flatten plugin was having trouble 
figure dependency versions – particularly hadoop versions

{{[ERROR] Failed to execute goal 
org.codehaus.mojo:flatten-maven-plugin:1.3.0:flatten (flatten) on project 
hbase-hadoop2-compat: 3 problems were encountered while building the effective 
model for org.apache.hbase:hbase-hadoop2-compat:2.5.1-SNAPSHOT}}

{{[ERROR] [WARNING] 'build.plugins.plugin.version' for 
org.codehaus.mojo:flatten-maven-plugin is missing. @}}

{{[ERROR] [ERROR] 'dependencies.dependency.version' for 
org.apache.hadoop:hadoop-mapreduce-client-core:jar is missing. @}}

{{[ERROR] [ERROR] 'dependencies.dependency.version' for 
javax.activation:javax.activation-api:jar is missing. @}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release candidate for HBase 2.5.0 (RC1) is available

[Stack]

+1

On amd64 bash -x ./dev-support/hbase-vote.sh --source
https://dist.apache.org/repos/dist/dev/hbase/2.5.0RC1/ --output-dir
~/Downloads/250rc1 -P runSmallTests -D hadoop.profile=3.0

* Signature: ok
* Checksum : ok
* Rat check (11.0.12): ok
 - mvn clean apache-rat:check -D hadoop.profile=3.0
* Built from source (11.0.12): ok
 - mvn clean install -D hadoop.profile=3.0 -DskipTests
* Unit tests pass (11.0.12): failed
 - mvn package -P runSmallTests -D hadoop.profile=3.0
-Dsurefire.rerunFailingTestsCount=3

On arm64  bash -x ./dev-support/hbase-vote.sh --source
https://dist.apache.org/repos/dist/dev/hbase/2.5.0RC1/ --output-dir
~/Downloads/250rc1 -D skipTests -D hadoop.profile=3.0

* Signature: ok
* Checksum : failed
* Rat check (11.0.15): ok
 - mvn clean apache-rat:check -D skipTests -D hadoop.profile=3.0
* Built from source (11.0.15): ok
 - mvn clean install -D skipTests -D hadoop.profile=3.0 -DskipTests
* Unit tests pass (11.0.15): ok
 - mvn package -P runAllTests -D skipTests -D hadoop.profile=3.0
-Dsurefire.rerunFailingTestsCount=3

(Didn't runSmallTests on M1/arm because fails with HBASE-27338 brotli
compression lib tests fail on arm64
)

* Looked at the compatibility report. The red-flagged items seem internals
or items unlikely to have been sub-classed by a downstreamer
(ColumnFamilyDescriptor).
* Ran ITBLL on 6-node "cluster" on jdk17. 25 clients doing 1M each (Took
some time to get a working build... it looks like I had to specify
hadoop.version and hadoop-three.version building: TBD).
* Ran 25 cilents doing 10M each. Some kills. Stuff came back fine and
promptly. Canary works.
* UI looks fine. Poked around. Fancy region visualizer doo-hickey!
* CLI works.

S




On Wed, Aug 24, 2022 at 2:53 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.5.0RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.5.0
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.5.0RC1:
>
>   https://github.com/apache/hbase/tree/2.5.0RC1
>
> This tag currently points to git reference
>
>   2ecd8bd6d615ca49bfb329b3c0c126c80846d4ab
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.5.0RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1494/
>
> Artifacts were signed with the 0x18567F39 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Created] (HBASE-27338) brotli compression lib tests fail on arm64

[Michael Stack (Jira)]

Michael Stack created HBASE-27338:
-

 Summary: brotli compression lib tests fail on arm64
 Key: HBASE-27338
 URL: https://issues.apache.org/jira/browse/HBASE-27338
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.5.0
Reporter: Michael Stack


The brotli tests fail on M1 macs

 

{{[INFO] Running org.apache.hadoop.hbase.io.compress.brotli.TestBrotliCodec}}
{{[INFO] Running 
org.apache.hadoop.hbase.io.compress.brotli.TestHFileCompressionBrotli}}
{{[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.33 
s <<< FAILURE! - in 
org.apache.hadoop.hbase.io.compress.brotli.TestHFileCompressionBrotli}}
{{[ERROR] 
org.apache.hadoop.hbase.io.compress.brotli.TestHFileCompressionBrotli.test  
Time elapsed: 0.225 s  <<< ERROR!}}
{{java.lang.UnsatisfiedLinkError: Failed to load Brotli native library}}

{{...}}

 

The lib is installed on this machine. A new release of 
*[Brotli4j|https://github.com/hyperxpro/Brotli4j]* lib, 1.8.0, done a few days 
ago fixes the issue... (See [https://github.com/hyperxpro/Brotli4j/pull/34).] I 
tried it .

{{[INFO] ---}}
{{[INFO]  T E S T S}}
{{[INFO] ---}}
{{[INFO] Running org.apache.hadoop.hbase.io.compress.brotli.TestBrotliCodec}}
{{[INFO] Running 
org.apache.hadoop.hbase.io.compress.brotli.TestHFileCompressionBrotli}}
{{[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.036 
s - in org.apache.hadoop.hbase.io.compress.brotli.TestHFileCompressionBrotli}}

{{[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.42 s 
- in org.apache.hadoop.hbase.io.compress.brotli.TestBrotliCodec}}
{{[INFO]}}
{{[INFO] Results:}}
{{[INFO]}}
{{[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0}}
{{[INFO]}}
{{[INFO]}}
{{[INFO] --- maven-surefire-plugin:3.0.0-M6:test (secondPartTestsExecution) @ 
hbase-compression-brotli ---}}
{{[INFO] Tests are skipped.}}
{{[INFO]}}
{{[INFO] --- maven-jar-plugin:3.2.0:test-jar (default) @ 
hbase-compression-brotli ---}}
{{[INFO] Building jar: 
/Users/stack/checkouts/hbase/2.5.0RC1/hbase-2.5.0/hbase-compression/hbase-compression-brotli/target/hbase-compression-brotli-2.5.0-tests.jar}}
{{[INFO]}}
{{[INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ 
hbase-compression-brotli ---}}
{{[INFO] Building jar: 
/Users/stack/checkouts/hbase/2.5.0RC1/hbase-2.5.0/hbase-compression/hbase-compression-brotli/target/hbase-compression-brotli-2.5.0.jar}}
{{[INFO]}}
{{[INFO] --- maven-site-plugin:3.12.0:attach-descriptor (attach-descriptor) @ 
hbase-compression-brotli ---}}
{{[INFO] Skipping because packaging 'jar' is not pom.}}
{{[INFO]}}
{{[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ 
hbase-compression-brotli ---}}
{{[INFO] Installing 
/Users/stack/checkouts/hbase/2.5.0RC1/hbase-2.5.0/hbase-compression/hbase-compression-brotli/target/hbase-compression-brotli-2.5.0.jar
 to 
/Users/stack/.m2/repository/org/apache/hbase/hbase-compression-brotli/2.5.0/hbase-compression-brotli-2.5.0.jar}}
{{[INFO] Installing 
/Users/stack/checkouts/hbase/2.5.0RC1/hbase-2.5.0/hbase-compression/hbase-compression-brotli/pom.xml
 to 
/Users/stack/.m2/repository/org/apache/hbase/hbase-compression-brotli/2.5.0/hbase-compression-brotli-2.5.0.pom}}
{{[INFO] Installing 
/Users/stack/checkouts/hbase/2.5.0RC1/hbase-2.5.0/hbase-compression/hbase-compression-brotli/target/hbase-compression-brotli-2.5.0-tests.jar
 to 
/Users/stack/.m2/repository/org/apache/hbase/hbase-compression-brotli/2.5.0/hbase-compression-brotli-2.5.0-tests.jar}}
{{[INFO] 
}}
{{[INFO] BUILD SUCCESS}}
{{[INFO] 
}}
{{[INFO] Total time:  16.805 s}}
{{[INFO] Finished at: 2022-08-26T11:30:13-07:00}}
{{[INFO] 
}}

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-26321) Post blog to hbase.apache.org on SCR cache sizing

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26321.
---
Fix Version/s: 3.0.0-alpha-2
 Hadoop Flags: Reviewed
 Release Note: Pushed blog at 
https://blogs.apache.org/hbase/entry/an-hbase-hdfs-short-circuit
 Assignee: Michael Stack
   Resolution: Fixed

Thanks for taking a look [~psomogyi]

I pushed it here 
[https://blogs.apache.org/hbase/entry/an-hbase-hdfs-short-circuit]

Shout if anyone wants to add edits.

> Post blog to hbase.apache.org on SCR cache sizing
> -
>
> Key: HBASE-26321
> URL: https://issues.apache.org/jira/browse/HBASE-26321
> Project: HBase
>  Issue Type: Task
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> [~huaxiangsun] and I wrote up our experience debugging a Short-circuit Read 
> cache size issue. Let me attach link here and leave it hang here a few days 
> in case edits or input from others.  Intend to put it up here 
> https://blogs.apache.org/hbase/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26321) Post blog to hbase.apache.org on SCR cache sizing

[Michael Stack (Jira)]

Michael Stack created HBASE-26321:
-

 Summary: Post blog to hbase.apache.org on SCR cache sizing
 Key: HBASE-26321
 URL: https://issues.apache.org/jira/browse/HBASE-26321
 Project: HBase
  Issue Type: Task
Reporter: Michael Stack


[~huaxiangsun] and I wrote up our experience debugging a Short-circuit Read 
cache size issue. Let me attach link here and leave it hang here a few days in 
case edits or input from others.  Intend to put it up here 
https://blogs.apache.org/hbase/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26103) conn.getBufferedMutator(tableName) leaks thread executors and other problems (for master branch)

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26103.
---
Fix Version/s: 3.0.0-alpha-2
 Hadoop Flags: Reviewed
 Release Note: Deprecate (unused) BufferedMutatorParams#pool and 
BufferedMutatorParams#getPool
   Resolution: Fixed

Merged the PR. Thanks for the contrib [~shahrs87]  (and review [~anoop.hbase] ).

> conn.getBufferedMutator(tableName) leaks thread executors and other problems 
> (for master branch)
> 
>
> Key: HBASE-26103
> URL: https://issues.apache.org/jira/browse/HBASE-26103
> Project: HBase
>  Issue Type: Sub-task
>  Components: Client
>Affects Versions: 3.0.0-alpha-1
>Reporter: Rushabh Shah
>Assignee: Rushabh Shah
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> This is same as HBASE-26088  but created separate ticket for master branch.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24337) Backport HBASE-23968 to branch-2

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24337.
---
Fix Version/s: 2.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Backported to branch-2. Resolving.

> Backport HBASE-23968 to branch-2
> 
>
> Key: HBASE-24337
> URL: https://issues.apache.org/jira/browse/HBASE-24337
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Minwoo Kang
>Assignee: Minwoo Kang
>Priority: Minor
> Fix For: 2.5.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Split Meta Design Reset Status

[Stack]

No meeting tomorrow, August 17th. Lets try for next week, the 24th.
S

On Tue, Jan 5, 2021 at 9:13 AM Stack  wrote:

> FYI, a few of us have been working on the redo/reset of the split meta
> design (HBASE-25382). We (think we've) finished the requirements. Are there
> any others to consider?
>
> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>
> Thanks,
> S
>

[jira] [Resolved] (HBASE-24842) make export snapshot report size can be config

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24842.
---
Fix Version/s: 3.0.0-alpha-2
 Hadoop Flags: Reviewed
 Release Note: Set new config snapshot.export.report.size to size at which 
you want to see reporting.
   Resolution: Fixed

Merged to master. Make subtask if you'd like it backported. Thanks for the PR 
[~chenyechao]

> make export snapshot report size can be config
> --
>
> Key: HBASE-24842
> URL: https://issues.apache.org/jira/browse/HBASE-24842
> Project: HBase
>  Issue Type: Improvement
>  Components: mapreduce
>Reporter: Yechao Chen
>Assignee: Yechao Chen
>Priority: Minor
> Fix For: 3.0.0-alpha-2
>
>
> current export snapshot will be report ONE MB (1*1024*1024 Bytes),
> we can make it can be config 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24652) master-status UI make date type fields sortable

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24652.
---
Fix Version/s: 2.3.7
   2.4.6
   2.5.0
 Hadoop Flags: Reviewed
 Release Note: Makes RegionServer 'Start time' sortable in the Master UI
 Assignee: jeongmin kim
   Resolution: Fixed

[~jeongmin.kim] pardon me. I forgot about this one.  I pushed to branch-2.3+  
Thanks for the fix and please pardon my oversight.

> master-status UI make date type fields sortable
> ---
>
> Key: HBASE-24652
> URL: https://issues.apache.org/jira/browse/HBASE-24652
> Project: HBase
>  Issue Type: Improvement
>  Components: master, Operability, UI, Usability
>Affects Versions: 3.0.0-alpha-1, 2.2.0, 2.3.0, 2.1.5, 2.2.1, 2.1.6
>Reporter: Jeongmin Kim
>Assignee: jeongmin kim
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
> Attachments: SCREEN_SHOT1.png
>
>
> Revisit of HBASE-21207, HBASE-22543
> date type values such as regionserver list 'Start time' field on 
> master-status page, are not sorted by time.
> HBASE-21207, HBASE-22543 missed it. so before this fix, date sorted as String.
> The first field of it is 'day'. therefore always Friday goes first Wednesday 
> goes last, no matter what date it is.
>    * SCREEN_SHOT1.png
>  
> this fix make date type values sorted by time and date.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26200) Undo 'HBASE-25165 Change 'State time' in UI so sorts (#2508)' in favor of HBASE-24652

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26200.
---
Fix Version/s: 2.3.7
   2.4.6
   3.0.0-alpha-2
   2.5.0
 Release Note: Undid showing RegionServer 'Start time' in ISO-8601 format. 
Revert.
 Assignee: Michael Stack
   Resolution: Fixed

> Undo 'HBASE-25165 Change 'State time' in UI so sorts (#2508)' in favor of 
> HBASE-24652
> -
>
> Key: HBASE-26200
> URL: https://issues.apache.org/jira/browse/HBASE-26200
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> The below change by me does not actually work and I found an old issue that 
> does the proper job that was neglected. I'm undoing the below in favor of 
> HBASE-24652.
>  
> kalashnikov:hbase.apache.git stack$ git show 
> d07d181ea4a9da316659bb21fd4fffc979b5f77a
> commit d07d181ea4a9da316659bb21fd4fffc979b5f77a
> Author: Michael Stack 
> Date: Thu Oct 8 09:10:30 2020 -0700
> HBASE-25165 Change 'State time' in UI so sorts (#2508)
> Display startcode in iso8601.
> Signed-off-by: Nick Dimiduk 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26200) Undo 'HBASE-25165 Change 'State time' in UI so sorts (#2508)' in favor of HBASE-24652

[Michael Stack (Jira)]

Michael Stack created HBASE-26200:
-

 Summary: Undo 'HBASE-25165 Change 'State time' in UI so sorts 
(#2508)' in favor of HBASE-24652
 Key: HBASE-26200
 URL: https://issues.apache.org/jira/browse/HBASE-26200
 Project: HBase
  Issue Type: Bug
  Components: UI
Reporter: Michael Stack


The below change by me does not actually work and I found an old issue that 
does the proper job that was neglected. I'm undoing the below in favor of 
HBASE-24652.

 

kalashnikov:hbase.apache.git stack$ git show 
d07d181ea4a9da316659bb21fd4fffc979b5f77a
commit d07d181ea4a9da316659bb21fd4fffc979b5f77a
Author: Michael Stack 
Date: Thu Oct 8 09:10:30 2020 -0700

HBASE-25165 Change 'State time' in UI so sorts (#2508)

Display startcode in iso8601.

Signed-off-by: Nick Dimiduk 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24339) Backport HBASE-23968 to branch-1

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24339.
---
Resolution: Won't Fix

> Backport HBASE-23968 to branch-1
> 
>
> Key: HBASE-24339
> URL: https://issues.apache.org/jira/browse/HBASE-24339
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Minwoo Kang
>Assignee: Minwoo Kang
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26037) Implement namespace and table level access control for thrift & thrift2

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26037.
---
Fix Version/s: 3.0.0-alpha-2
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to master. Thanks for the PR [~xytss123]  (and review [~zhangduo] ). I 
tried to go back to branch-2 so could be in 2.5.0 but CONFLICT. Make a sub-task 
if you want a backport. Thank you.

> Implement namespace and table level access control for thrift & thrift2
> ---
>
> Key: HBASE-26037
> URL: https://issues.apache.org/jira/browse/HBASE-26037
> Project: HBase
>  Issue Type: Improvement
>  Components: Admin, Thrift
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> Client can grant or revoke ns & table level user permissions through thrift & 
> thrift2. This is implemented with AccessControlClient.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Split Meta Design Reset Status

[Stack]

On Fri, Aug 13, 2021 at 8:39 AM Andrew Purtell 
wrote:

> Also let me say that your entreaty for more to join your calls is fine and
> I hope that happens, but this is Apache, and discussions should take place
> in suitably asynchronous fashion so all can participate - even those who
> have competing responsibilities and commitments and cannot attend the
> calls. This is not criticism of the calls! I’m so glad to see you lot are
> continuing to communicate. But we need to allow and support asynchronous
> communication and comment either by mailing list or on a document. Perhaps
> a doc that lists notes from each of your session… Or otherwise randos like
> me will be posting responses to your emailed notes here.
>
>
Commenting on the notes here is great. Thanks. Otherwise, there is the
(public) design doc which is what the calls are trying to move forward; all
are free to comment/trash and improve it.

(We're thinking of moving the doc off gdoc to
dev-support/design-docs/split-meta to make it easier to see changes).

Thanks Andrew,
S




> > On Aug 13, 2021, at 8:30 AM, Andrew Purtell 
> wrote:
> >
> > Moving state into zookeeper is the wrong direction. Absolutely not. No,
> no, no. We can discuss it but it is very unlikely I would even not veto
> such a change. It doesn’t seem like a problem because you have other MUCH
> better and more viable options.
> >
> >> On Aug 13, 2021, at 7:50 AM, Stack  wrote:
> >>
> >> Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021
> >>
> >> Attendees: Francis, Duo, Francis.
> >>
> >> We moved to discuss the proposed agenda item, how to implement the
> >> "ROOT" Region.
> >>
> >> Duo had a new proposal that we host meta Regions in zookeeper (either
> >> all in a single znode or a znode per location).
> >>
> >> We talked about it. Mainly, it was noted that zookeeper doesn't scale,
> >> that a metatable in zk would not be able to satisfy
> >> "2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
> >> in violation of the old invariant that we keep no permanent state in
> >> zookeeper [2].
> >>
> >> Much back and forth on the fact that we violate the invariant
> currently; i.e.
> >> there is info in zk now that can't be scrubbed; that we need to fix
> this before
> >> we can go forward -- that the fact that meta location is currently
> persistent
> >> means we could make split meta Region locations also zk persistent.
> Counter
> >> arguments note that the original invariant script allowed that we had
> >> violations,
> >> that we have worked to undo, that addressing invariant violations was
> off-topic,
> >> as well as variations on two wrongs do not make a right, etc.
> >>
> >> We tried to reset to talk of previously suggested options of ROOT as a
> >> new table to host hbase:meta Regions or just use first-Region in
> >> hbase:meta as special case Region. A new ROOT table was generally
> >> dismissed as an impediment when small installs in no need of split meta.
> >>
> >> The second approach was called 'ugly'  -- I think 'ugly' was the term
> >> used -- and too much if/else and possibly in the way of future
> evolutions.
> >> Counter suggestions thought any uglyness could be hidden from client and
> >> isolated and that this approach seemed to meet requirements.
> Counter-arguments
> >> to the counter-argument where this approach can not hide everything
> from client,
> >> as opposed to separate ROOT table or master local region... see [1] for
> detail.
> >>
> >> We ran out of time. Will try again next week (meantime on-going
> >> developments on design doc [1] and in master).
> >>
> >> (Would be good if we could get others on the call, other PMCers
> >> with in an interest in split meta. Could be of help when we hit
> >> blockage)
> >> Thanks,
> >> S
> >>
> >> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
> >> 2. http://hbase.apache.org/book.html#design.invariants.zk.data
> >>
> >>
> >>>> On Tue, Aug 10, 2021 at 6:11 PM Stack  wrote:
> >>>
> >>> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> >>> continue our Split Meta Design Reset discussion.
> >>>
> >>> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
> >>>
> >>> Join Zoom Meeting
> >>>
> https://us04

Re: Split Meta Design Reset Status

[Stack]

Rough minutes from our meeting of  Wed Aug 11 17:04:44 PDT 2021

Attendees: Francis, Duo, Francis.

We moved to discuss the proposed agenda item, how to implement the
"ROOT" Region.

Duo had a new proposal that we host meta Regions in zookeeper (either
all in a single znode or a znode per location).

We talked about it. Mainly, it was noted that zookeeper doesn't scale,
that a metatable in zk would not be able to satisfy
"2.0.6 Support an hbase:meta Table of 10k Regions" [1], and that it also
in violation of the old invariant that we keep no permanent state in
zookeeper [2].

Much back and forth on the fact that we violate the invariant currently; i.e.
there is info in zk now that can't be scrubbed; that we need to fix this before
we can go forward -- that the fact that meta location is currently persistent
means we could make split meta Region locations also zk persistent. Counter
arguments note that the original invariant script allowed that we had
violations,
that we have worked to undo, that addressing invariant violations was off-topic,
as well as variations on two wrongs do not make a right, etc.

We tried to reset to talk of previously suggested options of ROOT as a
new table to host hbase:meta Regions or just use first-Region in
hbase:meta as special case Region. A new ROOT table was generally
dismissed as an impediment when small installs in no need of split meta.

The second approach was called 'ugly'  -- I think 'ugly' was the term
used -- and too much if/else and possibly in the way of future evolutions.
Counter suggestions thought any uglyness could be hidden from client and
isolated and that this approach seemed to meet requirements.  Counter-arguments
to the counter-argument where this approach can not hide everything from client,
as opposed to separate ROOT table or master local region... see [1] for detail.

We ran out of time. Will try again next week (meantime on-going
developments on design doc [1] and in master).

(Would be good if we could get others on the call, other PMCers
with in an interest in split meta. Could be of help when we hit
blockage)
Thanks,
S

1. 
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wv90yxilng75
2. http://hbase.apache.org/book.html#design.invariants.zk.data


On Tue, Aug 10, 2021 at 6:11 PM Stack  wrote:

> Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
> continue our Split Meta Design Reset discussion.
>
> Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09
>
> Meeting ID: 739 0014 2989
> Passcode: hbase
>
> Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
> the meeting.
>
> Topics for discussion:
>
> * Reports on progress since last meeting.
> * How will we actually implement the ROOT table (as a distinct ROOT table,
> as the first region of hbase:meta, etc.)
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>
> Thanks,
> S
>
>
> On Tue, Aug 3, 2021 at 4:13 PM Stack  wrote:
>
>> Any time Yu Li!
>>
>> No meeting tomorrow... Lets meet next week, the 10th.
>>
>> Thanks,
>> S
>>
>> On Wed, Jul 28, 2021 at 11:25 PM Yu Li  wrote:
>>
>>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>>> progress and latest status of the work!
>>>
>>> Best Regards,
>>> Yu
>>>
>>>
>>> On Wed, 28 Jul 2021 at 12:13, Stack  wrote:
>>>
>>> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>>> > August.
>>> > Thanks,
>>> > S
>>> >
>>> > On Thu, Jul 22, 2021 at 8:53 PM Stack  wrote:
>>> >
>>> > > Notes from yesterday's meeting (attendees, please amend if I
>>> misrepresent
>>> > > or if you have anything extra to add!)
>>> > >
>>> > > Split Meta Design Reset Status
>>> > >
>>> > > Wed Jul 21 21:24:38 PDT 2021
>>> > >
>>> > > Attendees: Bharath, Stack, Duo, and Francis
>>> > >
>>> > > We went over the new updates to the Brainstorming [1] section under
>>> > >
>>> > > Design in the Super Split Meta Design doc [2].
>>> > >
>>> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>>> > hide
>>> > >
>>> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
>>> > >
>>> > > implemented behind the new API in Conne

[jira] [Created] (HBASE-26191) Annotate shaded generated protobuf as InterfaceAudience.Private

[Michael Stack (Jira)]

Michael Stack created HBASE-26191:
-

 Summary: Annotate shaded generated protobuf as 
InterfaceAudience.Private
 Key: HBASE-26191
 URL: https://issues.apache.org/jira/browse/HBASE-26191
 Project: HBase
  Issue Type: Task
  Components: Coprocessors, Protobufs
Reporter: Michael Stack


Annotate generated shaded protobufs as InterfaceAudience.Private. It might not 
be able to add the annotation to each class; at a minimum update the doc on our 
story around shaded internal protobufs.

See the prompting mailing list discussion here: 
[https://lists.apache.org/thread.html/r9e6eb11106727d245f6eb2a5023823901637971d6ed0f0aedaf8d149%40%3Cdev.hbase.apache.org%3E]

So far the consensus has it that the shaded generated protobuf should be made 
IA.Private.  Will wait on it to settle.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-16756) InterfaceAudience annotate our protobuf; distinguish internal; publish public

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-16756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-16756.
---
Resolution: Won't Fix

Not doing this.

> InterfaceAudience annotate our protobuf; distinguish internal; publish public
> -
>
> Key: HBASE-16756
> URL: https://issues.apache.org/jira/browse/HBASE-16756
> Project: HBase
>  Issue Type: Task
>  Components: Protobufs
>    Reporter: Michael Stack
>Priority: Major
>
> This is a follow-on from the work done over in HBASE-15638 Shade protobuf.
> Currently protobufs are not annotated as our java classes are even though 
> they are being used by downstream Coprocessor Endpoints; i.e. if a CPEP wants 
> to update a Cell in HBase or refer to a server in the cluster, 9 times out of 
> 10 they will depend on the HBase Cell.proto and its generated classes or the 
> ServerName definition in HBase.proto file.
> This makes it so we cannot make breaking changes to the Cell type or relocate 
> the ServerName definition to another file if we want CPEPs to keep working.
> The issue gets compounded by HBASE-15638 "Shade protobuf" where protos used 
> internally are relocated, and given another package name altogether. 
> Currently we leave behind the old protos (sort-of duplicated) so CPEPs keep 
> working but going forward, IF WE CONTINUE DOWN THIS PATH OF SHADING PROTOS 
> (we may revisit if hadoop ends up isolating its classpath), then we need to 
> 'publish' protos that we will honor as we would classes annotate with 
> @InterfaceAudience.Public as part of our public API going forward.
> What is involved is a review of the current protos under hbase-protocol. Sort 
> out what is to be made public. We will likely have to break up current proto 
> files into smaller collections since they currently contain mixes of public 
> and private types. Deprecate the fat Admin and Client protos.  This will 
> allow us to better narrow the set of what we make public. These new files 
> could live in the hbase-protocol module suitably annotated or they could be 
> done up in a new module altogether. TODO.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] InterfaceAudience for protobufs

[Stack]

On Tue, Aug 10, 2021 at 10:48 AM Bryan Beaudreault
 wrote:

> Hello dev list,
>
> This question came up in https://github.com/apache/hbase/pull/3536, where
> I
> renamed a field on the BalanceRequest message in Master.proto.
>
> This change is backwards compatible to the majority of users, because
> protobuf does not actually care about the name of the field (only the
> ordinal/id). So long as someone is interacting with MasterRpcServices
> through the Admin/AsyncAdmin interfaces, they would be completely unaware
> of the renamed field under the covers.
>
> One concern I had was for people with their own forks, or 3rd party clients
> as Nick brought up. In those cases, they would get a compile time error
> when building.
>
> We don't explicitly cover protobufs in our
> http://hbase.apache.org/book.html#hbase.versioning.post10 documentation.
> Nick suggested that it might make sense to consider the protobufs part of
> the LimitedPrivate API but would like to open that up to other opinions.
>
>
We have the modules hbase-protocol and hbase-protocol-shaded. They host our
'protocol' spec'd in protobuf proto files and the classes generated from
proto files by protoc. The latter, shaded version, was made so we could
freely update the protobuf version we used internally and change protobufs
w/o having to worry about downstreamers or running into conflict w/
upstream (hadoop). Effectively, it was treated as though it
interfaceaudience private. We should probably stamp it so (and I think this
means your change can be made w/o concern for downstreamers).

hbase-protobuf was for use in those locations where we had protobuf as part
of our public API; in particular, our use of protobuf describing interfaces
for coprocessor endpoints. This latter module uses the EOL'd protobuf
version 2.5.0. It was sort-of public but also was being let atrophy as
often protobuf changes were made in the shaded module and not made here,
particularly if for internal-use only.  hbase-protocol needs some work
(Related https://issues.apache.org/jira/browse/HBASE-16756).

S




> If we reach a consensus I would be happy to submit a jira to update our
> docs.
>

Re: HBASE-6908 Solution Proposal

[Stack]

On Fri, Aug 6, 2021 at 9:15 AM Rich Marscher 
wrote:

> Hi,
> we have recently been interested in extending our options for RPC queues in
> the hbase-server and in searching for prior issues found HBASE-6908 which
> aligns exactly with our needs. I put together a PR with a proposed solution
> to have a pluggable queue type that takes in the FQCN of the queue class to
> load from the classpath and also is wired up to be a ConfigurationObserver,
> in the event the pluggable code wants to receive configuration updates.
>
> Issue: https://issues.apache.org/jira/browse/HBASE-6908
> PR: https://github.com/apache/hbase/pull/3522
>
> I would appreciate some attention for feedback or approval.
>
> Thanks,
> Richard
>

Done. Thanks for the work Richard.
S

Re: Split Meta Design Reset Status

[Stack]

Meeting tomorrow afternoon @ 5pm (to accommodate Beijing time). Lets
continue our Split Meta Design Reset discussion.

Time: Aug 11, 2021 05:00 PM Pacific Time (US and Canada)

Join Zoom Meeting
https://us04web.zoom.us/j/73900142989?pwd=b2s1UmJ2a2hiUFRsRERYQzZBV0NHdz09

Meeting ID: 739 0014 2989
Passcode: hbase

Please review the design doc "Design/Brainstorming" Section 4.1 [1] before
the meeting.

Topics for discussion:

* Reports on progress since last meeting.
* How will we actually implement the ROOT table (as a distinct ROOT table,
as the first region of hbase:meta, etc.)

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle

Thanks,
S


On Tue, Aug 3, 2021 at 4:13 PM Stack  wrote:

> Any time Yu Li!
>
> No meeting tomorrow... Lets meet next week, the 10th.
>
> Thanks,
> S
>
> On Wed, Jul 28, 2021 at 11:25 PM Yu Li  wrote:
>
>> Thanks for the notes and efforts Stack, it's pretty helpful to know the
>> progress and latest status of the work!
>>
>> Best Regards,
>> Yu
>>
>>
>> On Wed, 28 Jul 2021 at 12:13, Stack  wrote:
>>
>> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
>> > August.
>> > Thanks,
>> > S
>> >
>> > On Thu, Jul 22, 2021 at 8:53 PM Stack  wrote:
>> >
>> > > Notes from yesterday's meeting (attendees, please amend if I
>> misrepresent
>> > > or if you have anything extra to add!)
>> > >
>> > > Split Meta Design Reset Status
>> > >
>> > > Wed Jul 21 21:24:38 PDT 2021
>> > >
>> > > Attendees: Bharath, Stack, Duo, and Francis
>> > >
>> > > We went over the new updates to the Brainstorming [1] section under
>> > >
>> > > Design in the Super Split Meta Design doc [2].
>> > >
>> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
>> > hide
>> > >
>> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
>> > >
>> > > implemented behind the new API in ConnectionRegistry. On option 1.,
>> > >
>> > > replicating master-local Region to RegionServers, options considered
>> > >
>> > > included
>> > >
>> > >  * Listener on master-local Region WAL to replicate.
>> > >
>> > >  * Perhaps Read-Replica but master-local is not an actual Region
>> > >
>> > >  * Needs to be incremental edits because ROOT could get too big to
>> ship
>> > >
>> > >in a lump; need to visit how...
>> > >
>> > >  * Possibly in-memory-only Regions on RS replicated from master-local
>> > >
>> > >Region via WAL tailing <= zhang...@apache.org
>> > >
>> > >  * Which RegionServers? Those hosting ROOT replicas?
>> > >
>> > >  * How to bootstrap? Failure scenarios.
>> > >
>> > >  * This would be a new replication system alongside current; could
>> > >
>> > >evolve to replace/improve old?
>> > >
>> > > Duo offered to look into means of replicating the master-local Region
>> > >
>> > > out to RegionServers.
>> > >
>> > > Next up was discussion constrasting ROOT as a standalone table vs
>> > >
>> > > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>> > >
>> > > options 2 and 3 for how we'd implement a ROOT. One item that came up
>> > >
>> > > was whether a need to specify one replica count for a ROOT table vs
>> > >
>> > > another for hbase:meta. If so, then it would be argument for ROOT as
>> > >
>> > > standalone table (Others of us argued it not a concern of
>> consequence).
>> > >
>> > > If ROOT access is behind a new simple API in ConnectionRegistry, how
>> > >
>> > > to stop clients reading hbase:meta table if not Master or fronted by
>> > >
>> > > a ConnectionRegistry request? (Should be able to switch on client
>> > >
>> > > identity/source). One suggestion for First-meta-Region-as-ROOT was
>> > >
>> > > NOT returning the first Region to the client post-meta split when
>> > >
>> > > accessing via the simple API. Some concern this would confuse old
>> > >
>> > > Clients (Francis was going to take a look).
>> > >
>> > &

[jira] [Resolved] (HBASE-6908) Pluggable Call BlockingQueue for HBaseServer

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-6908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-6908.
--
Fix Version/s: 2.4.6
   3.0.0-alpha-2
   2.5.0
 Hadoop Flags: Reviewed
 Release Note: 
Can pass in a FQCN to load as the call queue implementation.

Standardized arguments to the constructor are the max queue length, the 
PriorityFunction, and the Configuration.

PluggableBlockingQueue abstract class provided to help guide the correct 
constructor signature.

Hard fails with PluggableRpcQueueNotFound if the class fails to load as a 
BlockingQueue

Upstreaming on behalf of Hubspot, we are interested in defining our own custom 
RPC queue and don't want to get involved in necessarily upstreaming internal 
requirements/iterations. 

   Resolution: Fixed

Merged to branch-2.4+. Thanks for the clean pluggable Interface [~rmarsch]  
I put your PR comment as release note. Edit if you see fit.

> Pluggable Call BlockingQueue for HBaseServer
> 
>
> Key: HBASE-6908
> URL: https://issues.apache.org/jira/browse/HBASE-6908
> Project: HBase
>  Issue Type: New Feature
>  Components: IPC/RPC
>Reporter: James Taylor
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> Allow the BlockingQueue implementation class to be specified in the HBase 
> config to enable different behavior than a FIFO queue. The use case we have 
> is around fairness and starvation for big scans that are parallelized on the 
> client. It's easy to fill up the HBase server Call BlockingQueue when 
> processing a single parallelized scan, leadng other scans to time out. 
> Instead, doing round robin processesing on a dequeue through a different 
> BlockingQueue implementation will prevent this from occurring.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26170) handleTooBigRequest in NettyRpcServer didn't skip enough bytes

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26170.
---
Fix Version/s: 2.3.7
   2.4.6
   3.0.0-alpha-2
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to branch-2.3+. Nice fix [~Xiaolin Ha]

> handleTooBigRequest in NettyRpcServer didn't skip enough bytes
> --
>
> Key: HBASE-26170
> URL: https://issues.apache.org/jira/browse/HBASE-26170
> Project: HBase
>  Issue Type: Bug
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-2, 2.4.6, 2.3.7
>
> Attachments: error-logs.png
>
>
> We found there are always coredump problems after too big requests, the logs 
> are as follows,
> !error-logs.png|width=1040,height=187!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26153) [create-release] Use cmd-line defined env vars

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26153.
---
Fix Version/s: 3.0.0-alpha-2
 Assignee: Michael Stack
   Resolution: Fixed

Merged trivial create-release script changes.

> [create-release] Use cmd-line defined env vars
> --
>
> Key: HBASE-26153
> URL: https://issues.apache.org/jira/browse/HBASE-26153
> Project: HBase
>  Issue Type: Improvement
>  Components: RC
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Trivial
> Fix For: 3.0.0-alpha-2
>
>
> Minor item. The create-release scripts allows defining some of the variables 
> used on the command line but not all. Fix.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Split Meta Design Reset Status

[Stack]

Any time Yu Li!

No meeting tomorrow... Lets meet next week, the 10th.

Thanks,
S

On Wed, Jul 28, 2021 at 11:25 PM Yu Li  wrote:

> Thanks for the notes and efforts Stack, it's pretty helpful to know the
> progress and latest status of the work!
>
> Best Regards,
> Yu
>
>
> On Wed, 28 Jul 2021 at 12:13, Stack  wrote:
>
> > Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of
> > August.
> > Thanks,
> > S
> >
> > On Thu, Jul 22, 2021 at 8:53 PM Stack  wrote:
> >
> > > Notes from yesterday's meeting (attendees, please amend if I
> misrepresent
> > > or if you have anything extra to add!)
> > >
> > > Split Meta Design Reset Status
> > >
> > > Wed Jul 21 21:24:38 PDT 2021
> > >
> > > Attendees: Bharath, Stack, Duo, and Francis
> > >
> > > We went over the new updates to the Brainstorming [1] section under
> > >
> > > Design in the Super Split Meta Design doc [2].
> > >
> > > First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry;
> > hide
> > >
> > > ROOT from Client [3]. In particular, filling out how "ROOT" might be
> > >
> > > implemented behind the new API in ConnectionRegistry. On option 1.,
> > >
> > > replicating master-local Region to RegionServers, options considered
> > >
> > > included
> > >
> > >  * Listener on master-local Region WAL to replicate.
> > >
> > >  * Perhaps Read-Replica but master-local is not an actual Region
> > >
> > >  * Needs to be incremental edits because ROOT could get too big to ship
> > >
> > >in a lump; need to visit how...
> > >
> > >  * Possibly in-memory-only Regions on RS replicated from master-local
> > >
> > >Region via WAL tailing <= zhang...@apache.org
> > >
> > >  * Which RegionServers? Those hosting ROOT replicas?
> > >
> > >  * How to bootstrap? Failure scenarios.
> > >
> > >  * This would be a new replication system alongside current; could
> > >
> > >evolve to replace/improve old?
> > >
> > > Duo offered to look into means of replicating the master-local Region
> > >
> > > out to RegionServers.
> > >
> > > Next up was discussion constrasting ROOT as a standalone table vs
> > >
> > > First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
> > >
> > > options 2 and 3 for how we'd implement a ROOT. One item that came up
> > >
> > > was whether a need to specify one replica count for a ROOT table vs
> > >
> > > another for hbase:meta. If so, then it would be argument for ROOT as
> > >
> > > standalone table (Others of us argued it not a concern of consequence).
> > >
> > > If ROOT access is behind a new simple API in ConnectionRegistry, how
> > >
> > > to stop clients reading hbase:meta table if not Master or fronted by
> > >
> > > a ConnectionRegistry request? (Should be able to switch on client
> > >
> > > identity/source). One suggestion for First-meta-Region-as-ROOT was
> > >
> > > NOT returning the first Region to the client post-meta split when
> > >
> > > accessing via the simple API. Some concern this would confuse old
> > >
> > > Clients (Francis was going to take a look).
> > >
> > > Moved to discussion how we'd move ConnectionRegistry from
> > >
> > > hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
> > >
> > > system came up? Where do clients go? How do they know which
> > >
> > > RegionServers (special regionserver group?  Every RS fields
> > >
> > > ConnectionRegistry requests but only designated core serve the ROOT
> > >
> > > lookup APIs?). This was a TODO.
> > >
> > > This led naturally into 4.1.5 System RS group for client meta services
> > >
> > > [5], a new addition under Brainstorming. Discussion. Bharath to look
> > >
> > > into feedback.
> > >
> > > On the end of the discussion, group expressed support for adding
> > >
> > > simple API to the ConnectionRegistry to hide ROOT implementation
> > >
> > > detail from client. Support was expressed for moving ConnectionRegistry
> > >
> > > from Master to RegionServers. Intent is to move forward on design of
> > >
> > > these pieces: e.g. how

[ANNOUNCE] Apache HBase 2.3.6 is now available for download

[stack]

The HBase team is happy to announce the immediate availability of HBase
2.3.6.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database.

Apache HBase gives you low latency random access to billions of rows with
millions of columns atop non-specialized hardware. To learn more about
HBase, see https://hbase.apache.org/.

HBase 2.3.6 is the sixth patch release in the HBase 2.3.x line. The full
list of issues can be found in the included CHANGES and RELEASENOTES,
or via our issue tracker:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12350029

To download please follow the links and instructions on our website:

https://hbase.apache.org/downloads.html

Questions, comments, and problems are always welcome at:
dev@hbase.apache.org.

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[jira] [Resolved] (HBASE-26162) Release 2.3.6

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26162.
---
Fix Version/s: 2.3.7
 Assignee: Michael Stack
   Resolution: Fixed

Sent announcement email, ran all steps in above list. Downloads will update 
tonight. Resolving.

> Release 2.3.6
> -
>
> Key: HBASE-26162
> URL: https://issues.apache.org/jira/browse/HBASE-26162
> Project: HBase
>  Issue Type: Task
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 2.3.7
>
> Attachments: image-2021-08-02-09-54-56-469.png
>
>
> 2.3.6RC3 was voted as 2.3.6 release.
> Run the release steps listed here for 2.3.6
> !image-2021-08-02-09-54-56-469.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26162) Release 2.3.6

[Michael Stack (Jira)]

Michael Stack created HBASE-26162:
-

 Summary: Release 2.3.6
 Key: HBASE-26162
 URL: https://issues.apache.org/jira/browse/HBASE-26162
 Project: HBase
  Issue Type: Task
Reporter: Michael Stack
 Attachments: image-2021-08-02-09-54-56-469.png

2.3.6RC3 was voted as 2.3.6 release.

Run the release steps listed here for 2.3.6

!image-2021-08-02-09-54-56-469.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[RESULT][VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

With 10 binding +1s and two non-binding +1s, the vote passes.

Let me push out the release.

Thank you for taking the time to vote (Your 'I Voted!' sticker is in the
mail!).

S

On Wed, Jul 28, 2021 at 7:56 PM Stack  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.3.6RC3
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.3.6
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.3.6RC3:
>
>   https://github.com/apache/hbase/tree/2.3.6RC3
>
> This tag currently points to git reference
>
>   7414579f2620fca6b75146c29ab2726fc4643ac9
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1463/
>
> Artifacts were signed with the 0xA7874F29 key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [ANNOUNCE] New HBase PMC Bharath Vissapragada

[Stack]

Good on you B.
S

On Fri, Jul 30, 2021 at 6:26 AM Viraj Jasani  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Bharath
> Vissapragada has accepted our invitation to become a PMC member on the
> HBase project. We appreciate Bharath stepping up to take more
> responsibility for the project.
>
> Congratulations and welcome, Bharath!
>

[jira] [Created] (HBASE-26153) [create-release] Use cmd-line defined env vars

[Michael Stack (Jira)]

Michael Stack created HBASE-26153:
-

 Summary: [create-release] Use cmd-line defined env vars
 Key: HBASE-26153
 URL: https://issues.apache.org/jira/browse/HBASE-26153
 Project: HBase
  Issue Type: Improvement
  Components: RC
Reporter: Michael Stack


Minor item. The create-release scripts allows defining some of the variables 
used on the command line but not all. Fix.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

On Wed, Jul 28, 2021 at 10:58 PM Huaxiang Sun  wrote:

> I got a checksum error, is it just me?


I just did a pull and checked the checksums and seems good to me. Try again?
S



> Otherwise, I checked release file
> and jiras there look good.
>
> ~/work/hbase
>
> * Signature: ok
>
> * Checksum : failed
>
> * Rat check (1.8.0_242): ok
>
>  - mvn clean apache-rat:check
>
> * Built from source (1.8.0_242): ok
>
>  - mvn clean install  -DskipTests
>
> * Unit tests pass (1.8.0_242): ok
>
>  - mvn package -P runSmallTests
> -Dsurefire.rerunFailingTestsCount=3
>
> iMac-Pro:hbase hsun$ ./dev-support/hbase-vote.sh -s
> https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3 -P runSmallTests
>
> On Wed, Jul 28, 2021 at 9:20 PM Stack  wrote:
>
> > On Wed, Jul 28, 2021 at 9:16 PM Stack  wrote:
> >
> > > Looks like. Let me fix.
> > > S
> > >
> > > Done.
> > Thanks,
> > S
> >
> >
> >
> > > On Wed, Jul 28, 2021 at 8:27 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > >> Not related to vote but I saw a 2.3.6 branch in the git repo? Mistake?
> > >>
> > >> Stack  于2021年7月29日周四 上午10:56写道：
> > >> >
> > >> > Please vote on this Apache hbase release candidate,
> > >> > hbase-2.3.6RC3
> > >> >
> > >> > The VOTE will remain open for at least 72 hours.
> > >> >
> > >> > [ ] +1 Release this package as Apache hbase 2.3.6
> > >> > [ ] -1 Do not release this package because ...
> > >> >
> > >> > The tag to be voted on is 2.3.6RC3:
> > >> >
> > >> >   https://github.com/apache/hbase/tree/2.3.6RC3
> > >> >
> > >> > This tag currently points to git reference
> > >> >
> > >> >   7414579f2620fca6b75146c29ab2726fc4643ac9
> > >> >
> > >> > The release files, including signatures, digests, as well as
> > CHANGES.md
> > >> > and RELEASENOTES.md included in this RC can be found at:
> > >> >
> > >> >   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/
> > >> >
> > >> > Maven artifacts are available in a staging repository at:
> > >> >
> > >> >
> > >>
> https://repository.apache.org/content/repositories/orgapachehbase-1463/
> > >> >
> > >> > Artifacts were signed with the 0xA7874F29 key which can be found in:
> > >> >
> > >> >   https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >> >
> > >> > To learn more about Apache hbase, please see
> > >> >
> > >> >   http://hbase.apache.org/
> > >> >
> > >> > Thanks,
> > >> > Your HBase Release Manager
> > >>
> > >
> >
>

Re: [VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

On Wed, Jul 28, 2021 at 9:16 PM Stack  wrote:

> Looks like. Let me fix.
> S
>
> Done.
Thanks,
S



> On Wed, Jul 28, 2021 at 8:27 PM 张铎(Duo Zhang) 
> wrote:
>
>> Not related to vote but I saw a 2.3.6 branch in the git repo? Mistake?
>>
>> Stack  于2021年7月29日周四 上午10:56写道：
>> >
>> > Please vote on this Apache hbase release candidate,
>> > hbase-2.3.6RC3
>> >
>> > The VOTE will remain open for at least 72 hours.
>> >
>> > [ ] +1 Release this package as Apache hbase 2.3.6
>> > [ ] -1 Do not release this package because ...
>> >
>> > The tag to be voted on is 2.3.6RC3:
>> >
>> >   https://github.com/apache/hbase/tree/2.3.6RC3
>> >
>> > This tag currently points to git reference
>> >
>> >   7414579f2620fca6b75146c29ab2726fc4643ac9
>> >
>> > The release files, including signatures, digests, as well as CHANGES.md
>> > and RELEASENOTES.md included in this RC can be found at:
>> >
>> >   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/
>> >
>> > Maven artifacts are available in a staging repository at:
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachehbase-1463/
>> >
>> > Artifacts were signed with the 0xA7874F29 key which can be found in:
>> >
>> >   https://dist.apache.org/repos/dist/release/hbase/KEYS
>> >
>> > To learn more about Apache hbase, please see
>> >
>> >   http://hbase.apache.org/
>> >
>> > Thanks,
>> > Your HBase Release Manager
>>
>

Re: [VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

+1

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_292): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_292): ok
 - mvn clean install  -DskipTests
* Unit tests pass (1.8.0_292): ok
 - mvn package -P runSmallTests  -Dsurefire.rerunFailingTestsCount=3

S

On Wed, Jul 28, 2021 at 7:56 PM Stack  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.3.6RC3
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.3.6
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.3.6RC3:
>
>   https://github.com/apache/hbase/tree/2.3.6RC3
>
> This tag currently points to git reference
>
>   7414579f2620fca6b75146c29ab2726fc4643ac9
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1463/
>
> Artifacts were signed with the 0xA7874F29 key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

Looks like. Let me fix.
S

On Wed, Jul 28, 2021 at 8:27 PM 张铎(Duo Zhang)  wrote:

> Not related to vote but I saw a 2.3.6 branch in the git repo? Mistake?
>
> Stack  于2021年7月29日周四 上午10:56写道：
> >
> > Please vote on this Apache hbase release candidate,
> > hbase-2.3.6RC3
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache hbase 2.3.6
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.3.6RC3:
> >
> >   https://github.com/apache/hbase/tree/2.3.6RC3
> >
> > This tag currently points to git reference
> >
> >   7414579f2620fca6b75146c29ab2726fc4643ac9
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/
> >
> > Maven artifacts are available in a staging repository at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1463/
> >
> > Artifacts were signed with the 0xA7874F29 key which can be found in:
> >
> >   https://dist.apache.org/repos/dist/release/hbase/KEYS
> >
> > To learn more about Apache hbase, please see
> >
> >   http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
>

[VOTE] Second release candidate for HBASE 2.3.6 (RC3)

[Stack]

Please vote on this Apache hbase release candidate,
hbase-2.3.6RC3

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 2.3.6
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.3.6RC3:

  https://github.com/apache/hbase/tree/2.3.6RC3

This tag currently points to git reference

  7414579f2620fca6b75146c29ab2726fc4643ac9

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC3/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1463/

Artifacts were signed with the 0xA7874F29 key which can be found in:

  https://dist.apache.org/repos/dist/release/hbase/KEYS

To learn more about Apache hbase, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

[jira] [Resolved] (HBASE-26146) Allow custom opts for hbck in hbase bin

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26146.
---
Fix Version/s: 2.4.6
 Release Note: Adds HBASE_HBCK_OPTS environment variable to bin/hbase for 
passing extra options to hbck/hbck2. Defaults to HBASE_SERVER_JAAS_OPTS if 
specified, or HBASE_REGIONSERVER_OPTS.
   Resolution: Fixed

Pushed #3537 to branch-2.4. Re-resolving.

 

Added a release note [~anoop.hbase]

> Allow custom opts for hbck in hbase bin
> ---
>
> Key: HBASE-26146
> URL: https://issues.apache.org/jira/browse/HBASE-26146
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6
>
>
> https://issues.apache.org/jira/browse/HBASE-15145 made it so that when you 
> execute {{hbase hbck}}, the regionserver or JAAS opts are added automatically 
> to the command line. This is problematic in some cases depending on what 
> regionserver opts have been set. For instance, one might configure a jmx port 
> for the regionserver but then hbck will fail due to a port conflict if run on 
> the same host as a regionserver. Another example would be that a regionserver 
> might define an {{-Xms}} value which is significantly more than hbck requires.
>  
> We should make it possible for users to define their own HBASE_HBCK_OPTS 
> which take precedence over the server opts added by default.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-26146) Allow custom opts for hbck in hbase bin

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack reopened HBASE-26146:
---

Reopening to apply backport to branch-2.3 (#3537)

> Allow custom opts for hbck in hbase bin
> ---
>
> Key: HBASE-26146
> URL: https://issues.apache.org/jira/browse/HBASE-26146
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> https://issues.apache.org/jira/browse/HBASE-15145 made it so that when you 
> execute {{hbase hbck}}, the regionserver or JAAS opts are added automatically 
> to the command line. This is problematic in some cases depending on what 
> regionserver opts have been set. For instance, one might configure a jmx port 
> for the regionserver but then hbck will fail due to a port conflict if run on 
> the same host as a regionserver. Another example would be that a regionserver 
> might define an {{-Xms}} value which is significantly more than hbck requires.
>  
> We should make it possible for users to define their own HBASE_HBCK_OPTS 
> which take precedence over the server opts added by default.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26148) Backport HBASE-26146 to branch-2.4

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26148.
---
Resolution: Invalid

> Backport HBASE-26146 to branch-2.4
> --
>
> Key: HBASE-26148
> URL: https://issues.apache.org/jira/browse/HBASE-26148
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Split Meta Design Reset Status

[Stack]

Note, no meeting tomorrow. We'll meet next week on the 4th or 5th of August.
Thanks,
S

On Thu, Jul 22, 2021 at 8:53 PM Stack  wrote:

> Notes from yesterday's meeting (attendees, please amend if I misrepresent
> or if you have anything extra to add!)
>
> Split Meta Design Reset Status
>
> Wed Jul 21 21:24:38 PDT 2021
>
> Attendees: Bharath, Stack, Duo, and Francis
>
> We went over the new updates to the Brainstorming [1] section under
>
> Design in the Super Split Meta Design doc [2].
>
> First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; hide
>
> ROOT from Client [3]. In particular, filling out how "ROOT" might be
>
> implemented behind the new API in ConnectionRegistry. On option 1.,
>
> replicating master-local Region to RegionServers, options considered
>
> included
>
>  * Listener on master-local Region WAL to replicate.
>
>  * Perhaps Read-Replica but master-local is not an actual Region
>
>  * Needs to be incremental edits because ROOT could get too big to ship
>
>in a lump; need to visit how...
>
>  * Possibly in-memory-only Regions on RS replicated from master-local
>
>Region via WAL tailing <= zhang...@apache.org
>
>  * Which RegionServers? Those hosting ROOT replicas?
>
>  * How to bootstrap? Failure scenarios.
>
>  * This would be a new replication system alongside current; could
>
>evolve to replace/improve old?
>
> Duo offered to look into means of replicating the master-local Region
>
> out to RegionServers.
>
> Next up was discussion constrasting ROOT as a standalone table vs
>
> First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.
>
> options 2 and 3 for how we'd implement a ROOT. One item that came up
>
> was whether a need to specify one replica count for a ROOT table vs
>
> another for hbase:meta. If so, then it would be argument for ROOT as
>
> standalone table (Others of us argued it not a concern of consequence).
>
> If ROOT access is behind a new simple API in ConnectionRegistry, how
>
> to stop clients reading hbase:meta table if not Master or fronted by
>
> a ConnectionRegistry request? (Should be able to switch on client
>
> identity/source). One suggestion for First-meta-Region-as-ROOT was
>
> NOT returning the first Region to the client post-meta split when
>
> accessing via the simple API. Some concern this would confuse old
>
> Clients (Francis was going to take a look).
>
> Moved to discussion how we'd move ConnectionRegistry from
>
> hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a
>
> system came up? Where do clients go? How do they know which
>
> RegionServers (special regionserver group?  Every RS fields
>
> ConnectionRegistry requests but only designated core serve the ROOT
>
> lookup APIs?). This was a TODO.
>
> This led naturally into 4.1.5 System RS group for client meta services
>
> [5], a new addition under Brainstorming. Discussion. Bharath to look
>
> into feedback.
>
> On the end of the discussion, group expressed support for adding
>
> simple API to the ConnectionRegistry to hide ROOT implementation
>
> detail from client. Support was expressed for moving ConnectionRegistry
>
> from Master to RegionServers. Intent is to move forward on design of
>
> these pieces: e.g. how client bootstraps.
>
> Support was expressed for getting at least the bones of a split meta
>
> into an hbase3 before the RCs.
>
> Where we'd actually store hbase:meta Region locations -- i.e. how a
>
> "ROOT' would be implemented -- was for our next meeting informed by
>
> research of the various approaches noted mostly above. It was
>
> also thought that the new ConnectionRegistry should not preclude
>
> making progress on the "ROOT" implementation.
>
> Will post notice of next meeting (next Weds or the one
>
> following).
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>
> 2.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq
>
> 3.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153
>
> 4.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
>
> 5.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b
>
> On Tue, Jul 20, 2021 at 11:00 AM Stack  wrote:
>
>> Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
>> Section 4.1 [1] before the meeting if you 

Re: [VOTE] Second release candidate for HBase 2.4.5 (RC1) is available

[Stack]

+1

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_292): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_292): ok
 - mvn clean install  -DskipTests
* Unit tests pass (1.8.0_292): ok
 - mvn package -P runSmallTests  -Dsurefire.rerunFailingTestsCount=3

Compat report looks good.

Built from src w/ release profile. Started it up in standalone mode. Loaded
up data w/ pe. Verified it all made it. Poked around UI and shell. Looks
good to me.

S

On Tue, Jul 27, 2021 at 11:09 AM Andrew Purtell 
wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.4.5RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.4.5
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.4.5RC1:
>
>   https://github.com/apache/hbase/tree/2.4.5RC1
>
> This tag currently points to git reference 03b8c0cf42.
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.4.5RC1/
>
> The compatibility report for this RC can be found at:
>
>
> https://dist.apache.org/repos/dist/dev/hbase/2.4.5RC1/api_compare_2.4.4_to_2.4.5RC1.html
>
> There are 0 reported compatibility issues.
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1462/
>
> Artifacts were signed with the 0xD5365CCD key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] First release candidate for HBase 2.3.6 (RC2) is available

[Stack]

On Tue, Jul 27, 2021 at 4:12 PM Huaxiang Sun  wrote:

> Sorry, I have to put -1 on this RC.
>
> In release note,
> https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC2/CHANGES.md,
> HBASE-24492 is listed in 2.3.6.
> Actually, this jira is in 2.3.3, not 2.3.6. The release version in the jira
> is messed up. I corrected it in the jira.
>
> | [HBASE-24492](https://issues.apache.org/jira/browse/HBASE-24492) |
> ProtobufLogReader.readNext does not need looping |  Minor |
> Replication, wal |
>
>

Ok. Let me spin another. Thanks Huaxiang.
S



> Huaxiang
>
>
> On Tue, Jul 27, 2021 at 4:06 PM Nick Dimiduk  wrote:
>
> > +1
> >
> > * Compatibility report: ok
> > * Nightly builds good: okay
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_292): ok
> >  - mvn clean apache-rat:check -D skipTests
> > * Rat check (11.0.11): ok
> >  - mvn clean apache-rat:check -D skipTests -D hadoop.profile=3.0
> > * Built from source (1.8.0_292): ok
> >  - mvn clean install -D skipTests -DskipTests
> > * Built from source (11.0.11): ok
> >  - mvn clean install -D skipTests -D hadoop.profile=3.0
> -DskipTests
> > * Unit tests pass (1.8.0_292): ok
> >  - mvn package -P runAllTests -D skipTests
> > -Dsurefire.rerunFailingTestsCount=3
> > * Unit tests pass (11.0.11): ok
> >  - mvn package -P runAllTests -D skipTests -D hadoop.profile=3.0
> > -Dsurefire.rerunFailingTestsCount=3
> >
> > On Mon, Jul 26, 2021 at 11:14 PM Stack  wrote:
> >
> > > Please vote on this Apache hbase release candidate,
> > > hbase-2.3.6RC2
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.3.6
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.3.6RC2:
> > >
> > >   https://github.com/apache/hbase/tree/2.3.6RC2
> > >
> > > This tag currently points to git reference
> > >
> > >   79d57a453ac840873ca15fc151b6f9f342b467b9
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > >   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC2/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1461/
> > >
> > > Artifacts were signed with the 0xA7874F29 key which can be found in:
> > >
> > >   https://dist.apache.org/repos/dist/release/hbase/KEYS
> > >
> > > Known issues to be aware of:
> > >
> > >  HBASE-26120 New replication gets stuck or data loss when multiwal
> groups
> > > more than 10
> > >
> > > To learn more about Apache hbase, please see
> > >
> > >   http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
>

Re: [VOTE] First release candidate for HBase 2.3.6 (RC2) is available

[Stack]

(Hit send by mistake).

Built src tgz and run pe and verified data landed. Poked around shell and
UI. Compatibility API looks good.

+1

S

On Tue, Jul 27, 2021 at 3:19 PM Stack  wrote:

> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_292): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_292): ok
>  - mvn clean install  -DskipTests
> * Unit tests pass (1.8.0_292): ok
>  - mvn package -P runSmallTests
>  -Dsurefire.rerunFailingTestsCount=3
>
>
>
> On Mon, Jul 26, 2021 at 11:14 PM Stack  wrote:
>
>> Please vote on this Apache hbase release candidate,
>> hbase-2.3.6RC2
>>
>> The VOTE will remain open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache hbase 2.3.6
>> [ ] -1 Do not release this package because ...
>>
>> The tag to be voted on is 2.3.6RC2:
>>
>>   https://github.com/apache/hbase/tree/2.3.6RC2
>>
>> This tag currently points to git reference
>>
>>   79d57a453ac840873ca15fc151b6f9f342b467b9
>>
>> The release files, including signatures, digests, as well as CHANGES.md
>> and RELEASENOTES.md included in this RC can be found at:
>>
>>   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC2/
>>
>> Maven artifacts are available in a staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1461/
>>
>> Artifacts were signed with the 0xA7874F29 key which can be found in:
>>
>>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>>
>> Known issues to be aware of:
>>
>>  HBASE-26120 New replication gets stuck or data loss when multiwal groups
>> more than 10
>>
>> To learn more about Apache hbase, please see
>>
>>   http://hbase.apache.org/
>>
>> Thanks,
>> Your HBase Release Manager
>>
>

Re: [VOTE] First release candidate for HBase 2.3.6 (RC2) is available

[Stack]

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_292): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_292): ok
 - mvn clean install  -DskipTests
* Unit tests pass (1.8.0_292): ok
 - mvn package -P runSmallTests  -Dsurefire.rerunFailingTestsCount=3



On Mon, Jul 26, 2021 at 11:14 PM Stack  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.3.6RC2
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.3.6
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.3.6RC2:
>
>   https://github.com/apache/hbase/tree/2.3.6RC2
>
> This tag currently points to git reference
>
>   79d57a453ac840873ca15fc151b6f9f342b467b9
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC2/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1461/
>
> Artifacts were signed with the 0xA7874F29 key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> Known issues to be aware of:
>
>  HBASE-26120 New replication gets stuck or data loss when multiwal groups
> more than 10
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Resolved] (HBASE-26146) Allow custom opts for hbck in hbase bin

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26146.
---
Fix Version/s: 3.0.0-alpha-2
   2.5.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Thanks for the patch [~bbeaudreault] Merged to branch-2+.  It didn't go to 
branch-2.4. Conflicts. Make a subtask for a backport if you want it in 2.4/2.3 
boss.

> Allow custom opts for hbck in hbase bin
> ---
>
> Key: HBASE-26146
> URL: https://issues.apache.org/jira/browse/HBASE-26146
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Minor
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> https://issues.apache.org/jira/browse/HBASE-15145 made it so that when you 
> execute {{hbase hbck}}, the regionserver or JAAS opts are added automatically 
> to the command line. This is problematic in some cases depending on what 
> regionserver opts have been set. For instance, one might configure a jmx port 
> for the regionserver but then hbck will fail due to a port conflict if run on 
> the same host as a regionserver. Another example would be that a regionserver 
> might define an {{-Xms}} value which is significantly more than hbck requires.
>  
> We should make it possible for users to define their own HBASE_HBCK_OPTS 
> which take precedence over the server opts added by default.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] First release candidate for HBase 2.3.6 (RC2) is available

[Stack]

Please vote on this Apache hbase release candidate,
hbase-2.3.6RC2

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 2.3.6
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.3.6RC2:

  https://github.com/apache/hbase/tree/2.3.6RC2

This tag currently points to git reference

  79d57a453ac840873ca15fc151b6f9f342b467b9

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/2.3.6RC2/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1461/

Artifacts were signed with the 0xA7874F29 key which can be found in:

  https://dist.apache.org/repos/dist/release/hbase/KEYS

Known issues to be aware of:

 HBASE-26120 New replication gets stuck or data loss when multiwal groups
more than 10

To learn more about Apache hbase, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

[jira] [Resolved] (HBASE-26001) When turn on access control, the cell level TTL of Increment and Append operations is invalid.

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26001.
---
Resolution: Fixed

Removed 2.3.6 as fix version after revert.

> When turn on access control, the cell level TTL of Increment and Append 
> operations is invalid.
> --
>
> Key: HBASE-26001
> URL: https://issues.apache.org/jira/browse/HBASE-26001
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Minor
> Fix For: 2.6.7, 2.5.0, 2.4.5, 3.0.0-alpha-1
>
>
> AccessController postIncrementBeforeWAL() and postAppendBeforeWAL() methods 
> will rewrite the new cell's tags by the old cell's. This will makes the other 
> kinds of tag in new cell invisible (such as TTL tag) after this. As in 
> Increment and Append operations, the new cell has already catch forward all 
> tags of the old cell and TTL tag from mutation operation, here in 
> AccessController we do not need to rewrite the tags once again. Also, the TTL 
> tag of newCell will be invisible in the new created cell. Actually, in 
> Increment and Append operations, the newCell has already copied all tags of 
> the oldCell. So the oldCell is useless here.
> {code:java}
> private Cell createNewCellWithTags(Mutation mutation, Cell oldCell, Cell 
> newCell) {
> // Collect any ACLs from the old cell
> List tags = Lists.newArrayList();
> List aclTags = Lists.newArrayList();
> ListMultimap perms = ArrayListMultimap.create();
> if (oldCell != null) {
>   Iterator tagIterator = PrivateCellUtil.tagsIterator(oldCell);
>   while (tagIterator.hasNext()) {
> Tag tag = tagIterator.next();
> if (tag.getType() != PermissionStorage.ACL_TAG_TYPE) {
>   // Not an ACL tag, just carry it through
>   if (LOG.isTraceEnabled()) {
> LOG.trace("Carrying forward tag from " + oldCell + ": type " + 
> tag.getType()
> + " length " + tag.getValueLength());
>   }
>   tags.add(tag);
> } else {
>   aclTags.add(tag);
> }
>   }
> }
> // Do we have an ACL on the operation?
> byte[] aclBytes = mutation.getACL();
> if (aclBytes != null) {
>   // Yes, use it
>   tags.add(new ArrayBackedTag(PermissionStorage.ACL_TAG_TYPE, aclBytes));
> } else {
>   // No, use what we carried forward
>   if (perms != null) {
> // TODO: If we collected ACLs from more than one tag we may have a
> // List of size > 1, this can be collapsed into a single
> // Permission
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Carrying forward ACLs from " + oldCell + ": " + perms);
> }
> tags.addAll(aclTags);
>   }
> }
> // If we have no tags to add, just return
> if (tags.isEmpty()) {
>   return newCell;
> }
> // Here the new cell's tags will be in visible.
> return PrivateCellUtil.createCell(newCell, tags);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-26001) When turn on access control, the cell level TTL of Increment and Append operations is invalid.

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack reopened HBASE-26001:
---

Reopening to revert from branch-2.3.   The  new test added here is failing 100% 
on branch-2.3. See bottom of 
[https://ci-hadoop.apache.org/view/HBase/job/HBase/job/HBase-Find-Flaky-Tests/job/branch-2.3/lastSuccessfulBuild/artifact/output/dashboard.html]
 Thanks.

> When turn on access control, the cell level TTL of Increment and Append 
> operations is invalid.
> --
>
> Key: HBASE-26001
> URL: https://issues.apache.org/jira/browse/HBASE-26001
> Project: HBase
>  Issue Type: Bug
>  Components: Coprocessors
>Reporter: Yutong Xiao
>Assignee: Yutong Xiao
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.6.7, 2.5.0, 2.3.6, 2.4.5
>
>
> AccessController postIncrementBeforeWAL() and postAppendBeforeWAL() methods 
> will rewrite the new cell's tags by the old cell's. This will makes the other 
> kinds of tag in new cell invisible (such as TTL tag) after this. As in 
> Increment and Append operations, the new cell has already catch forward all 
> tags of the old cell and TTL tag from mutation operation, here in 
> AccessController we do not need to rewrite the tags once again. Also, the TTL 
> tag of newCell will be invisible in the new created cell. Actually, in 
> Increment and Append operations, the newCell has already copied all tags of 
> the oldCell. So the oldCell is useless here.
> {code:java}
> private Cell createNewCellWithTags(Mutation mutation, Cell oldCell, Cell 
> newCell) {
> // Collect any ACLs from the old cell
> List tags = Lists.newArrayList();
> List aclTags = Lists.newArrayList();
> ListMultimap perms = ArrayListMultimap.create();
> if (oldCell != null) {
>   Iterator tagIterator = PrivateCellUtil.tagsIterator(oldCell);
>   while (tagIterator.hasNext()) {
> Tag tag = tagIterator.next();
> if (tag.getType() != PermissionStorage.ACL_TAG_TYPE) {
>   // Not an ACL tag, just carry it through
>   if (LOG.isTraceEnabled()) {
> LOG.trace("Carrying forward tag from " + oldCell + ": type " + 
> tag.getType()
> + " length " + tag.getValueLength());
>   }
>   tags.add(tag);
> } else {
>   aclTags.add(tag);
> }
>   }
> }
> // Do we have an ACL on the operation?
> byte[] aclBytes = mutation.getACL();
> if (aclBytes != null) {
>   // Yes, use it
>   tags.add(new ArrayBackedTag(PermissionStorage.ACL_TAG_TYPE, aclBytes));
> } else {
>   // No, use what we carried forward
>   if (perms != null) {
> // TODO: If we collected ACLs from more than one tag we may have a
> // List of size > 1, this can be collapsed into a single
> // Permission
> if (LOG.isTraceEnabled()) {
>   LOG.trace("Carrying forward ACLs from " + oldCell + ": " + perms);
> }
> tags.addAll(aclTags);
>   }
> }
> // If we have no tags to add, just return
> if (tags.isEmpty()) {
>   return newCell;
> }
> // Here the new cell's tags will be in visible.
> return PrivateCellUtil.createCell(newCell, tags);
>   }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-25165) Change 'State time' in UI so sorts

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack reopened HBASE-25165:
---

Reopen to push on branch-2.3

> Change 'State time' in UI so sorts
> --
>
> Key: HBASE-25165
> URL: https://issues.apache.org/jira/browse/HBASE-25165
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: Screen Shot 2020-10-07 at 4.15.32 PM.png, Screen Shot 
> 2020-10-07 at 4.15.42 PM.png
>
>
> Here is a minor issue.
> I had an issue w/ crashing servers. The servers were auto-restarted on crash.
> To find the crashing servers, I was sorting on the 'Start time' column in the 
> Master UI. This basically worked but the sort is unreliable as the date we 
> display starts with days-of-the-week.
> This issue is about moving to display start time in iso8601 which is sortable 
> (and occupies less real estate). Let me add some images.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25165) Change 'State time' in UI so sorts

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25165.
---
Resolution: Fixed

Pushed on branch-2.3. Re-resolving.

> Change 'State time' in UI so sorts
> --
>
> Key: HBASE-25165
> URL: https://issues.apache.org/jira/browse/HBASE-25165
> Project: HBase
>  Issue Type: Bug
>  Components: UI
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Minor
> Fix For: 2.3.6, 2.4.0, 3.0.0-alpha-1
>
> Attachments: Screen Shot 2020-10-07 at 4.15.32 PM.png, Screen Shot 
> 2020-10-07 at 4.15.42 PM.png
>
>
> Here is a minor issue.
> I had an issue w/ crashing servers. The servers were auto-restarted on crash.
> To find the crashing servers, I was sorting on the 'Start time' column in the 
> Master UI. This basically worked but the sort is unreliable as the date we 
> display starts with days-of-the-week.
> This issue is about moving to display start time in iso8601 which is sortable 
> (and occupies less real estate). Let me add some images.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Rolling a 2.3.6 in next week or so (WAS => Delaying 2.3.6)

[Stack]

On Thu, Jul 15, 2021 at 10:16 AM Stack  wrote:

> Get any 'critical' commits landed in next few days. Give me a shout if you
> need help or are wondering whether a commit belongs or not.
>
>
Starting the 2.3.6 roll shout if you have a critical to get in!
S




> Thanks,
> S
>
> On Fri, Apr 30, 2021 at 9:19 AM Nick Dimiduk  wrote:
>
>> Heya,
>>
>> We only have about 25 commits on branch-2.3 since 2.3.5, which is less
>> than
>> my benchmark of at least 30 commits for a patch release. I propose we hold
>> off another month or so. If there's a fix in that set which is critical
>> for
>> you, speak up and we can discuss it. Otherwise, let's save the PMC
>> attention for the next 2.4 release.
>>
>> Thanks,
>> Nick
>>
>

Re: Split Meta Design Reset Status

[Stack]

Notes from yesterday's meeting (attendees, please amend if I misrepresent
or if you have anything extra to add!)

Split Meta Design Reset Status

Wed Jul 21 21:24:38 PDT 2021

Attendees: Bharath, Stack, Duo, and Francis

We went over the new updates to the Brainstorming [1] section under

Design in the Super Split Meta Design doc [2].

First was the new addition, 4.1.2 Extend (& Move) ConnectionRegistry; hide

ROOT from Client [3]. In particular, filling out how "ROOT" might be

implemented behind the new API in ConnectionRegistry. On option 1.,

replicating master-local Region to RegionServers, options considered

included

 * Listener on master-local Region WAL to replicate.

 * Perhaps Read-Replica but master-local is not an actual Region

 * Needs to be incremental edits because ROOT could get too big to ship

   in a lump; need to visit how...

 * Possibly in-memory-only Regions on RS replicated from master-local

   Region via WAL tailing <= zhang...@apache.org

 * Which RegionServers? Those hosting ROOT replicas?

 * How to bootstrap? Failure scenarios.

 * This would be a new replication system alongside current; could

   evolve to replace/improve old?

Duo offered to look into means of replicating the master-local Region

out to RegionServers.

Next up was discussion constrasting ROOT as a standalone table vs

First-meta Region-as-Root (see 4.1.1 hbase:meta,,1 as ROOT [4]); i.e.

options 2 and 3 for how we'd implement a ROOT. One item that came up

was whether a need to specify one replica count for a ROOT table vs

another for hbase:meta. If so, then it would be argument for ROOT as

standalone table (Others of us argued it not a concern of consequence).

If ROOT access is behind a new simple API in ConnectionRegistry, how

to stop clients reading hbase:meta table if not Master or fronted by

a ConnectionRegistry request? (Should be able to switch on client

identity/source). One suggestion for First-meta-Region-as-ROOT was

NOT returning the first Region to the client post-meta split when

accessing via the simple API. Some concern this would confuse old

Clients (Francis was going to take a look).

Moved to discussion how we'd move ConnectionRegistry from

hosted-by-Master to hosted-by-RegionServers. How to bootstrap such a

system came up? Where do clients go? How do they know which

RegionServers (special regionserver group?  Every RS fields

ConnectionRegistry requests but only designated core serve the ROOT

lookup APIs?). This was a TODO.

This led naturally into 4.1.5 System RS group for client meta services

[5], a new addition under Brainstorming. Discussion. Bharath to look

into feedback.

On the end of the discussion, group expressed support for adding

simple API to the ConnectionRegistry to hide ROOT implementation

detail from client. Support was expressed for moving ConnectionRegistry

from Master to RegionServers. Intent is to move forward on design of

these pieces: e.g. how client bootstraps.

Support was expressed for getting at least the bones of a split meta

into an hbase3 before the RCs.

Where we'd actually store hbase:meta Region locations -- i.e. how a

"ROOT' would be implemented -- was for our next meeting informed by

research of the various approaches noted mostly above. It was

also thought that the new ConnectionRegistry should not preclude

making progress on the "ROOT" implementation.

Will post notice of next meeting (next Weds or the one

following).

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n

2.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.9s666p6no9cq

3.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.90th11txi153

4.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle

5.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.utoenf10t05b

On Tue, Jul 20, 2021 at 11:00 AM Stack  wrote:

> Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
> Section 4.1 [1] before the meeting if you can (No harm if a refresh of the
> requirements section while you are at it).
>
> Topic: Split Meta Design Reset Status
> Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09
>
> Meeting ID: 773 1892 0525
> Passcode: hbase
>
> Thanks,
> S
>
> 1. 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n
>
> On Thu, Jul 8, 2021 at 1:04 PM Stack  wrote:
>
>> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
>> room' -- sorry Duo)
>>
>> Split Meta Status Zoom Meeting
>> Wed Jul  7, 2021 @ 

[jira] [Resolved] (HBASE-26062) SIGSEGV in AsyncFSWAL consume

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-26062.
---
Resolution: Duplicate

Thanks [~anoop.hbase]  There is ASYNC_WAL on this cluster afterall (when I 
wrote the above, thought there was none). Resolving as duplicate of what we see 
over on HBASE-24984

> SIGSEGV in AsyncFSWAL consume
> -
>
> Key: HBASE-26062
> URL: https://issues.apache.org/jira/browse/HBASE-26062
> Project: HBase
>  Issue Type: Bug
>    Reporter: Michael Stack
>Priority: Major
>
> Seems related to the parent issue. Its happened a few times on one of our 
> clusters here. Below are two examples. Need more detail but perhaps the call 
> has timed out, the buffer has thus been freed, but the late consume on the 
> other side of the ringbuffer doesn't know that and goes ahead (Just 
> speculation).
>  
> {code:java}
> #  SIGSEGV (0xb) at pc=0x7f8b3ef5b77c, pid=37631, tid=0x7f61560ed700
> RAX=0xdf6e is an unknown valueRBX=0x7f8a38d7b6f8 is an 
> oopjava.nio.DirectByteBuffer - klass: 
> 'java/nio/DirectByteBuffer'RCX=0x7f60e2767898 is pointing into 
> metadataRDX=0x0de7 is an unknown valueRSP=0x7f61560ec6f0 is 
> pointing into the stack for thread: 0x7f8b3017b800RBP=[error occurred 
> during error reporting (printing register info), id 0xb]
> Stack: [0x7f6155fed000,0x7f61560ee000],  sp=0x7f61560ec6f0,  free 
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, 
> C=native code)J 23901 C2 
> java.util.stream.MatchOps$1MatchSink.accept(Ljava/lang/Object;)V (44 bytes) @ 
> 0x7f8b3ef5b77c [0x7f8b3ef5b640+0x13c]J 16165 C2 
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
>  (79 bytes) @ 0x7f8b3d67b344 [0x7f8b3d67b2c0+0x84]J 16160 C2 
> java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
>  (7 bytes) @ 0x7f8b3d67bc9c [0x7f8b3d67b900+0x39c]J 17729 C2 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(Lorg/apache/hadoop/hbase/wal/WALKey;Lorg/apache/hadoop/hbase/wal/WALEdit;)V
>  (10 bytes) @ 0x7f8b3fc39010 [0x7f8b3fc388a0+0x770]J 29991 C2 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync()V (261 
> bytes) @ 0x7f8b3fd03d90 [0x7f8b3fd039e0+0x3b0]J 20773 C2 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume()V (474 bytes) @ 
> 0x7f8b40283728 [0x7f8b40283480+0x2a8]J 15191 C2 
> org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL$$Lambda$76.run()V (8 
> bytes) @ 0x7f8b3ed69ecc [0x7f8b3ed69ea0+0x2c]J 17383% C2 
> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
>  (225 bytes) @ 0x7f8b3d9423f8 [0x7f8b3d942260+0x198]j  
> java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5j  
> java.lang.Thread.run()V+11v  ~StubRoutines::call_stubV  [libjvm.so+0x66b9ba]  
> JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, 
> Thread*)+0xe1aV  [libjvm.so+0x669073]  JavaCalls::call_virtual(JavaValue*, 
> KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263V  
> [libjvm.so+0x669647]  JavaCalls::call_virtual(JavaValue*, Handle, 
> KlassHandle, Symbol*, Symbol*, Thread*)+0x57V  [libjvm.so+0x6aaa4c]  
> thread_entry(JavaThread*, Thread*)+0x6cV  [libjvm.so+0xa224cb]  
> JavaThread::thread_main_inner()+0xdbV  [libjvm.so+0xa22816]  
> JavaThread::run()+0x316V  [libjvm.so+0x8c4202]  java_start(Thread*)+0x102C  
> [libpthread.so.0+0x76ba]  start_thread+0xca {code}
>  
> This one is from a month previous and has a deeper stack... we're trying to 
> read a Cell...
>  
> {code:java}
> Stack: [0x7fa1d5fb8000,0x7fa1d60b9000],  sp=0x7fa1d60b7660,  free 
> space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, 
> C=native code)J 30665 C2 
> org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[BII)Z
>  (59 bytes) @ 0x7fcc2d29eeb2 [0x7fcc2d29e7c0+0x6f2]J 25816 C2 
> org.apache.hadoop.hbase.CellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[B)Z
>  (28 bytes) @ 0x7fcc2a0430f8 [0x7fcc2a0430e0+0x18]J 17236 C2 
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener$$Lambda$254.test(Ljava/lang/Object;)Z
>  (8 bytes) @ 0x7fcc2b40bc68 [0x7fcc2b40bc20+0x48]J 13735 C2 
> java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
>  (79 bytes) @ 0x7fcc

Re: Split Meta Design Reset Status

[Stack]

Lets meet tomorrow. Please review the design doc "Design/Brainstorming"
Section 4.1 [1] before the meeting if you can (No harm if a refresh of the
requirements section while you are at it).

Topic: Split Meta Design Reset Status
Time: Jul 21, 2021 05:00 PM Pacific Time (US and Canada)

Join Zoom Meeting
https://us04web.zoom.us/j/77318920525?pwd=OFZXZFVPSHJLaGNsby9SN25OV1F2Zz09

Meeting ID: 773 1892 0525
Passcode: hbase

Thanks,
S

1. 1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.wr0fwvw06j7n

On Thu, Jul 8, 2021 at 1:04 PM Stack  wrote:

> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
> room' -- sorry Duo)
>
> Split Meta Status Zoom Meeting
> Wed Jul  7, 2021 @ 5pm for ~90minutes
> Attendees: Duo, Francis, Stack, and Clay
> Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
> added to the
> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
> approach)
>
> Duo suggested no advantage treating the first meta of hbase:meta table
> special; as a "root"
> and other comments (see remarks in [2]).
>
> Some pushback. When meta is NOT split, all works as it did before (big
> on backward-compatible).
>
> Duo suggested just intro a ROOT table altogether rather than treat the
> first Region
> in the hbase:meta table as a 'root' and then mirror to zk the first meta
> region; if no
> split, then old clients should just work even though now hbase cluster has
> a ROOT table.
> Discussion. If no split, what's to do, etc.?
>
> Duo expressed concern that if split-meta is not on always -- enabled
> optionally -- then
> the code path will not see exercise and split-meta will likely fail and go
> the way of
> prefix tree and distributed log replay -- a feature that failed, cluttered
> up the
> code base, and only later was removed. Discussion. Was allowed this could
> happen.
> Counter examples: RegionServer Groups. A few of the attendees volunteered
> they need
> split meta for their production so would try to see it through.
>
> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
> if we don't
> split; not much changes.
>
> Comment on the PoC that its all
>
>   if 'first special meta region' do this
>   else do something else...
>
> (all over the codebase) but it was suggested that this will be the case if
> ROOT table
> added also and argued any implementation will have this issue (if ROOT
> then) and
> THAT ugly 'root' comparator too whether a ROOT table or the
>  'hbase:meta,,1 as ROOT'
> approach.
>
> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> 'new-to-the-topic'
> roles). Some discussion around how indeed it is and why we want to split
> meta at all.
>
> Clay mentioned how Master is inline for data now (Master-hosted
> Registry)
> Discussion. Hopefully temporary state -- Registry doesn't need to be
> hosted on
> Master -- and Master will return to its background role -- soon.
>
> Clay brought up rollback after meta split. Discussion. Some agreement it
> could be done for
> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
> client had
> started to use ROOT, it might be harder...
>
> Duo suggested that we not change the client at all; that client stays same
> however split
> meta is implemented (with addition of a root table or using hbase:meta,,1
> as 'root').
> This sounded attractive. We discussed how this could be done in a backward
> compatible way;
> add simple location lookup API to Registry...A write-up on how this might
> work will be posted
> in next day or so for others to review (Need to figure getting Registry
> off Master).
>
> Clay suggested, as an operator, how he thought the split meta should roll
> out. One of us
> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>
> Duo had to go to work so we called it quits and said we'd meet again same
> time, next week.
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> 2. https://issues.apache.org/jira/browse/HBASE-25761
>
> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) 
> wrote:
>
>> Is it only me? I tried to join the meeting but it always tell me that I
>> need to wait for the presenter to invite me to join...
>>
>> Stack  于2021年7月8日周四 上午1:04写道：
>>
>> > Short notice but reminder that this meeting is today at 5pm PST (We
>> talked
>> > of doing it earlier but in the end lets just keep the original 5pm
>> time).
>> > Thanks,
>> > S
>> >
>> > On Tue, Jul 6, 2021 at 11:36 AM Stack  wrote:
>

Re: [DISCUSS] Remove FirstKeyValueMatchingQualifiersFilter

[Stack]

I suggest keeping the proto.
S

On Sat, Jul 17, 2021 at 1:15 PM 黄卓越  wrote:

> Hi,
>
> As https://issues.apache.org/jira/browse/HBASE-13347 describes, the
> FirstKeyValueMatchingQualifiersFilter was deprecated in 2.0 and will be
> removed in 3.0. Now we need to remove it in 3.0 but there are proto
> declareation of FirstKeyValueMatchingQualifiersFilter in Filter.proto.
>
> If the proto declaration of FirstKeyValueMatchingQualifiersFilter is
> removed, serialization problems and unknown errors may occur when the old
> version of hbase-client accesses the new version of the server, which may
> puzzle users.
>
> Should we remove the proto declaration directly? Or throw RuntimeException
> when the user uses FirstKeyValueMatchingQualifiersFilter? Hope to get your
> suggestions.
>
> Thanks,
> Zhuoyue Huang
>
> 发自我的iPhone

[jira] [Resolved] (HBASE-25739) TableSkewCostFunction need to use aggregated deviation

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25739.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Resovling after merging PRs for branch-2.3+. Thanks for the patch 
[~clarax98007]  (and reviews [~busbey] and [~zhangduo] )

> TableSkewCostFunction need to use aggregated deviation
> --
>
> Key: HBASE-25739
> URL: https://issues.apache.org/jira/browse/HBASE-25739
> Project: HBase
>  Issue Type: Sub-task
>  Components: Balancer, master
>Reporter: Clara Xiong
>Assignee: Clara Xiong
>Priority: Major
> Fix For: 2.5.0, 2.3.6, 3.0.0-alpha-2, 2.4.5
>
> Attachments: 
> TEST-org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerBalanceCluster.xml,
>  
> org.apache.hadoop.hbase.master.balancer.TestStochasticLoadBalancerBalanceCluster.txt
>
>
> TableSkewCostFunction uses the sum of the max deviation region per server for 
> all tables as the measure of unevenness. It doesn't work in a very common 
> scenario in operations. Say we have 100 regions on 50 nodes, two on each. We 
> add 50 new nodes and they have 0 each. The max deviation from the mean is 1, 
> compared to 99 in the worst case scenario of 100 regions on a single server. 
> The normalized cost is 1/99 = 0.011 < default threshold of 0.05. Balancer 
> wouldn't move.  The proposal is to use aggregated deviation of the count per 
> region server to detect this scenario, generating a cost of 100/198 = 0.5 in 
> this case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Rolling a 2.3.6 in next week or so (WAS => Delaying 2.3.6)

[Stack]

Get any 'critical' commits landed in next few days. Give me a shout if you
need help or are wondering whether a commit belongs or not.

Thanks,
S

On Fri, Apr 30, 2021 at 9:19 AM Nick Dimiduk  wrote:

> Heya,
>
> We only have about 25 commits on branch-2.3 since 2.3.5, which is less than
> my benchmark of at least 30 commits for a patch release. I propose we hold
> off another month or so. If there's a fix in that set which is critical for
> you, speak up and we can discuss it. Otherwise, let's save the PMC
> attention for the next 2.4 release.
>
> Thanks,
> Nick
>

Re: Split Meta Design Reset Status

[Stack]

No meeting tomorrow (A few of us can't make it). Lets do it next week.
Meantime, some updates on the design doc under brainstorming. Check it out.
Feedback/opinions/input appreciated.
S

On Thu, Jul 8, 2021 at 1:04 PM Stack  wrote:

> Meeting notes (Meeting happend after I figured the zoom had a 'waiting
> room' -- sorry Duo)
>
> Split Meta Status Zoom Meeting
> Wed Jul  7, 2021 @ 5pm for ~90minutes
> Attendees: Duo, Francis, Stack, and Clay
> Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
> added to the
> split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
> approach)
>
> Duo suggested no advantage treating the first meta of hbase:meta table
> special; as a "root"
> and other comments (see remarks in [2]).
>
> Some pushback. When meta is NOT split, all works as it did before (big
> on backward-compatible).
>
> Duo suggested just intro a ROOT table altogether rather than treat the
> first Region
> in the hbase:meta table as a 'root' and then mirror to zk the first meta
> region; if no
> split, then old clients should just work even though now hbase cluster has
> a ROOT table.
> Discussion. If no split, what's to do, etc.?
>
> Duo expressed concern that if split-meta is not on always -- enabled
> optionally -- then
> the code path will not see exercise and split-meta will likely fail and go
> the way of
> prefix tree and distributed log replay -- a feature that failed, cluttered
> up the
> code base, and only later was removed. Discussion. Was allowed this could
> happen.
> Counter examples: RegionServer Groups. A few of the attendees volunteered
> they need
> split meta for their production so would try to see it through.
>
> Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
> if we don't
> split; not much changes.
>
> Comment on the PoC that its all
>
>   if 'first special meta region' do this
>   else do something else...
>
> (all over the codebase) but it was suggested that this will be the case if
> ROOT table
> added also and argued any implementation will have this issue (if ROOT
> then) and
> THAT ugly 'root' comparator too whether a ROOT table or the
>  'hbase:meta,,1 as ROOT'
> approach.
>
> Clay asking if meta is a bottleneck (Clay played the 'operator' and
> 'new-to-the-topic'
> roles). Some discussion around how indeed it is and why we want to split
> meta at all.
>
> Clay mentioned how Master is inline for data now (Master-hosted
> Registry)
> Discussion. Hopefully temporary state -- Registry doesn't need to be
> hosted on
> Master -- and Master will return to its background role -- soon.
>
> Clay brought up rollback after meta split. Discussion. Some agreement it
> could be done for
> 'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
> client had
> started to use ROOT, it might be harder...
>
> Duo suggested that we not change the client at all; that client stays same
> however split
> meta is implemented (with addition of a root table or using hbase:meta,,1
> as 'root').
> This sounded attractive. We discussed how this could be done in a backward
> compatible way;
> add simple location lookup API to Registry...A write-up on how this might
> work will be posted
> in next day or so for others to review (Need to figure getting Registry
> off Master).
>
> Clay suggested, as an operator, how he thought the split meta should roll
> out. One of us
> claimed this described the 'hbase:meta,,1 as ROOT' approach.
>
> Duo had to go to work so we called it quits and said we'd meet again same
> time, next week.
>
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
> 2. https://issues.apache.org/jira/browse/HBASE-25761
>
> On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang) 
> wrote:
>
>> Is it only me? I tried to join the meeting but it always tell me that I
>> need to wait for the presenter to invite me to join...
>>
>> Stack  于2021年7月8日周四 上午1:04写道：
>>
>> > Short notice but reminder that this meeting is today at 5pm PST (We
>> talked
>> > of doing it earlier but in the end lets just keep the original 5pm
>> time).
>> > Thanks,
>> > S
>> >
>> > On Tue, Jul 6, 2021 at 11:36 AM Stack  wrote:
>> >
>> > > Lets do a quick chat on current state of hbase split-meta project.
>> > >
>> > > Francis has posted a suggested one-pager design in HBASE-25761 which
>> > would
>> > > be good to review before the meeting if you can. Lets discuss this and
>> > any
>> > > other suggestions for 

Re: Split Meta Design Reset Status

[Stack]

Meeting notes (Meeting happend after I figured the zoom had a 'waiting
room' -- sorry Duo)

Split Meta Status Zoom Meeting
Wed Jul  7, 2021 @ 5pm for ~90minutes
Attendees: Duo, Francis, Stack, and Clay
Agenda: Mainly talk about the one-pager design and PoC proposed in [2] and
added to the
split-meta design doc here [1] (hereafter, the 'hbase:meta,,1 as ROOT'
approach)

Duo suggested no advantage treating the first meta of hbase:meta table
special; as a "root"
and other comments (see remarks in [2]).

Some pushback. When meta is NOT split, all works as it did before (big
on backward-compatible).

Duo suggested just intro a ROOT table altogether rather than treat the
first Region
in the hbase:meta table as a 'root' and then mirror to zk the first meta
region; if no
split, then old clients should just work even though now hbase cluster has
a ROOT table.
Discussion. If no split, what's to do, etc.?

Duo expressed concern that if split-meta is not on always -- enabled
optionally -- then
the code path will not see exercise and split-meta will likely fail and go
the way of
prefix tree and distributed log replay -- a feature that failed, cluttered
up the
code base, and only later was removed. Discussion. Was allowed this could
happen.
Counter examples: RegionServer Groups. A few of the attendees volunteered
they need
split meta for their production so would try to see it through.

Some back and forth. Duo allowed that merit to the 'hbase:meta,,1 as ROOT'
if we don't
split; not much changes.

Comment on the PoC that its all

  if 'first special meta region' do this
  else do something else...

(all over the codebase) but it was suggested that this will be the case if
ROOT table
added also and argued any implementation will have this issue (if ROOT
then) and
THAT ugly 'root' comparator too whether a ROOT table or the  'hbase:meta,,1
as ROOT'
approach.

Clay asking if meta is a bottleneck (Clay played the 'operator' and
'new-to-the-topic'
roles). Some discussion around how indeed it is and why we want to split
meta at all.

Clay mentioned how Master is inline for data now (Master-hosted
Registry)
Discussion. Hopefully temporary state -- Registry doesn't need to be hosted
on
Master -- and Master will return to its background role -- soon.

Clay brought up rollback after meta split. Discussion. Some agreement it
could be done for
'hbase:meta,,1 as ROOT' approach (but would be UGLY!). If root table and
client had
started to use ROOT, it might be harder...

Duo suggested that we not change the client at all; that client stays same
however split
meta is implemented (with addition of a root table or using hbase:meta,,1
as 'root').
This sounded attractive. We discussed how this could be done in a backward
compatible way;
add simple location lookup API to Registry...A write-up on how this might
work will be posted
in next day or so for others to review (Need to figure getting Registry off
Master).

Clay suggested, as an operator, how he thought the split meta should roll
out. One of us
claimed this described the 'hbase:meta,,1 as ROOT' approach.

Duo had to go to work so we called it quits and said we'd meet again same
time, next week.

1.
https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.ikbhxlcthjle
2. https://issues.apache.org/jira/browse/HBASE-25761

On Wed, Jul 7, 2021 at 5:09 PM 张铎(Duo Zhang)  wrote:

> Is it only me? I tried to join the meeting but it always tell me that I
> need to wait for the presenter to invite me to join...
>
> Stack  于2021年7月8日周四 上午1:04写道：
>
> > Short notice but reminder that this meeting is today at 5pm PST (We
> talked
> > of doing it earlier but in the end lets just keep the original 5pm time).
> > Thanks,
> > S
> >
> > On Tue, Jul 6, 2021 at 11:36 AM Stack  wrote:
> >
> > > Lets do a quick chat on current state of hbase split-meta project.
> > >
> > > Francis has posted a suggested one-pager design in HBASE-25761 which
> > would
> > > be good to review before the meeting if you can. Lets discuss this and
> > any
> > > other suggestions for moving the project forward.
> > >
> > > Meeting details below.
> > >
> > > Thanks,
> > > S
> > >
> > > Topic Split Meta Design Reset Status
> > > Description Review one-pager design attached to
> > > https://issues.apache.org/jira/browse/HBASE-25761
> > > Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
> > >
> > > Meeting ID 736 3907 8852
> > > Security
> > > Passcode *hbase* Hide
> > > Waiting Room
> > > Invite Link
> > >
> >
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09
> > Copy
> > > Invitation
> > >
> > > On Mon, Mar 22, 2021 at 4:28 PM Stack 

Re: Split Meta Design Reset Status

[Stack]

Short notice but reminder that this meeting is today at 5pm PST (We talked
of doing it earlier but in the end lets just keep the original 5pm time).
Thanks,
S

On Tue, Jul 6, 2021 at 11:36 AM Stack  wrote:

> Lets do a quick chat on current state of hbase split-meta project.
>
> Francis has posted a suggested one-pager design in HBASE-25761 which would
> be good to review before the meeting if you can. Lets discuss this and any
> other suggestions for moving the project forward.
>
> Meeting details below.
>
> Thanks,
> S
>
> Topic Split Meta Design Reset Status
> Description Review one-pager design attached to
> https://issues.apache.org/jira/browse/HBASE-25761
> Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)
>
> Meeting ID 736 3907 8852
> Security
> Passcode *hbase* Hide
> Waiting Room
> Invite Link
> https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 
> Copy
> Invitation
>
> On Mon, Mar 22, 2021 at 4:28 PM Stack  wrote:
>
>> Now the requirements are in [1], we're going to move to the next stage --
>> actual design for split-meta -- and have set up a chat for this thursday
>> afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
>> Please come if interested. Zoom details are below.
>>
>> Yours,
>> S
>> 1.
>> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>>
>>
>> Topic: hbase split-meta design warmup chat
>> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>>
>> Join Zoom Meeting
>> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>>
>> Meeting ID: 759 8800 3798
>> Passcode: hbase
>>
>>
>> On Tue, Jan 5, 2021 at 9:13 AM Stack  wrote:
>>
>>> FYI, a few of us have been working on the redo/reset of the split meta
>>> design (HBASE-25382). We (think we've) finished the requirements. Are there
>>> any others to consider?
>>>
>>> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>>>
>>> Thanks,
>>> S
>>>
>>

Re: Split Meta Design Reset Status

[Stack]

 Lets do a quick chat on current state of hbase split-meta project.

Francis has posted a suggested one-pager design in HBASE-25761 which would
be good to review before the meeting if you can. Lets discuss this and any
other suggestions for moving the project forward.

Meeting details below.

Thanks,
S

Topic Split Meta Design Reset Status
Description Review one-pager design attached to
https://issues.apache.org/jira/browse/HBASE-25761
Time Jul 7, 2021 05:00 PM Pacific Time (US and Canada)

Meeting ID 736 3907 8852
Security
Passcode *hbase* Hide
Waiting Room
Invite Link
https://us04web.zoom.us/j/73639078852?pwd=eHE2VCt2RG5FV2V2RXVNazlPVTVyQT09 Copy
Invitation

On Mon, Mar 22, 2021 at 4:28 PM Stack  wrote:

> Now the requirements are in [1], we're going to move to the next stage --
> actual design for split-meta -- and have set up a chat for this thursday
> afternoon (4PM California time/8AM Beijing time) to get the ball rolling.
> Please come if interested. Zoom details are below.
>
> Yours,
> S
> 1.
> https://docs.google.com/document/d/11ChsSb2LGrSzrSJz8pDCAw5IewmaMV0ZDN1LrMkAj4s/edit#heading=h.hdf0rnyevxz2
>
>
> Topic: hbase split-meta design warmup chat
> Time: Mar 25, 2021 04:00 PM Pacific Time (US and Canada)
>
> Join Zoom Meeting
> https://us04web.zoom.us/j/75988003798?pwd=Wi9mU0w0T2ZjTFNBaE9lUmtTbHRpQT09
>
> Meeting ID: 759 8800 3798
> Passcode: hbase
>
>
> On Tue, Jan 5, 2021 at 9:13 AM Stack  wrote:
>
>> FYI, a few of us have been working on the redo/reset of the split meta
>> design (HBASE-25382). We (think we've) finished the requirements. Are there
>> any others to consider?
>>
>> Feedback and contribs welcome. Otherwise, on to the next phase -- design.
>>
>> Thanks,
>> S
>>
>

[jira] [Created] (HBASE-26062) SIGSEGV in AsyncFSWAL consume

[Michael Stack (Jira)]

Michael Stack created HBASE-26062:
-

 Summary: SIGSEGV in AsyncFSWAL consume
 Key: HBASE-26062
 URL: https://issues.apache.org/jira/browse/HBASE-26062
 Project: HBase
  Issue Type: Sub-task
Reporter: Michael Stack


Seems related to the parent issue. Its happened a few times on one of our 
clusters here. Below are two examples. Need more detail but perhaps the call 
has timed out, the buffer has thus been freed, but the late consume on the 
other side of the ringbuffer doesn't know that and goes ahead (Just 
speculation).

 
{code:java}
#  SIGSEGV (0xb) at pc=0x7f8b3ef5b77c, pid=37631, tid=0x7f61560ed700

RAX=0xdf6e is an unknown valueRBX=0x7f8a38d7b6f8 is an 
oopjava.nio.DirectByteBuffer - klass: 
'java/nio/DirectByteBuffer'RCX=0x7f60e2767898 is pointing into 
metadataRDX=0x0de7 is an unknown valueRSP=0x7f61560ec6f0 is 
pointing into the stack for thread: 0x7f8b3017b800RBP=[error occurred 
during error reporting (printing register info), id 0xb]

Stack: [0x7f6155fed000,0x7f61560ee000],  sp=0x7f61560ec6f0,  free 
space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, 
C=native code)J 23901 C2 
java.util.stream.MatchOps$1MatchSink.accept(Ljava/lang/Object;)V (44 bytes) @ 
0x7f8b3ef5b77c [0x7f8b3ef5b640+0x13c]J 16165 C2 
java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
 (79 bytes) @ 0x7f8b3d67b344 [0x7f8b3d67b2c0+0x84]J 16160 C2 
java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
 (7 bytes) @ 0x7f8b3d67bc9c [0x7f8b3d67b900+0x39c]J 17729 C2 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(Lorg/apache/hadoop/hbase/wal/WALKey;Lorg/apache/hadoop/hbase/wal/WALEdit;)V
 (10 bytes) @ 0x7f8b3fc39010 [0x7f8b3fc388a0+0x770]J 29991 C2 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync()V (261 
bytes) @ 0x7f8b3fd03d90 [0x7f8b3fd039e0+0x3b0]J 20773 C2 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume()V (474 bytes) @ 
0x7f8b40283728 [0x7f8b40283480+0x2a8]J 15191 C2 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL$$Lambda$76.run()V (8 bytes) 
@ 0x7f8b3ed69ecc [0x7f8b3ed69ea0+0x2c]J 17383% C2 
java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
 (225 bytes) @ 0x7f8b3d9423f8 [0x7f8b3d942260+0x198]j  
java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5j  
java.lang.Thread.run()V+11v  ~StubRoutines::call_stubV  [libjvm.so+0x66b9ba]  
JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, 
Thread*)+0xe1aV  [libjvm.so+0x669073]  JavaCalls::call_virtual(JavaValue*, 
KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x263V  
[libjvm.so+0x669647]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, 
Symbol*, Symbol*, Thread*)+0x57V  [libjvm.so+0x6aaa4c]  
thread_entry(JavaThread*, Thread*)+0x6cV  [libjvm.so+0xa224cb]  
JavaThread::thread_main_inner()+0xdbV  [libjvm.so+0xa22816]  
JavaThread::run()+0x316V  [libjvm.so+0x8c4202]  java_start(Thread*)+0x102C  
[libpthread.so.0+0x76ba]  start_thread+0xca {code}
 

This one is from a month previous and has a deeper stack... we're trying to 
read a Cell...

 
{code:java}
Stack: [0x7fa1d5fb8000,0x7fa1d60b9000],  sp=0x7fa1d60b7660,  free 
space=1021kNative frames: (J=compiled Java code, j=interpreted, Vv=VM code, 
C=native code)J 30665 C2 
org.apache.hadoop.hbase.PrivateCellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[BII)Z
 (59 bytes) @ 0x7fcc2d29eeb2 [0x7fcc2d29e7c0+0x6f2]J 25816 C2 
org.apache.hadoop.hbase.CellUtil.matchingFamily(Lorg/apache/hadoop/hbase/Cell;[B)Z
 (28 bytes) @ 0x7fcc2a0430f8 [0x7fcc2a0430e0+0x18]J 17236 C2 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener$$Lambda$254.test(Ljava/lang/Object;)Z
 (8 bytes) @ 0x7fcc2b40bc68 [0x7fcc2b40bc20+0x48]J 13735 C2 
java.util.ArrayList$ArrayListSpliterator.tryAdvance(Ljava/util/function/Consumer;)Z
 (79 bytes) @ 0x7fcc2b7d936c [0x7fcc2b7d92c0+0xac]J 17162 C2 
java.util.stream.MatchOps$MatchOp.evaluateSequential(Ljava/util/stream/PipelineHelper;Ljava/util/Spliterator;)Ljava/lang/Object;
 (7 bytes) @ 0x7fcc29bc05e8 [0x7fcc29bbfe80+0x768]J 16934 C2 
org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceWALActionListener.visitLogEntryBeforeWrite(Lorg/apache/hadoop/hbase/wal/WALKey;Lorg/apache/hadoop/hbase/wal/WALEdit;)V
 (10 bytes) @ 0x7fcc2bb313f8 [0x7fcc2bb30c60+0x798]J 30732 C2 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.appendAndSync()V (261 
bytes) @ 0x7fcc2ae5a420 [0x7fcc2ae59d60+0x6c0]J 22203 C2 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.consume()V (474

Re: [DISCUSS] 1.7.0.1/1.7.1 release

[Stack]

+1 for 1.7.1
S

On Thu, Jul 1, 2021 at 10:39 AM Bharath Vissapragada 
wrote:

> Hi,
>
> As some of you may know, we have an incompatible serialization backport
> that landed in 1.7.0 (courtesy of yours truly) that is causing upgrade
> issues (thanks to Viraj for noticing it, more details in HBASE-26021
> ). We have to withdraw
> the release so as to not let 1.x users upgrade to it and instead do another
> release in the same line. We can either do a 1.7.0.1 (= 1.7.0 + HBASE-26021
> fix) or do a 1.7.1 which includes all the commits since 1.7.0 which is
> fairly small (listed below).
>
>  delta since 1.7.0 release =
> 7d0a72be14 (origin/branch-1) HBASE-22923 min version of RegionServer to
> move system table regions (#3438)
> 28f36f4619 HBASE-26021: Undo the incompatible serialization change in
> HBASE-7767 (#3435)
> 395eb0c8e0 HBASE-25130 - Fix master in-memory server holding map after:
> (#3402)
> b2f8ec993e HBASE-26025 Add a flag to mark if the IOError can be solved by
> retry in thrift IOError (#3429)
> fd2f8a581f HBASE-26013 Get operations readRows metrics becomes zero after
> HBASE-25677 (#3410)
> 7e57fecda8 HBASE-21674:Port HBASE-21652 (Refactor ThriftServer making
> thrift2 server inherited from thrift1 server) to branch-1 (#2941)
> 5263b8cf40 HBASE-26004: port HBASE-26001 (cell level tags invisible in
> atomic operations when access control is on)to branch-1 (#3387)
> 2e24bad826 HBASE-25984: Avoid premature reuse of sync futures in FSHLog
> (#3371) (#3398)
> a40f4583e3 Set version on branch-1 to 1.7.1-SNAPSHOT
> 0fd6eeb012 HBASE-25910 - Fix port assignment test (#3308)
> 782e24bd9b HBASE-25924 Re-compute size of WAL file while removing from
> WALEntryStream (#3316)
> =
>
> One of these (marked in red above) is a critical fix that was causing us
> issues, so I'd prefer to include it in either of the paths we take. Andrew
> was suggesting 1.7.0.1 in the jira comments (correct me if your definition
> of 1.7.0.1 is different than mine) while I'm leaning towards doing a 1.7.1
> since the delta is fairly small. Thoughts?
>
> I'm happy to be an RM for this release unless there is any objection.
>
> Thanks,
> Bharath
>

[jira] [Resolved] (HBASE-25821) Sorting by "Start time" in the Master UI does so alphabetically and not by date

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25821.
---
Resolution: Duplicate

Resolving as duplicate. Looks like it was done a while ago and will show in 
hbase-2.4.x.  See HBASE-25165 Shout if it ain't what you want [~ekrettek]

> Sorting by "Start time" in the Master UI does so alphabetically and not by 
> date
> ---
>
> Key: HBASE-25821
> URL: https://issues.apache.org/jira/browse/HBASE-25821
> Project: HBase
>  Issue Type: Bug
>  Components: master, UI
>Affects Versions: 2.4.0, 2.3.4, 2.4.1, 2.3.5, 2.4.2
>Reporter: Evan Krettek
>Priority: Minor
>
> When sorting by the start time column in the Maser UI it sorts alphabetically 
> and not by date



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25729) Upgrade to latest hbase-thirdparty

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25729.
---
Hadoop Flags: Reviewed
Assignee: Andrew Kyle Purtell
  Resolution: Duplicate

Making this a duplicate of [~pankajkumar] issue; Pankaj did the work of this 
issues subject matter over in HBASE-25918 .

This is a re-resolve because we reopened the issue to discuss whether or not 
latest hbase-thirdparty appropriate in branch-2.4 and conclude it fine. 
Re-resolving.

> Upgrade to latest hbase-thirdparty
> --
>
> Key: HBASE-25729
> URL: https://issues.apache.org/jira/browse/HBASE-25729
> Project: HBase
>  Issue Type: Sub-task
>  Components: build, thirdparty
>Affects Versions: 2.4.2
>Reporter: Andrew Kyle Purtell
>Assignee: Andrew Kyle Purtell
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.5
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25960) Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25960.
---
Fix Version/s: thirdparty-4.0.0
   thirdparty-3.5.2
 Assignee: Michael Stack
   Resolution: Cannot Reproduce

Resolving this issue as "Cannot Reproduce" (could be "Implemented"). Whatever 
was up w/ thirdparty in older versions no longer seems to happen. I tried to 
add enforcer rules but no amenable plugin to check a file in a jar. I could add 
a bit of ant hackery to the thirdparty netty build to set properties if file 
present and fail bulid if so but it'd be ugly. Closing this out.

> Build includes unshaded netty .so; clashes w/ downstreamers who would use a 
> different version of netty
> --
>
> Key: HBASE-25960
> URL: https://issues.apache.org/jira/browse/HBASE-25960
> Project: HBase
>  Issue Type: Bug
>  Components: build
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: thirdparty-3.5.2, thirdparty-4.0.0
>
>
> A coworker was trying to use hbase client in a fat application that uses a 
> different netty version to what hbase uses internally. Their app would fail 
> to launch because it kept bumping into an incompatible netty .so lib. Here 
> are the unshaded netty .so's we bundle looking at hbase-2.4.1...:
> ./lib/hbase-shaded-netty-3.4.1.jar has:
> {code}
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> META-INF/native/libnetty_transport_native_epoll_x86_64.so
> {code}
> (HBASE-25959 should fix the non-relocation of 
> libnetty_transport_native_epoll_aarch_64).
> ./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the 
> same three .sos as does 
> ./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
> and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar
> We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the 
> netty .sos in it.
> Looking at published builds of hbase-thirdparty, I see that these too include 
> the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1 
> https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/
>  as does 3.5.0.
> I just tried running a build of hbase-thirdparty and it does NOT include the 
> extras
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
> META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
> (it has the fix for aarch included... when I built)
> Here is link to the snapshot I made:
> https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26042) WAL lockup on 'sync failed'

[Michael Stack (Jira)]

Michael Stack created HBASE-26042:
-

 Summary: WAL lockup on 'sync failed'
 Key: HBASE-26042
 URL: https://issues.apache.org/jira/browse/HBASE-26042
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.3.5
Reporter: Michael Stack


Making note of issue seen in production cluster.

Node had been struggling under load for a few days with slow syncs up to 10 
seconds, a few STUCK MVCCs from which it recovered and some java pauses up to 
three seconds in length.

Then the below happened:
{code:java}
2021-06-27 13:41:27,604 WARN  [AsyncFSWAL-0-hdfs://:8020/hbase] 
wal.AsyncFSWAL: sync 
failedorg.apache.hbase.thirdparty.io.netty.channel.unix.Errors$NativeIoException:
 readAddress(..) failed: Connection reset by peer {code}
... and WAL turned dead in the water. Scanners start expiring. RPC prints text 
versions of requests complaining requestsTooSlow. Then we start to see these:
{code:java}
org.apache.hadoop.hbase.exceptions.TimeoutIOException: Failed to get sync 
result after 30 ms for txid=552128301, WAL system stuck? {code}
Whats supposed to happen when other side goes away like this is that we will 
roll the WAL – go set up a new one. You can see it happening if you run
{code:java}
mvn test 
-Dtest=org.apache.hadoop.hbase.regionserver.wal.TestAsyncFSWAL#testBrokenWriter 
{code}
I tried hacking the test to repro the above hang by throwing same exception in 
above test (on linux because need epoll to repro) but all just worked.

Thread dumps of the hungup WAL subsystem are a little odd. The log roller is 
stuck w/o timeout trying to write a long on the WAL header:

 
{code:java}
Thread 9464: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may 
be imprecise)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, 
line=175 (Compiled frame)
 - java.util.concurrent.CompletableFuture$Signaller.block() @bci=19, line=1707 
(Compiled frame)
 - 
java.util.concurrent.ForkJoinPool.managedBlock(java.util.concurrent.ForkJoinPool$ManagedBlocker)
 @bci=119, line=3323 (Compiled frame)
 - java.util.concurrent.CompletableFuture.waitingGet(boolean) @bci=115, 
line=1742 (Compiled frame)
 - java.util.concurrent.CompletableFuture.get() @bci=11, line=1908 (Compiled 
frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.write(java.util.function.Consumer)
 @bci=16, line=189 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.writeMagicAndWALHeader(byte[],
 org.apache.hadoop.hbase.shaded.protobuf.generated.WALProtos$WALHeader) @bci=9, 
line=202 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(org.apache.hadoop.fs.FileSystem,
 org.apache.hadoop.fs.Path, org.apache.hadoop.conf.Configuration, boolean, 
long) @bci=107, line=170 (Compiled frame)
 - 
org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(org.apache.hadoop.conf.Configuration,
 org.apache.hadoop.fs.FileSystem, org.apache.hadoop.fs.Path, boolean, long, 
org.apache.hbase.thirdparty.io.netty.channel.EventLoopGroup, java.lang.Class) 
@bci=61, line=113 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
 @bci=22, line=651 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(org.apache.hadoop.fs.Path)
 @bci=2, line=128 (Compiled frame)
 - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(boolean) 
@bci=101, line=797 (Compiled frame)
 - org.apache.hadoop.hbase.wal.AbstractWALRoller$RollController.rollWal(long) 
@bci=18, line=263 (Compiled frame)
 - org.apache.hadoop.hbase.wal.AbstractWALRoller.run() @bci=198, line=179 
(Compiled frame) {code}
 

Other threads are BLOCKED trying to append the WAL w/ flush markers etc. unable 
to add the ringbuffer:

 
{code:java}
Thread 9465: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may 
be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(long) @bci=11, line=338 
(Compiled frame)
 - com.lmax.disruptor.MultiProducerSequencer.next(int) @bci=82, line=136 
(Compiled frame)
 - com.lmax.disruptor.MultiProducerSequencer.next() @bci=2, line=105 
(Interpreted frame)
 - com.lmax.disruptor.RingBuffer.next() @bci=4, line=263 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.lambda$stampSequenceIdAndPublishToRingBuffer$1(org.apache.commons.lang3.mutable.MutableLong,
 com.lmax.disruptor.RingBuffer) @bci=2, line=1031 (Compiled frame)
 - org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL$$Lambda$270.run() 
@bci=8 (Compiled frame)
 - 
org.apache.hadoop.hbase.regionserver.MultiVersionConcurrencyControl.begin(java.lang.Runnable)
 @bci=36, line=140 (Interpreted frame

[jira] [Resolved] (HBASE-25990) Add donated buildbots for jenkins

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25990.
---
Resolution: Incomplete

Closing. I was unable to hook up billing. Might be back but this present effort 
is being aborted.

> Add donated buildbots for jenkins
> -
>
> Key: HBASE-25990
> URL: https://issues.apache.org/jira/browse/HBASE-25990
> Project: HBase
>  Issue Type: Task
>  Components: build
>    Reporter: Michael Stack
>Priority: Major
> Attachments: Screen Shot 2021-06-22 at 1.43.12 PM.png
>
>
> This issue is for keeping notes on how to add a donated buildbot to our 
> apache build.
> My employer donated budget (I badly under-estimated cost but whatever...). 
> This issue is about adding 5 GCP nodes.
> There is this page up on apache on donating machines for build 
> https://infra.apache.org/hosting-external-agent.html It got me some of the 
> ways... at least as far as the bit about mailing root@a.o(nada).
> At [~zhangduo]'s encouragement -- he has been this route already adding in 
> the xiaomi donation -- I filed a JIRA after deploying a machine on GCP, 
> INFRA-21973.
> I then reached out on slack and the gentleman Gavin MacDonald picked up the 
> task.
> He told me run this script on all hosts after making edits (comment out line 
> #39 where we set hostname -- doesn't work):
> https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh
> (For more context on the above script and as a good backgrounder, see the 
> nice C* page on how to do this setup: 
> https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md)
> After doing the above, I had to do a visudo on each host to add a line for an 
> infra account to allow passwordless access.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25989) FanOutOneBlockAsyncDFSOutput using shaded protobuf in hdfs 3.3+

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25989.
---
Fix Version/s: 2.4.5
   2.5.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
 Assignee: Michael Stack
   Resolution: Fixed

Pushed to branch-2.4+ Thanks for reviews [~zhangduo] and [~weichiu]

> FanOutOneBlockAsyncDFSOutput using shaded protobuf in hdfs 3.3+
> ---
>
> Key: HBASE-25989
> URL: https://issues.apache.org/jira/browse/HBASE-25989
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> The parent added some fancy dancing to make it so on hadoop-3.3.0+ we'd use 
> hadoops shaded protobuf rather than the non-relocated protobuf. When hdfs 
> 3.3, the 'trick' is not working so we continue to use the unshaded protobuf. 
> Fix is trivial.
> Found this testing the 3.3.1RC3. Hard to see because whether we use shaded or 
> unshaded is at DEBUG level.  If you set DEBUG level and run 
> TestFanOutOneBlockAsyncDFSOutput with hdfs 3.3.1 RC candidate in place you'll 
> see it uses the unshaded protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25920) Support Hadoop 3.3.1

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25920.
---
Hadoop Flags: Reviewed
Release Note: Fixes to make unit tests pass and to make it so an hbase 
built from branch-2 against a 3.3.1RC can run on a 3.3.1RC small cluster.
  Resolution: Fixed

Resolving as done at least for now.

> Support Hadoop 3.3.1
> 
>
> Key: HBASE-25920
> URL: https://issues.apache.org/jira/browse/HBASE-25920
> Project: HBase
>  Issue Type: Task
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> The Hadoop 3.3.1 is a big release, quite different from 3.3.0.
> File this jira to track the support for Hadoop 3.3.1.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25971) FanOutOneBlockAsyncDFSOutputHelper stuck when run against hadoop-3.3.1-RC3

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25971.
---
Resolution: Not A Problem

Resolving as 'Not a problem'. This seems to have been some issue around 
building artifacts for RC testing. Subsequent builds worked (after HBASE-25989)

> FanOutOneBlockAsyncDFSOutputHelper stuck when run against hadoop-3.3.1-RC3
> --
>
> Key: HBASE-25971
> URL: https://issues.apache.org/jira/browse/HBASE-25971
> Project: HBase
>  Issue Type: Bug
>    Reporter: Michael Stack
>Priority: Major
>
> This is in the log:
> {code}
> 2021-06-04 21:29:39,138 DEBUG [master/oss-master-1:16000:becomeActiveMaster] 
> ipc.ProtobufRpcEngine: Call: addBlock took 6ms
> 2021-06-04 21:29:39,169 WARN  [RS-EventLoopGroup-1-1] 
> concurrent.DefaultPromise: An exception was thrown by 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete()
> java.lang.IllegalArgumentException: object is not an instance of declaring 
> class
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at 
> org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:69)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:343)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:425)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:419)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:477)
> at 
> org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:472)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
> at 
> org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
> at 
> org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
> at 
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653)
> at 
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691)
> at 
> org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
> 

[jira] [Created] (HBASE-25990) Add donated buildbots for jenkins

[Michael Stack (Jira)]

Michael Stack created HBASE-25990:
-

 Summary: Add donated buildbots for jenkins
 Key: HBASE-25990
 URL: https://issues.apache.org/jira/browse/HBASE-25990
 Project: HBase
  Issue Type: Task
  Components: build
Reporter: Michael Stack


This issue is for keeping notes on how to add a donated buildbot to our apache 
build.

My employer donated budget (I badly under-estimated cost but whatever...). This 
issue is about adding 5 GCP nodes.

There is this page up on apache on donating machines for build 
https://infra.apache.org/hosting-external-agent.html It got me some of the 
ways... at least as far as the bit about mailing root@a.o(nada).

At [~zhangduo]'s encouragement -- he has been this route already adding in the 
xiaomi donation -- I filed a JIRA after deploying a machine on GCP, INFRA-21973.

I then reached out on slack and the gentleman Gavin MacDonald picked up the 
task.

He told me run this script on all hosts after making edits (comment out line 
#39 where we set hostname -- doesn't work):

https://github.com/apache/cassandra-builds/blob/trunk/jenkins-dsl/agent-install.sh

(For more context on the above script and as a good backgrounder, see the nice 
C* page on how to do this setup: 
https://github.com/apache/cassandra-builds/blob/trunk/ASF-jenkins-agents.md)

After doing the above, I had to do a visudo on each host to add a line for an 
infra account to allow passwordless access.












--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25989) FanOutOneBlockAsyncDFSOutput using shaded protobuf in hdfs 3.3+

[Michael Stack (Jira)]

Michael Stack created HBASE-25989:
-

 Summary: FanOutOneBlockAsyncDFSOutput using shaded protobuf in 
hdfs 3.3+
 Key: HBASE-25989
 URL: https://issues.apache.org/jira/browse/HBASE-25989
 Project: HBase
  Issue Type: Sub-task
Reporter: Michael Stack


The parent added some fancy dancing to make it so on hadoop-3.3.0+ we'd use 
hadoops shaded protobuf rather than the non-relocated protobuf. When hdfs 3.3, 
the 'trick' is not working so we continue to use the unshaded protobuf. Fix is 
trivial.

Found this testing the 3.3.1RC3. Hard to see because whether we use shaded or 
unshaded is at DEBUG level.  If you set DEBUG level and run 
TestFanOutOneBlockAsyncDFSOutput with hdfs 3.3.1 RC candidate in place you'll 
see it uses the unshaded protobuf.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25969) Cleanup netty-all transitive includes

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25969.
---
Fix Version/s: 2.4.5
   2.5.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
 Release Note: We have an (old) netty-all in our produced artifacts. It is 
transitively included from hadoop. It is needed by MiniMRCluster referenced 
from a few MR tests in hbase. This commit adds netty-all excludes everywhere 
else but where tests will fail unless the transitive is allowed through. TODO: 
move MR and/or MR tests out of hbase core.
 Assignee: Michael Stack
   Resolution: Fixed

Thanks for reviews [~haxiaolin] and [~pankajkumar] Pushed to branch-2.4+

> Cleanup netty-all transitive includes
> -
>
> Key: HBASE-25969
> URL: https://issues.apache.org/jira/browse/HBASE-25969
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.5
>
>
> Our releases include lib/netty-all.jar as a transitive include from hadoop. 
> -Purge.-
> ... looks like I can't purge the transitive netty-all includes just yet, not 
> w/o moving MR out of hbase core. The transitively included netty-all w/ the 
> old version is needed to run the tests that put up a MR cluster.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Update images in old blog posts

[Stack]

On Mon, Jun 7, 2021 at 9:50 AM Peter Somogyi  wrote:

> Thanks Stack! I've managed to update the post.
> I have to admit the blog tool is not that user friendly...
>
>
It is a bit rough (my favorite was silently not taking the update if post
was 'too long...')
S



> Peter
>
> On Fri, Jun 4, 2021 at 5:34 PM Stack  wrote:
>
> > On Fri, Jun 4, 2021 at 2:54 AM Peter Somogyi 
> wrote:
> >
> > > Tried this track now but I was told it is self-serve by the project.
> Can
> > > you give me permission if you are an admin on HBase blogs?
> > >
> > >
> > Done.
> > S
> >
> >
> >
> >
> > > Thanks,
> > > Peter
> > >
> > > On Thu, Jun 3, 2021 at 5:18 PM Stack  wrote:
> > >
> > > > I can help Peter.
> > > >
> > > > If you want to make your own login for the blog, its an infra ticket
> > [1].
> > > >
> > > > Thanks for doing the fix-up.
> > > > S
> > > >
> > > > 1.
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-4327?jql=project%20%3D%20INFRA%20AND%20text%20~%20%22blog%20account%22
> > > >
> > > > On Thu, Jun 3, 2021 at 6:28 AM Peter Somogyi 
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I was asked about an HBase blog post [1] at $dayjob if we could
> > update
> > > it
> > > > > because the images are not available anymore.
> > > > > Do we have the possibility of updating old posts? I've never
> created
> > a
> > > > blog
> > > > > post in blogs.apache.org so probably I don't have the right
> > > permissions.
> > > > >
> > > > > Luckily, the images can be found in WaybackMachine [2].
> > > > >
> > > > > Can someone help me with some guidance on it or help me updating
> this
> > > > post?
> > > > >
> > > > > Thanks,
> > > > > Peter
> > > > >
> > > > > [1]
> > > > >
> > >
> https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
> > > > > [2]
> > > > >
> > > > >
> > > >
> > >
> >
> https://web.archive.org/web/20161022173657/https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
> > > > >
> > > >
> > >
> >
>

[jira] [Resolved] (HBASE-18562) [AMv2] expireServers and ServerCrashProcedure cleanup

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-18562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-18562.
---
Resolution: Not A Problem

Agree.

Resolving as 'Not a problem' since so much has changed since.

> [AMv2] expireServers and ServerCrashProcedure cleanup
> -
>
> Key: HBASE-18562
> URL: https://issues.apache.org/jira/browse/HBASE-18562
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>    Reporter: Michael Stack
>Priority: Critical
>
> In review of HBASE-18551, [~uagashe] posed a scenario that revealed a hole in 
> our processing of unassigns; there is case where a UP might not get 
> notification from ServerCrashProcedure if the UP is scheduled AFTER a SCP has 
> gotten past its handleRIT call (No new SCP will be queued because 
> expireServer won't let it happen if crashed server is in dead server list 
> which it will be).
> Chatting on it, expireServers is doing checks that belong inside 
> ServerCrashProcedure. expireServers scheduling an SCP each time it is called 
> would make it so SCP processing is serialized one behind the other. If the 
> first does the clean up all subsequent will do no work but Procedures 
> dependent on them will get their wakeup call.
> This issue is about implementing the above cleanup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-20179) [TESTING] Scale for 2.0.0

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-20179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-20179.
---
Fix Version/s: (was: 3.0.0-alpha-1)
   2.0.0
 Assignee: Michael Stack
   Resolution: Fixed

Finished task. Set fix version as 2.0.0 though it released long ago.

> [TESTING] Scale for 2.0.0
> -
>
> Key: HBASE-20179
> URL: https://issues.apache.org/jira/browse/HBASE-20179
> Project: HBase
>  Issue Type: Umbrella
>  Components: test
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Critical
> Fix For: 2.0.0
>
>
> Umbrella issue for scale testing for 2.0.
> At least keep account of what testing has been done in here.
> TODO as subtasks
>  * Long-running test
>  * ITBLL w/o killing Master
>  * Big ITBLL (1B works, 10B needs more verification, 100B, and 1T).
>  * Many Regions (43k regions takes about 12 minutes on a 6node cluster to 
> deploy and 3 1/2 minutes to go down -- needs tuning).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25971) FanOutOneBlockAsyncDFSOutputHelper stuck when run against hadoop-3.3.1-RC3

[Michael Stack (Jira)]

Michael Stack created HBASE-25971:
-

 Summary: FanOutOneBlockAsyncDFSOutputHelper stuck when run against 
hadoop-3.3.1-RC3
 Key: HBASE-25971
 URL: https://issues.apache.org/jira/browse/HBASE-25971
 Project: HBase
  Issue Type: Bug
Reporter: Michael Stack


This is in the log:

{code}
2021-06-04 21:29:39,138 DEBUG [master/oss-master-1:16000:becomeActiveMaster] 
ipc.ProtobufRpcEngine: Call: addBlock took 6ms
2021-06-04 21:29:39,169 WARN  [RS-EventLoopGroup-1-1] 
concurrent.DefaultPromise: An exception was thrown by 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete()
java.lang.IllegalArgumentException: object is not an instance of declaring class
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:69)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:343)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:425)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:419)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:477)
at 
org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:472)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at 
org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84)
at 
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653)
at 
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691)
at 
org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567)
at 
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:470)
at 
org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at 
org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at 
org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}

These are WARNs but Master startup is stuck here:

{code}
"master/oss-master-1:16000:becomeActiveMaster" #88 daemon prio=5 os_prio=0 
cp

Re: Update images in old blog posts

[Stack]

On Fri, Jun 4, 2021 at 2:54 AM Peter Somogyi  wrote:

> Tried this track now but I was told it is self-serve by the project. Can
> you give me permission if you are an admin on HBase blogs?
>
>
Done.
S




> Thanks,
> Peter
>
> On Thu, Jun 3, 2021 at 5:18 PM Stack  wrote:
>
> > I can help Peter.
> >
> > If you want to make your own login for the blog, its an infra ticket [1].
> >
> > Thanks for doing the fix-up.
> > S
> >
> > 1.
> >
> >
> https://issues.apache.org/jira/browse/INFRA-4327?jql=project%20%3D%20INFRA%20AND%20text%20~%20%22blog%20account%22
> >
> > On Thu, Jun 3, 2021 at 6:28 AM Peter Somogyi 
> wrote:
> >
> > > Hi,
> > >
> > > I was asked about an HBase blog post [1] at $dayjob if we could update
> it
> > > because the images are not available anymore.
> > > Do we have the possibility of updating old posts? I've never created a
> > blog
> > > post in blogs.apache.org so probably I don't have the right
> permissions.
> > >
> > > Luckily, the images can be found in WaybackMachine [2].
> > >
> > > Can someone help me with some guidance on it or help me updating this
> > post?
> > >
> > > Thanks,
> > > Peter
> > >
> > > [1]
> > >
> https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
> > > [2]
> > >
> > >
> >
> https://web.archive.org/web/20161022173657/https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
> > >
> >
>

[jira] [Created] (HBASE-25969) Purge netty-all transitive includes

[Michael Stack (Jira)]

Michael Stack created HBASE-25969:
-

 Summary: Purge netty-all transitive includes
 Key: HBASE-25969
 URL: https://issues.apache.org/jira/browse/HBASE-25969
 Project: HBase
  Issue Type: Sub-task
Reporter: Michael Stack


Our releases include lib/netty-all.jar as a transitive include from hadoop. 
Purge.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Update images in old blog posts

[Stack]

I can help Peter.

If you want to make your own login for the blog, its an infra ticket [1].

Thanks for doing the fix-up.
S

1.
https://issues.apache.org/jira/browse/INFRA-4327?jql=project%20%3D%20INFRA%20AND%20text%20~%20%22blog%20account%22

On Thu, Jun 3, 2021 at 6:28 AM Peter Somogyi  wrote:

> Hi,
>
> I was asked about an HBase blog post [1] at $dayjob if we could update it
> because the images are not available anymore.
> Do we have the possibility of updating old posts? I've never created a blog
> post in blogs.apache.org so probably I don't have the right permissions.
>
> Luckily, the images can be found in WaybackMachine [2].
>
> Can someone help me with some guidance on it or help me updating this post?
>
> Thanks,
> Peter
>
> [1]
> https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
> [2]
>
> https://web.archive.org/web/20161022173657/https://blogs.apache.org/hbase/entry/apache_hbase_internals_locking_and
>

Re: [VOTE] First release candidate for hbase-thirdparty 3.5.1 is available for download

[Stack]

+1

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_292): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_292): ok
 - mvn clean install  -DskipTests
* Unit tests pass (1.8.0_292): ok
 - mvn package -P runAllTests  -Dsurefire.rerunFailingTestsCount=3

CHANGES and RELEASENOTES look good. Successfully built using this version
of thirdparty.

S

On Wed, Jun 2, 2021 at 4:25 AM 张铎(Duo Zhang)  wrote:

> Please vote on this Apache hbase thirdparty release candidate,
> hbase-thirdparty-3.5.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase thirdparty 3.5.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 3.5.1RC0:
>
>   https://github.com/apache/hbase-thirdparty/tree/3.5.1RC0
>
> This tag currently points to git reference
>
>   c28a235236b9f63ec1d36431e5d1b6c8d4b66d90
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-3.5.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1453/
>
> Artifacts were signed with the 9AD2AE49 key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> To learn more about Apache hbase thirdparty, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Resolved] (HBASE-19515) Region server left in online servers list forever if it went down after registering to master and before creating ephemeral node

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-19515.
---
Resolution: Not A Problem

Resolving as 'Not a Problem', fixed by HBASE-25032. Thanks [~anoop.hbase] for 
taking a look.

> Region server left in online servers list forever if it went down after 
> registering to master and before creating ephemeral node
> 
>
> Key: HBASE-19515
> URL: https://issues.apache.org/jira/browse/HBASE-19515
> Project: HBase
>  Issue Type: Bug
>  Components: Region Assignment
>    Reporter: Michael Stack
>Priority: Critical
> Fix For: 3.0.0-alpha-2
>
>
> This one is interesting. It was supposedly fixed long time ago back in 
> HBASE-9593 (The issue has same subject as this one) but there was a problem 
> w/ the fix reported later, post-commit, long after the issue was closed. The 
> 'fix' was registering ephemeral node in ZK BEFORE reporting in to the Master 
> for the first time. The problem w/ this approach is that the Master tells the 
> RS what name it should use reporting in. If we register in ZK before we talk 
> to the Master, the name in ZK and the one the RS ends up using could deviate.
> In hbase2, we do the right thing registering the ephemeral node after we 
> report to the Master. So, the issue reported in HBASE-9593, that a RS that 
> dies between reporting to master and registering up in ZK, stays registered 
> at the Master for ever is back; we'll keep trying to assign it regions. Its a 
> real problem.
> That hbase2 has this issue has been suppressed up until now. The test that 
> was written for HBASE-9593, TestRSKilledWhenInitializing, is a good test but 
> a little sloppy. It puts up two RSs aborting one only after registering at 
> the Master before posting to ZK. That leaves one healthy server up. It is 
> hosting hbase:meta. This is enough for the test to bluster through. The only 
> assign it does is namespace table. It goes to the hbase:meta server. If the 
> test created a new table and did roundrobin, it'd fail.
> After HBASE-18946, where we do round robin on table create -- a desirable 
> attribute -- via the balancer so all is kosher, the test 
> TestRSKilledWhenInitializing now starts to fail because we chose the hobbled 
> server most of the time.
> So, this issue is about fixing the original issue properly for hbase2. We 
> don't have a timeout on assign in AMv2, not yet, that might be the fix, or 
> perhaps a double report before we online a server with the second report 
> coming in after ZK goes up (or we stop doing ephemeral nodes for RS up in ZK 
> and just rely on heartbeats).
> Making this a critical issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25960) Build includes unshaded netty .so; clashes w/ downstreamers who would use a different version of netty

[Michael Stack (Jira)]

Michael Stack created HBASE-25960:
-

 Summary: Build includes unshaded netty .so; clashes w/ 
downstreamers who would use a different version of netty
 Key: HBASE-25960
 URL: https://issues.apache.org/jira/browse/HBASE-25960
 Project: HBase
  Issue Type: Bug
  Components: build
Reporter: Michael Stack


A coworker was trying to use hbase client in a fat application that uses a 
different netty version to what hbase uses internally. Their app would fail to 
launch because it kept bumping into an incompatible netty .so lib. Here are the 
unshaded netty .so's we bundle looking at hbase-2.4.1...:

./lib/hbase-shaded-netty-3.4.1.jar has:

{code}
META-INF/native/libnetty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so
META-INF/native/libnetty_transport_native_epoll_x86_64.so
{code}

(HBASE-25959 should fix the non-relocation of 
libnetty_transport_native_epoll_aarch_64).

./lib/shaded-clients/hbase-shaded-client-byo-hadoop-2.4.1.1-apple.jar has the 
same three .sos as does 
./lib/shaded-clients/hbase-shaded-mapreduce-2.4.1.1-apple.jar
and ./lib/shaded-clients/hbase-shaded-client-2.4.1.1-apple.jar

We even bundle ./lib/netty-all-4.1.17.Final.jar which unsurprisingly has the 
netty .sos in it.

Looking at published builds of hbase-thirdparty, I see that these too include 
the above trio of .sos... The hbase-shaded-netty includes them in 3.4.1 
https://repo1.maven.org/maven2/org/apache/hbase/thirdparty/hbase-shaded-netty/3.4.1/
 as does 3.5.0.

I just tried running a build of hbase-thirdparty and it does NOT include the 
extras

META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_aarch_64.so
META-INF/native/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_64.so

(it has the fix for aarch included... when I built)

Here is link to the snapshot I made:

https://repository.apache.org/content/repositories/orgapachehbase-1451/org/apache/hbase/thirdparty/hbase-shaded-netty/3.5.1-stack4/






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25959.
---
Fix Version/s: hbase-thirdparty-3.5.1
 Assignee: Michael Stack
   Resolution: Fixed

Pushed the one-liner.

> Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty
> 
>
> Key: HBASE-25959
> URL: https://issues.apache.org/jira/browse/HBASE-25959
> Project: HBase
>  Issue Type: Bug
>  Components: hbase-thirdparty
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Minor
> Fix For: hbase-thirdparty-3.5.1
>
> Attachments: 
> 0001-HBASE-25959-Relocate-libnetty_transport_native_epoll.patch
>
>
> Minor item I came across while trying to figure where all the netty 
> native_epoll .so instances are coming from when I look at an hbase release.  
> We've relocated the x86 lib but not the aarch_64... Minor item.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25959) Relocate libnetty_transport_native_epoll_aarch_64.so in hbase-thirdparty

[Michael Stack (Jira)]

Michael Stack created HBASE-25959:
-

 Summary: Relocate libnetty_transport_native_epoll_aarch_64.so in 
hbase-thirdparty
 Key: HBASE-25959
 URL: https://issues.apache.org/jira/browse/HBASE-25959
 Project: HBase
  Issue Type: Bug
  Components: hbase-thirdparty
Reporter: Michael Stack


Minor item I came across while trying to figure where all the netty 
native_epoll .so instances are coming from when I look at an hbase release.  
We've relocated the x86 lib but not the aarch_64... Minor item.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-19701) Close without justification following succesful open

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-19701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-19701.
---
Resolution: Cannot Reproduce

Closing as 'Cannot Reporduce' ...  haven't seen it since original filing... May 
still be there but no work done on this item... Will open new one if seen again.

> Close without justification following succesful open
> 
>
> Key: HBASE-19701
> URL: https://issues.apache.org/jira/browse/HBASE-19701
> Project: HBase
>  Issue Type: Bug
>    Reporter: Michael Stack
>Priority: Critical
> Fix For: 3.0.0-alpha-2
>
>
> [~jmspaggi] conjured an interesting condition where we close a region soon 
> after open WITHOUT seemingly saying why (It looks like Master is asking for 
> region CLOSE but that is not clear looking at RegionServer log).
> Here is log snippet from https://pastebin.com/0r76Y6ap (in case the pastebin 
> evaporates)
> {code}
> 
> 2017-12-31 09:54:20,864 INFO  
> [PostOpenDeployTasks:f49f3cbb7f3db4cf96c7eb3b0cf83869] 
> regionserver.HRegionServer: Post open deploy tasks for 
> TestTable,0408944640,1505391191559.f49f3cbb7f3db4cf96c7eb3b0cf83869.
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,870 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:20,872 INFO  
> [StoreOpener-330f09f4a0eaf26811c320fbf1b14e70-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00; 
> throttle point 2684354560; major period 60480, major jitter 0,50, min 
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,903 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] 
> regionserver.CompactingMemStore: Setting in-memory flush size threshold to 
> 13421772 and immutable segments index to be of type CHUNK_MAP
> 2017-12-31 09:54:20,904 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] regionserver.HStore: 
> Memstore class name is org.apache.hadoop.hbase.regionserver.CompactingMemStore
> 2017-12-31 09:54:20,904 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] hfile.CacheConfig: Created 
> cacheConfig for info: blockCache=LruBlockCache{blockCount=0, 
> currentSize=2454760, freeSize=3347745560, maxSize=3350200320, 
> heapSize=2454760, minSize=3182690304, minFactor=0.95, multiSize=1591345152, 
> multiFactor=0.5, singleSize=795672576, singleFactor=0.25}, 
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, 
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, 
> prefetchOnOpen=false
> 2017-12-31 09:54:20,905 INFO  
> [StoreOpener-166b9c45d7724f72fd126adb4445d6ec-1] 
> compactions.CompactionConfiguration: size [134217728, 9223372036854775807, 
> 9223372036854775807); files [3, 10); ratio 1,20; off-peak ratio 5,00; 
> throttle point 2684354560; major period 60480, major jitter 0,50, min 
> locality to compact 0,00; tiered compaction: max_age 9223372036854775807, 
> incoming window min 6, compaction policy for tiered window 
> org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, 
> single output for minor true, compaction window factory 
> org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory
> 2017-12-31 09:54:20,929 INFO  [RS_OPEN_REGION-node1:16020-1] 

[jira] [Resolved] (HBASE-25941) TestRESTServerSSL fails because of jdk bug

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25941.
---
Fix Version/s: 2.4.4
   2.3.6
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
 Assignee: Michael Stack
   Resolution: Fixed

Pushed to branch-2.3+. Thanks for the review [~weichiu] (and for finding how to 
fix)

> TestRESTServerSSL fails because of jdk bug
> --
>
> Key: HBASE-25941
> URL: https://issues.apache.org/jira/browse/HBASE-25941
> Project: HBase
>  Issue Type: Sub-task
>  Components: test
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.6, 2.4.4
>
>
> [~weijing329] identified issue in TestRESTServerSSL when using jdk8 292+. It 
> came up in comment in the parent issue. I verified it fails for me using jdk8 
> v292. Here is the failure
> ```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s 
> <<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
> [ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL  Time elapsed: 1.387 s 
>  <<< ERROR!
> java.security.NoSuchAlgorithmException: unrecognized algorithm name: 
> PBEWithSHA1AndDESede
>   at 
> org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```
> For workaround, see https://github.com/bcgit/bc-java/issues/941



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25940) Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25940.
---
Hadoop Flags: Reviewed
Assignee: Michael Stack
  Resolution: Fixed

Merged to branch-2.4+. Thanks for review [~zhangduo]

> Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO
> 
>
> Key: HBASE-25940
> URL: https://issues.apache.org/jira/browse/HBASE-25940
> Project: HBase
>  Issue Type: Sub-task
>    Reporter: Michael Stack
>    Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.4
>
>
> LZ4 was changed in hadoop-3.3.1 to use a more amenable library, one that did 
> not require gymnastics installing lz4 native libs everywhere; rather, you 
> just add a jar to the classpath. See HADOOP-17292.
> Similar was done for SNAPPY. See HADOOP-17125.
> What this means is that our TestCompressionTest passes for hadoop before 
> 3.3.1 but at 3.3.1, the assert that SNAPPY and LZ4 compressors should fail -- 
> because no lib installed -- now no longer asserts.
> While in here, LZO is GPL and requires extra install to setup [1]. When 
> TestCompressionTest runs, it emits the below for the LZO check. The check is 
> kinda useless.
> {code}
> 2021-05-28T10:05:36,513 WARN  [Time-limited test] util.CompressionTest(75): 
> Can't instantiate codec: lzo
> org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'lzo' 
> previously failed test.
> {code}
> I think best thing for now is to comment out the asserts that LZ4 and SNAPPY 
> do NOT work when binary cannot be found -- since this holds only if hadoop < 
> 3.3..1; the test is a little weak anyways.
> 1. 
> https://stackoverflow.com/questions/23441142/class-com-hadoop-compression-lzo-lzocodec-not-found-for-spark-on-cdh-5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25941) TestRESTServerSSL fails because of jdk bug

[Michael Stack (Jira)]

Michael Stack created HBASE-25941:
-

 Summary: TestRESTServerSSL fails because of jdk bug
 Key: HBASE-25941
 URL: https://issues.apache.org/jira/browse/HBASE-25941
 Project: HBase
  Issue Type: Sub-task
  Components: test
Reporter: Michael Stack


[~weijing329] identified issue in TestRESTServerSSL when using jdk8 252+. It 
came up in comment in the parent issue. I verified it fails for me using jdk8 
v292. Here is the failure

```[INFO] Running org.apache.hadoop.hbase.rest.TestRESTServerSSL
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.4 s 
<<< FAILURE! - in org.apache.hadoop.hbase.rest.TestRESTServerSSL
[ERROR] org.apache.hadoop.hbase.rest.TestRESTServerSSL  Time elapsed: 1.387 s  
<<< ERROR!
java.security.NoSuchAlgorithmException: unrecognized algorithm name: 
PBEWithSHA1AndDESede
at 
org.apache.hadoop.hbase.rest.TestRESTServerSSL.beforeClass(TestRESTServerSSL.java:74)```

For workaround, see https://github.com/bcgit/bc-java/issues/941



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25940) Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO

[Michael Stack (Jira)]

Michael Stack created HBASE-25940:
-

 Summary: Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO
 Key: HBASE-25940
 URL: https://issues.apache.org/jira/browse/HBASE-25940
 Project: HBase
  Issue Type: Sub-task
Reporter: Michael Stack


LZ4 was changed in hadoop-3.3.1 to use a more amenable library, one that did 
not require gymnastics installing lz4 native libs everywhere; rather, you just 
add a jar to the classpath. See HADOOP-17292.

Similar was done for SNAPPY. See HADOOP-17125.

What this means is that our TestCompressionTest passes for hadoop before 3.3.1 
but at 3.3.1, the assert that SNAPPY and LZ4 compressors should fail -- because 
no lib installed -- now no longer asserts.

While in here, LZO is GPL and requires extra install to setup [1]. When 
TestCompressionTest runs, it emits the below for the LZO check. The check is 
kinda useless.

{code}
2021-05-28T10:05:36,513 WARN  [Time-limited test] util.CompressionTest(75): 
Can't instantiate codec: lzo
org.apache.hadoop.hbase.DoNotRetryIOException: Compression algorithm 'lzo' 
previously failed test.
{code}

I think best thing for now is to comment out the asserts that LZ4 and SNAPPY do 
NOT work when binary cannot be found -- since this holds only if hadoop < 
3.3..1; the test is a little weak anyways.



1. 
https://stackoverflow.com/questions/23441142/class-com-hadoop-compression-lzo-lzocodec-not-found-for-spark-on-cdh-5



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25861) Correct the usage of Configuration#addDeprecation

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25861.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Re-resolving. Issue addressed over in HBASE-25928

> Correct the usage of Configuration#addDeprecation
> -
>
> Key: HBASE-25861
> URL: https://issues.apache.org/jira/browse/HBASE-25861
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Baiqiang Zhao
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> When I was solving HBASE-25745 
> ([PR3139|https://github.com/apache/hbase/pull/3139]), I found that our use of 
> Configuration#addDeprecation API was wrong. 
>  
> At present, we will call Configuration#addDeprecation in the static block for 
> the deprecated configuration. But after testing, it is found that this does 
> not complete backward compatibility. When user upgrades HBase and does not 
> change the deprecated configuration to the new configuration, he will find 
> that the deprecated configuration does not effect, which may not be 
> consistent with expectations. The specific test results can be seen in the PR 
> above, and we can found the calling order of Configuration#addDeprecation is 
> very important.
>  
> Configuration#addDeprecation is a Hadoop API, looking through the Hadoop 
> source code, we will find that before creating the Configuration object, the 
> addDeprecatedKeys() method will be called first: 
> [https://github.com/apache/hadoop/blob/b93e448f9aa66689f1ce5059f6cdce8add130457/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java#L34]
>  .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25928) TestHBaseConfiguration#testDeprecatedConfigurations is broken with Hadoop 3.3

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25928.
---
Fix Version/s: 2.5.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2 and master. Thanks for finding the issue [~weichiu] and 
thanks for the fix [~DeanZ]

> TestHBaseConfiguration#testDeprecatedConfigurations is broken with Hadoop 3.3
> -
>
> Key: HBASE-25928
> URL: https://issues.apache.org/jira/browse/HBASE-25928
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.5.0
>Reporter: Wei-Chiu Chuang
>Assignee: Baiqiang Zhao
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> The test TestHBaseConfiguration#testDeprecatedConfigurations was added 
> recently by HBASE-25861 to address the usage of Hadoop Configuration 
> addDeprecations API.
> However, the API's behavior was changed to fix a bug.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25908) Exclude jakarta.activation-api

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-25908.
---
Fix Version/s: 2.5.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
 Tags: hadoop-3.3.1
   Resolution: Fixed

Merged to master and branch-2. Should I backport to branch-2.4 [~apurtell] ? 
You want to run on hadoop 3.3.1? Otherwise, hbase-2.5 to run on hadoop-3.3.1?

Thanks for the fix and the nice background [~weichiu].

> Exclude jakarta.activation-api
> --
>
> Key: HBASE-25908
> URL: https://issues.apache.org/jira/browse/HBASE-25908
> Project: HBase
>  Issue Type: Improvement
>  Components: hadoop3, shading
>Affects Versions: 2.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Hadoop 3.3.1 replaced its dependency of javax.activation 1.2.0 with 
> jakarta.activation 1.2.1.
> They are essentially the same thing (they even have the same classpath name), 
> but Eclipse took over JavaEE development and therefore changed group/artifact 
> id. 
> (https://stackoverflow.com/questions/46493613/what-is-the-replacement-for-javax-activation-package-in-java-9)
> See HADOOP-17049 for more details. Hadoop 3.3.0 updated jackson-databind to 
> 2.10 which shades jakarta.activation, causing classpath conflict.
> The solution to this issue will be similar to HBASE-22268



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: master branch issues when running with jdk8

[Stack]

On Thu, May 27, 2021 at 7:34 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> ... I guess this was
> not expected, and our official goal is to still support jdk8 for the
> upcoming 3.0 release?
>

I'd suggest dropping jdk8 support in hbase3.
S

Re: [VOTE] Second release candidate for HBase 2.4.3 (RC1) is available

[Stack]

+1

   * Signature: ok
* Checksum : ok
* Rat check (1.8.0_191): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_191): ok
 - mvn clean install -DskipTests
* Unit tests pass (1.8.0_191): failed
 - mvn package -P runSmallTests

The test failed because I had my vpn up which clashes w/ the kerberos test.

Changes and release notes look good (did not verify changes matches issues
in git).
API compat looks good.
Started up standalone with binary built from src. Loaded and verified data
made it. Exercised shell. Looked at UI.

S

On Thu, May 20, 2021 at 12:11 PM Andrew Purtell  wrote:

> Please vote on this Apache HBase release candidate, hbase-2.4.3RC1.
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache HBase 2.4.3
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.4.3RC1:
>
> https://github.com/apache/hbase/tree/2.4.3RC1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
> https://dist.apache.org/repos/dist/dev/hbase/2.4.3RC1/
>
> These sources correspond with the git tag "2.4.3RC1" (401b60b217).
>
> Temporary Maven artifacts are available in the staging repository:
>
>
> https://repository.apache.org/content/repositories/orgapachehbase-1447/
>
> Artifacts were signed with the apurt...@apache.org key which can be found
> in:
>
> https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> The API compatibility report for this RC can be found at:
>
>
>
> https://dist.apache.org/repos/dist/dev/hbase/2.4.3RC1/api_compare_2.4.2_to_2.4.3RC1.html
>
> We performed the following successful pre-flight checks before
> announcing the previous RC, RC0:
>
> - Unit tests
>
> - 10 TB Common Crawl data load via IntegrationTestLoadCommonCrawl,
>   slowDeterministic policy
>
> To learn more about Apache HBase, please see
>
> http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [DISCUSS] Breakout discussion on storefile tracking storage solutions

[Stack]

And, what is there currently is a nice write-up
S

On Wed, May 26, 2021 at 9:26 AM Stack  wrote:

> Can I have comment access please Josh?
> S
>
> On Tue, May 25, 2021 at 8:24 PM Josh Elser  wrote:
>
>> Hi folks,
>>
>> This is a follow-on for the HBASE-24749 discussion on storefile
>> tracking, specifically focusing on where/how do we store the list of
>> files for each Store.
>>
>> I tried to capture my thoughts and the suggestions by Duo and Wellington
>> in this google doc [1].
>>
>> Please feel free to ask for edit permission (and send me a note if your
>> email address isn't one that I would otherwise recognize :) ) to
>> correct, improve, or expand on any other sections.
>>
>> FWIW, I was initially not super excited about a per-Store file, but, the
>> more I think about it, the more I'm coming around to that idea. I think
>> it will be more "exception-handling", but avoid the long-term
>> operational burden of yet-another-important-system-table.
>>
>> - Josh
>>
>> [1]
>>
>> https://docs.google.com/document/d/1yzjvQvQfnT-M8ZgKdcQNedF8HssTnQR2loPkZtlJGVg/edit?usp=sharing
>>
>

Re: [DISCUSS] Breakout discussion on storefile tracking storage solutions

[Stack]

Can I have comment access please Josh?
S

On Tue, May 25, 2021 at 8:24 PM Josh Elser  wrote:

> Hi folks,
>
> This is a follow-on for the HBASE-24749 discussion on storefile
> tracking, specifically focusing on where/how do we store the list of
> files for each Store.
>
> I tried to capture my thoughts and the suggestions by Duo and Wellington
> in this google doc [1].
>
> Please feel free to ask for edit permission (and send me a note if your
> email address isn't one that I would otherwise recognize :) ) to
> correct, improve, or expand on any other sections.
>
> FWIW, I was initially not super excited about a per-Store file, but, the
> more I think about it, the more I'm coming around to that idea. I think
> it will be more "exception-handling", but avoid the long-term
> operational burden of yet-another-important-system-table.
>
> - Josh
>
> [1]
>
> https://docs.google.com/document/d/1yzjvQvQfnT-M8ZgKdcQNedF8HssTnQR2loPkZtlJGVg/edit?usp=sharing
>