Re: [ANNOUNCE] Please welcome Balazs Meszaros to the Apache HBase PMC
Congratulations! Balazs On Sat, Nov 2, 2019 at 9:16 PM Pankaj kr wrote: > > Congratulations Balazs Meszaros..!! > > Regards, > Pankaj > > -Original Message- > From: Sean Busbey [mailto:bus...@apache.org] > Sent: 24 October 2019 20:05 > To: dev ; Hbase-User > Subject: [ANNOUNCE] Please welcome Balazs Meszaros to the Apache HBase PMC > > On behalf of the Apache HBase PMC I am pleased to announce that Balazs > Meszaros has accepted our invitation to become a PMC member on the HBase > project. We appreciate Balazs stepping up to take more responsibility in > the HBase project. > > Please join me in welcoming Balazs to the HBase PMC! > > > > As a reminder, if anyone would like to nominate another person as a > committer or PMC member, even if you are not currently a committer or PMC > member, you can always drop a note to priv...@hbase.apache.org to let us > know. >
Re: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache HBase PMC
Congratulations! Wellington On Sat, Nov 2, 2019 at 9:14 PM Pankaj kr wrote: > Congratulations Wellington..!! > > Regards, > Pankaj > > > -Original Message- > From: Sean Busbey [mailto:bus...@apache.org] > Sent: 24 October 2019 01:47 > To: dev ; Hbase-User > Subject: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache > HBase PMC > > On behalf of the Apache HBase PMC I am pleased to announce that Wellington > Chevreuil has accepted our invitation to become a PMC member on the HBase > project. We appreciate Wellington stepping up to take more responsibility > in the HBase project. > > Please join me in welcoming Wellington to the HBase PMC! > > > > As a reminder, if anyone would like to nominate another person as a > committer or PMC member, even if you are not currently a committer or PMC > member, you can always drop a note to priv...@hbase.apache.org to let us > know. >
Re: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC
Congratulations! Sakthi On Sat, Nov 2, 2019 at 9:13 PM Pankaj kr wrote: > Congratulations Sakthi..!! > > Regards, > Pankaj > > > -Original Message- > From: Sean Busbey [mailto:bus...@apache.org] > Sent: 24 October 2019 01:45 > To: dev ; Hbase-User > Subject: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC > > On behalf of the Apache HBase PMC I am pleased to announce that Sakthi has > accepted our invitation to become a PMC member on the HBase project. We > appreciate Sakthi stepping up to take more responsibility in the HBase > project. > > Please join me in welcoming Jan to the HBase PMC! > > > > As a reminder, if anyone would like to nominate another person as a > committer or PMC member, even if you are not currently a committer or PMC > member, you can always drop a note to priv...@hbase.apache.org to let us > know. >
[jira] [Created] (HBASE-23247) [hbck2] Schedule SCPs for 'Unknown Servers'
Michael Stack created HBASE-23247:
-
Summary: [hbck2] Schedule SCPs for 'Unknown Servers'
Key: HBASE-23247
URL: https://issues.apache.org/jira/browse/HBASE-23247
Project: HBase
Issue Type: Bug
Components: hbck2
Affects Versions: 2.2.2
Reporter: Michael Stack
Assignee: Michael Stack
Fix For: 2.2.3
I've run into an 'Unknown Server' phenomenon; meta has regions assigned to
servers that the cluster no longer knows about. Fix is tough because new assign
is insistent on checking the close succeeded by trying to contact the 'unknown
server' and being insistent on not moving on until it succeeds; TODO. There are
a few ways of obtaining this state of affairs. I'll list a few below in a
minute.
Meantime, an hbck2 'fix' should be just scheduling an SCP using
scheduleRecoveries command only in this case it falis before scheduling SCP
with the below; i.e. a FNFE because no dir for the 'Unknown Server'.
{code}
22:41:13.909 [main] INFO
org.apache.hadoop.hbase.client.ConnectionImplementation - Closing master
protocol: MasterService
Exception in thread "main" java.io.IOException:
org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.FileNotFoundException):
java.io.FileNotFoundException: File
hdfs://nameservice1/hbase/genie/WALs/s1.d.com,16020,1571170081872 does not
exist.
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:986)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:122)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1046)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1043)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1053)
at
org.apache.hadoop.fs.FilterFileSystem.listStatus(FilterFileSystem.java:258)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1802)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1844)
at
org.apache.hadoop.hbase.master.MasterRpcServices.containMetaWals(MasterRpcServices.java:2709)
at
org.apache.hadoop.hbase.master.MasterRpcServices.scheduleServerCrashProcedure(MasterRpcServices.java:2488)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
at
org.apache.hadoop.hbase.client.HBaseHbck.scheduleServerCrashProcedures(HBaseHbck.java:175)
at
org.apache.hadoop.hbase.client.Hbck.scheduleServerCrashProcedure(Hbck.java:118)
at org.apache.hbase.HBCK2.scheduleRecoveries(HBCK2.java:345)
at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:746)
at org.apache.hbase.HBCK2.run(HBCK2.java:631)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hbase.HBCK2.main(HBCK2.java:865)
{code}
A simple fix makes it so I can schedule an SCP which indeed clears out the
'Unknown Server' to restore saneness on the cluster.
As to how to get 'Unknown Server':
1. The current scenario came about because of this exception while processing a
server crash procedure made it so the SCP exited just after splitting logs but
before it cleared old assigns. A new server instance that came up after this
one went down purged the server from dead servers list though there were still
Procedures in flight (The cluster was under a crippling overloading)
{code}
2019-11-02 21:02:34,775 DEBUG
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: Done splitting
WALs pid=112532, state=RUNNABLE:SERVER_CRASH_SPLIT_LOGS, locked=true;
ServerCrashProcedure server=s1.d.com,16020,1572668980355, splitWal=true,
meta=false
2019-11-02 21:02:34,775 DEBUG
org.apache.hadoop.hbase.procedure2.RootProcedureState: Add procedure
pid=112532, state=RUNNABLE:SERVER_CRASH_ASSIGN, locked=true;
ServerCrashProcedure server=s1.d.com,16020,1572668980355, splitWal=true,
meta=false as the 2th rollback step
2019-11-02 21:02:34,779 INFO
org.apache.hadoop.hbase.master.procedure.ServerCrashProcedure: pid=112532,
state=RUNNABLE:SERVER_CRASH_ASSIGN, locked=true; ServerCrashProcedure
server=s1.d.com,16020,1572668980355, splitWal=true, meta=false found RIT
pid=101251, ppid=101123, state=SUCCESS, bypass=LOG-REDACTED
TransitRegionStateProcedure
[jira] [Resolved] (HBASE-23242) Check style incorrectly formatted - Intellij complains
[
https://issues.apache.org/jira/browse/HBASE-23242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Vissapragada resolved HBASE-23242.
--
Resolution: Invalid
> Check style incorrectly formatted - Intellij complains
> --
>
> Key: HBASE-23242
> URL: https://issues.apache.org/jira/browse/HBASE-23242
> Project: HBase
> Issue Type: Bug
> Components: tooling
>Affects Versions: master
>Reporter: Bharath Vissapragada
>Assignee: Bharath Vissapragada
>Priority: Minor
>
> I get the following two errors when I try to import the checkstyle file into
> CheckStyle-IDEA plugin. Interestingly mvn target works ok.
>
> {noformat}
> - com.puppycrawl.tools.checkstyle.api.CheckstyleException: cannot initialize
> module TreeWalker - TreeWalker is not allowed as a parent of LineLength
> Please review 'Parent Module' section for this Check in web documentation if
> Check is standard.
> {noformat}
> As per the documentation [1]
> - LineLength's parent should be Checker.
> Fixing this parent-child relationship, I was able to import the file
> correctly. Will attach the patch shortly.
> [1]https://checkstyle.sourceforge.io/config_sizes.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-23246) Fix error prone warning in TestMetricsUserSourceImpl
Andrew Kyle Purtell created HBASE-23246: --- Summary: Fix error prone warning in TestMetricsUserSourceImpl Key: HBASE-23246 URL: https://issues.apache.org/jira/browse/HBASE-23246 Project: HBase Issue Type: Sub-task Reporter: Andrew Kyle Purtell Fix For: 1.6.0 TestMetricsUserSourceImpl.java:[50,29] [SelfComparison] An object is compared to itself -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23210) Backport HBASE-15519 (Add per-user metrics) to branch-1
[ https://issues.apache.org/jira/browse/HBASE-23210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-23210. - Resolution: Fixed > Backport HBASE-15519 (Add per-user metrics) to branch-1 > --- > > Key: HBASE-23210 > URL: https://issues.apache.org/jira/browse/HBASE-23210 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.5.1 > > > We will need HBASE-15519 in branch-1 for eventual backport of HBASE-23065. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23244) NPEs running Canary
[
https://issues.apache.org/jira/browse/HBASE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-23244.
---
Resolution: Fixed
Pushed an amendment. Re-resolving. Thanks for noticing [~busbey]
> NPEs running Canary
> ---
>
> Key: HBASE-23244
> URL: https://issues.apache.org/jira/browse/HBASE-23244
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> There's a few. Here is one:
> {code}
> 19/11/01 23:54:16 ERROR tool.Canary: Run regionMonitor failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1582)
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1562)
> at
> org.apache.hadoop.hbase.tool.CanaryTool.access$100(CanaryTool.java:124)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionMonitor.run(CanaryTool.java:1366)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HBASE-23185) High cpu usage because getTable()#put() gets config value every time
[ https://issues.apache.org/jira/browse/HBASE-23185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reopened HBASE-23185: --- Let me reopen and revert from the two branches. > High cpu usage because getTable()#put() gets config value every time > > > Key: HBASE-23185 > URL: https://issues.apache.org/jira/browse/HBASE-23185 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 1.5.0, 1.4.10, 1.2.12, 1.3.5 >Reporter: Shinya Yoshida >Assignee: Shinya Yoshida >Priority: Major > Labels: performance > Fix For: 1.6.0 > > Attachments: Screenshot from 2019-10-18 12-38-14.png, Screenshot from > 2019-10-18 13-03-24.png > > > When we analyzed the performance of our hbase application with many puts, we > found that Configuration methods use many CPU resources: > !Screenshot from 2019-10-18 12-38-14.png|width=460,height=205! > As you can see, getTable().put() is calling Configuration methods which cause > regex or synchronization by Hashtable. > This should not happen in 0.99.2 because > https://issues.apache.org/jira/browse/HBASE-12128 addressed such an issue. > However, it's reproducing nowadays by bugs or leakages after many code > evoluations between 0.9x and 1.x. > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java#L369-L374] > ** finishSetup is called every new HTable() e.g. every con.getTable() > ** So getInt is called everytime and it does regex > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/BufferedMutatorImpl.java#L115] > ** BufferedMutatorImpl is created every first put for HTable e.g. > con.getTable().put() > ** Create ConnectionConf every time in BufferedMutatorImpl constructor > ** ConnectionConf gets config value in the constructor > # > [https://github.com/apache/hbase/blob/dd9eadb00f9dcd071a246482a11dfc7d63845f00/hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncProcess.java#L326] > ** AsyncProcess is created in BufferedMutatorImpl constructor, so new > AsyncProcess is created by con.getTable().put() > ** AsyncProcess parse many configurations > So, con.getTable().put() is heavy operation for CPU because of getting config > value. > > With in-house patch for this issue, we observed about 10% improvement on > max-throughput (e.g. CPU usage) at client-side: > !Screenshot from 2019-10-18 13-03-24.png|width=508,height=223! > > Seems branch-2 is not affected because client implementation has been changed > dramatically. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HBASE-23244) NPEs running Canary
[
https://issues.apache.org/jira/browse/HBASE-23244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reopened HBASE-23244:
-
This caused a findbugs failure.
>From precommit report, same thing pinging nightly and all new precommits as a
>branch problem:
{code}
FindBugsmodule:hbase-server
Nullcheck of CanaryTool$RegionTask.region at line 495 of value previously
dereferenced in org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read() At
CanaryTool.java:495 of value previously dereferenced in
org.apache.hadoop.hbase.tool.CanaryTool$RegionTask.read() At
CanaryTool.java:[line 438]
{code}
> NPEs running Canary
> ---
>
> Key: HBASE-23244
> URL: https://issues.apache.org/jira/browse/HBASE-23244
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 2.2.2
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.1.8, 2.2.3
>
>
> There's a few. Here is one:
> {code}
> 19/11/01 23:54:16 ERROR tool.Canary: Run regionMonitor failed
> java.lang.NullPointerException
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1582)
> at org.apache.hadoop.hbase.tool.CanaryTool.sniff(CanaryTool.java:1562)
> at
> org.apache.hadoop.hbase.tool.CanaryTool.access$100(CanaryTool.java:124)
> at
> org.apache.hadoop.hbase.tool.CanaryTool$RegionMonitor.run(CanaryTool.java:1366)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
RE: [ANNOUNCE] Please welcome Balazs Meszaros to the Apache HBase PMC
Congratulations Balazs Meszaros..!! Regards, Pankaj -Original Message- From: Sean Busbey [mailto:bus...@apache.org] Sent: 24 October 2019 20:05 To: dev ; Hbase-User Subject: [ANNOUNCE] Please welcome Balazs Meszaros to the Apache HBase PMC On behalf of the Apache HBase PMC I am pleased to announce that Balazs Meszaros has accepted our invitation to become a PMC member on the HBase project. We appreciate Balazs stepping up to take more responsibility in the HBase project. Please join me in welcoming Balazs to the HBase PMC! As a reminder, if anyone would like to nominate another person as a committer or PMC member, even if you are not currently a committer or PMC member, you can always drop a note to priv...@hbase.apache.org to let us know.
RE: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache HBase PMC
Congratulations Wellington..!! Regards, Pankaj -Original Message- From: Sean Busbey [mailto:bus...@apache.org] Sent: 24 October 2019 01:47 To: dev ; Hbase-User Subject: [ANNOUNCE] Please welcome Wellington Chevreuil to the Apache HBase PMC On behalf of the Apache HBase PMC I am pleased to announce that Wellington Chevreuil has accepted our invitation to become a PMC member on the HBase project. We appreciate Wellington stepping up to take more responsibility in the HBase project. Please join me in welcoming Wellington to the HBase PMC! As a reminder, if anyone would like to nominate another person as a committer or PMC member, even if you are not currently a committer or PMC member, you can always drop a note to priv...@hbase.apache.org to let us know.
RE: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC
Congratulations Sakthi..!! Regards, Pankaj -Original Message- From: Sean Busbey [mailto:bus...@apache.org] Sent: 24 October 2019 01:45 To: dev ; Hbase-User Subject: [ANNOUNCE] Please welcome Sakthi to the Apache HBase PMC On behalf of the Apache HBase PMC I am pleased to announce that Sakthi has accepted our invitation to become a PMC member on the HBase project. We appreciate Sakthi stepping up to take more responsibility in the HBase project. Please join me in welcoming Jan to the HBase PMC! As a reminder, if anyone would like to nominate another person as a committer or PMC member, even if you are not currently a committer or PMC member, you can always drop a note to priv...@hbase.apache.org to let us know.
[jira] [Created] (HBASE-23245) All MutableHistogram implementations should remove maxExpected
Viraj Jasani created HBASE-23245: Summary: All MutableHistogram implementations should remove maxExpected Key: HBASE-23245 URL: https://issues.apache.org/jira/browse/HBASE-23245 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 3.0.0, 2.3.0, 1.6.0 Reporter: Viraj Jasani Assignee: Viraj Jasani MutableHistogram is using HistogramImpl() and not using maxExpected. From all implementation classes, maxExpected should be removed as constructor argument. Also, HistogramImpl() should pass long casting for Integer.MAX_VALUE << 2. -- This message was sent by Atlassian Jira (v8.3.4#803005)
