Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically

[Sukumar Maddineni]

Huge +1 and love to have that option in 1.6+ version as well, based on the
experience we always recommend to be extra careful running HBCK fix
inconsistencies directly as we had incidents where running hbck fix without
complete knowledge ended up causing more damage to already broken clusters.

--
Sukumar

On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad 
wrote:

> Hello HBase Folks
>
> Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to
> be fixed by manually by executing HBCK2 commands.
> HBCK2 requires high HBase skills. It is bit difficult for maintenance
> personals to figure out which command, when and in which order to be
> executed.
>
> Is there any effort going on in community to automate fixing these
> inconsistencies?  I also would like to contribute there.
>
> I was thinking, maybe we can expose CJ and HBCK chore reported
> inconsistencies through a new master API and then provide option to fix
> these inconsistencies. Basically adding two new commands in HBCK2
> -listInconsistencies list CJ and HBCK chore reported inconsistencies
> -fixInconsistencies  fix CJ and HBCK chore reported inconsistencies (Not
> sure if possible to fix all inconsistencies, need to analyze all
> inconsistencies case by case, but some are very straight forward for
> example holes and overlap)
>
> Any thoughts/inputs highly appreciated.
>
> Regards
> -Arshad
>
>
>

-- 

Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically

[Mohammad arshad]

Hello HBase Folks

Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies to be 
fixed by manually by executing HBCK2 commands.
HBCK2 requires high HBase skills. It is bit difficult for maintenance personals 
to figure out which command, when and in which order to be executed.

Is there any effort going on in community to automate fixing these 
inconsistencies?  I also would like to contribute there.

I was thinking, maybe we can expose CJ and HBCK chore reported inconsistencies 
through a new master API and then provide option to fix these inconsistencies. 
Basically adding two new commands in HBCK2
-listInconsistencies list CJ and HBCK chore reported inconsistencies
-fixInconsistencies  fix CJ and HBCK chore reported inconsistencies (Not sure 
if possible to fix all inconsistencies, need to analyze all inconsistencies 
case by case, but some are very straight forward for example holes and overlap)

Any thoughts/inputs highly appreciated.

Regards
-Arshad

[jira] [Created] (HBASE-24831) Avoid invoke Counter using reflection in SnapshotInputFormat

[Yechao Chen (Jira)]

Yechao Chen created HBASE-24831:
---

 Summary: Avoid invoke Counter using reflection  in 
SnapshotInputFormat
 Key: HBASE-24831
 URL: https://issues.apache.org/jira/browse/HBASE-24831
 Project: HBase
  Issue Type: Improvement
Reporter: Yechao Chen
Assignee: Yechao Chen


In TableRecordReaderImpl we invoke Counter increment by reflection

This will be called nextKeyValue() in TableSnapshotInputFormat 

reflection invoke is very slower than normal method call

we can avoid these to improve the read performance



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24830) Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2

[Toshihiro Suzuki (Jira)]

Toshihiro Suzuki created HBASE-24830:


 Summary: Some tests involving RS crash fail with 
NullPointerException after HBASE-24632 in branch-2
 Key: HBASE-24830
 URL: https://issues.apache.org/jira/browse/HBASE-24830
 Project: HBase
  Issue Type: Bug
Reporter: Toshihiro Suzuki


In some tests involving RS crash in branch-2, the following 
NullPointerException is happening repeatedly and the tests finally fail due to 
timeout:
{code:java}
2020-08-06 16:03:43,101 ERROR [RS_LOG_REPLAY_OPS-regionserver/10.0.1.11:0-1] 
handler.RSProcedureHandler(51): pid=17
java.lang.NullPointerException
at 
org.apache.hadoop.hbase.regionserver.SplitLogWorker.splitLog(SplitLogWorker.java:107)
at 
org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:100)
at 
org.apache.hadoop.hbase.regionserver.SplitWALCallable.call(SplitWALCallable.java:45)
at 
org.apache.hadoop.hbase.regionserver.handler.RSProcedureHandler.process(RSProcedureHandler.java:49)
at 
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24826) Add some comments for processlist in hbase shell

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24826.
---
Fix Version/s: 2.4.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2+. Thanks for the patch [~filtertip]  and review [~bitoffdev]

> Add some comments for processlist in hbase shell
> 
>
> Key: HBASE-24826
> URL: https://issues.apache.org/jira/browse/HBASE-24826
> Project: HBase
>  Issue Type: Improvement
>  Components: shell
>Reporter: Zheng Wang
>Assignee: Zheng Wang
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> It is not easy to understand the difference of the options for processlist, 
> so maybe some comments is necessary.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24659) Calculate FIXED_OVERHEAD automatically

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24659.
---
Fix Version/s: 2.4.0
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed on branch-2 and master (pushed because I held up the commit by not 
responding to ping). Thanks for the patch [~niuyulin]

> Calculate FIXED_OVERHEAD automatically
> --
>
> Key: HBASE-24659
> URL: https://issues.apache.org/jira/browse/HBASE-24659
> Project: HBase
>  Issue Type: Improvement
>Reporter: Duo Zhang
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
> Attachments: 50M-performance.pdf
>
>
> Now the FIXED_OVERHEAD in some classes are maintained manually, an we have a 
> method to TestHeapSizes to confirm that the value is correct.
> But it is really hard for developers to count the fields in a complicated 
> class like HRegion. Since we have the ability to calcuate the accurate size 
> in UT, I think we it is also possible to calcuate it when loading the class, 
> which is a one time operation so should not effect the performance too much.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24817) Allow configuring WALEntry filters on ReplicationSource

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24817.
---
Fix Version/s: 2.4.0
   3.0.0-alpha-1
 Assignee: Michael Stack
   Resolution: Fixed

Pushed to branch-2+. Thanks for reviews [~anoop.hbase]  and [~vjasani]

> Allow configuring WALEntry filters on ReplicationSource
> ---
>
> Key: HBASE-24817
> URL: https://issues.apache.org/jira/browse/HBASE-24817
> Project: HBase
>  Issue Type: Sub-task
>  Components: Replication, wal
>Affects Versions: 3.0.0-alpha-1, 2.4.0
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> The parent issue is about enabling memstore replication of meta Regions. 
> As-is, the ReplicationSource is hardcoded to filter out hbase:meta 
> WALEntries; they are not forwarded for Replication.
> This issue is all internals making it so can create an instance of 
> ReplicationSource with a different base set of WALEntry filters. We also add 
> a means of filtering WALs by path name.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24829) Adding Bulk Operation Support through PySpark

[Manas (Jira)]

Manas created HBASE-24829:
-

 Summary: Adding Bulk Operation Support through PySpark
 Key: HBASE-24829
 URL: https://issues.apache.org/jira/browse/HBASE-24829
 Project: HBase
  Issue Type: New Feature
  Components: hbase-connectors, spark
Reporter: Manas


Bulk Operations are supported in Scala but not in PySpark, we should add this 
support with some examples as well similar to the ones done in Scala and Java. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24828) Unable to call HBaseContext From PySpark

[Manas (Jira)]

Manas created HBASE-24828:
-

 Summary: Unable to call HBaseContext From PySpark
 Key: HBASE-24828
 URL: https://issues.apache.org/jira/browse/HBASE-24828
 Project: HBase
  Issue Type: Bug
  Components: hbase-connectors
Reporter: Manas


I'm able to call the JavaHBaseContext From PySpark but not HBaseContext

 
{code:java}
temp = sc._jvm.org.apache.hadoop.hbase.HBaseConfiguration
conf = temp.create()
hbaseCon = sc._jvm.org.apache.hadoop.hbase.spark.HBaseContext(sc, conf){code}
AttributeError: 'SparkContext' object has no attribute '_get_object_id'
AttributeErrorTraceback (most recent call last)
in engine
> 1 hbaseCon = sc._jvm.org.apache.hadoop.hbase.spark.HBaseContext(sc, conf)

/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py in __call__(self, 
*args)
   1543 
   1544 args_command = "".join(
-> 1545 [get_command_part(arg, self._pool) for arg in new_args])
   1546 
   1547 command = proto.CONSTRUCTOR_COMMAND_NAME +\

/usr/local/lib/python3.6/site-packages/py4j/java_gateway.py in (.0)
   1543 
   1544 args_command = "".join(
-> 1545 [get_command_part(arg, self._pool) for arg in new_args])
   1546 
   1547 command = proto.CONSTRUCTOR_COMMAND_NAME +\

/usr/local/lib/python3.6/site-packages/py4j/protocol.py in 
get_command_part(parameter, python_proxy_pool)
296 command_part += ";" + interface
297 else:
--> 298 command_part = REFERENCE_TYPE + parameter._get_object_id()
299 
300 command_part += "\n"

AttributeError: 'SparkContext' object has no attribute '_get_object_id'



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-21905) TestFIFOCompactionPolicy is flaky

[Bharath Vissapragada (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-21905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharath Vissapragada resolved HBASE-21905.
--
Resolution: Fixed

> TestFIFOCompactionPolicy is flaky
> -
>
> Key: HBASE-21905
> URL: https://issues.apache.org/jira/browse/HBASE-21905
> Project: HBase
>  Issue Type: Test
>  Components: test
>Affects Versions: 3.0.0-alpha-1, 1.5.0, 2.3.0
>Reporter: Andrew Kyle Purtell
>Assignee: Bharath Vissapragada
>Priority: Major
>  Labels: branch-1
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0, 2.2.7
>
> Attachments: 
> org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy-output.txt,
>  testFIFOCompactionPolicyExpiredEmptyHFiles-failure-log.txt
>
>
> java.lang.IllegalArgumentException , overlaps with 
> For example:
> [ERROR] 
> testFIFOCompactionPolicyExpiredEmptyHFiles(org.apache.hadoop.hbase.regionserver.compactions.TestFIFOCompactionPolicy)
>   Time elapsed: 3.321 s  <<< ERROR!
> java.io.IOException: 
> java.io.IOException: 
> [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69,
>  
> hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe]
>  overlaps with 
> [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69]
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2438)
>     at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:124)
>     at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:297)
>     at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> Caused by: java.lang.IllegalArgumentException: 
> [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69,
>  
> hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c65648691f614b2d8dd4b586c5923bfe]
>  overlaps with 
> [hdfs://localhost:41525/user/apurtell/test-data/734de07d-1f22-46a9-a1f5-96ad4578450b/data/default/testFIFOCompactionPolicyExpiredEmptyHFiles/c4f673438e09d7ef5a9b79b363639cde/f/c0c5836c1f714f78847cf00326586b69]
>     at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:119)
>     at 
> org.apache.hadoop.hbase.regionserver.HStore.addToCompactingFiles(HStore.java:1824)
>     at 
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1798)
>     at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:415)
>     at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:388)
>     at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompactionInternal(CompactSplitThread.java:317)
>     at 
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:306)
>     at 
> org.apache.hadoop.hbase.regionserver.RSRpcServices.compactRegion(RSRpcServices.java:1513)
>     at 
> org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:23649)
>     at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2380)
>     ... 3 more



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24662) Update DumpClusterStatusAction to notice changes in region server count

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-24662.
--
Resolution: Fixed

> Update DumpClusterStatusAction to notice changes in region server count
> ---
>
> Key: HBASE-24662
> URL: https://issues.apache.org/jira/browse/HBASE-24662
> Project: HBase
>  Issue Type: Task
>  Components: integration tests
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.1, 1.7.0, 2.4.0
>
>
> Sometimes running chaos monkey, I've found that we lose accounting of region 
> servers. I've taken to a manual process of checking the reported list against 
> a known reference. It occurs to me that ChaosMonkey has a known reference, 
> and it can do this accounting for me.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24827) BackPort HBASE-11554 Remove Reusable poolmap Rpc client type.

[Michael Stack (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Stack resolved HBASE-24827.
---
Fix Version/s: 3.0.0-alpha-1
   Resolution: Fixed

Thanks for the patch [~niuyulin]

> BackPort HBASE-11554 Remove Reusable poolmap Rpc client type.
> -
>
> Key: HBASE-24827
> URL: https://issues.apache.org/jira/browse/HBASE-24827
> Project: HBase
>  Issue Type: Improvement
>  Components: Client
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0-alpha-1
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24823) Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals to master branch

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HBASE-24823.
--
Fix Version/s: 2.4.0
   2.3.1
   3.0.0-alpha-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Port HBASE-22762 Print the delta between phases in the 
> split/merge/compact/flush transaction journals to master branch
> --
>
> Key: HBASE-24823
> URL: https://issues.apache.org/jira/browse/HBASE-24823
> Project: HBase
>  Issue Type: Improvement
>Reporter: Sandeep Guggilam
>Assignee: Sandeep Guggilam
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.3.1, 2.4.0
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24827) BackPort HBASE-11554 Remove Reusable poolmap Rpc client type.

[niuyulin (Jira)]

niuyulin created HBASE-24827:


 Summary: BackPort HBASE-11554 Remove Reusable poolmap Rpc client 
type.
 Key: HBASE-24827
 URL: https://issues.apache.org/jira/browse/HBASE-24827
 Project: HBase
  Issue Type: Improvement
  Components: Client
Reporter: niuyulin
Assignee: niuyulin






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24826) Add some comments for command of processlist

[Zheng Wang (Jira)]

Zheng Wang created HBASE-24826:
--

 Summary: Add some comments for command of processlist
 Key: HBASE-24826
 URL: https://issues.apache.org/jira/browse/HBASE-24826
 Project: HBase
  Issue Type: Improvement
  Components: shell
Reporter: Zheng Wang
Assignee: Zheng Wang


It is not easy to understand the different of the options for processlist, so 
maybe some comments is necessary.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24825) NettyRpcConnection.scheduleRelogin method should catch Throwable exception

[niuyulin (Jira)]

niuyulin created HBASE-24825:


 Summary: NettyRpcConnection.scheduleRelogin method should catch 
Throwable exception
 Key: HBASE-24825
 URL: https://issues.apache.org/jira/browse/HBASE-24825
 Project: HBase
  Issue Type: Improvement
  Components: Client
Reporter: niuyulin
Assignee: niuyulin






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-24819) Fix flaky test TestRaceBetweenSCPAndDTP and TestRaceBetweenSCPAndTRSP for branch-2.2

[Guanghao Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-24819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guanghao Zhang resolved HBASE-24819.

Fix Version/s: 2.2.6
   Resolution: Fixed

Pushed. Thanks [~meiyi] for reviewing.

> Fix flaky test TestRaceBetweenSCPAndDTP and TestRaceBetweenSCPAndTRSP for 
> branch-2.2
> 
>
> Key: HBASE-24819
> URL: https://issues.apache.org/jira/browse/HBASE-24819
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Major
> Fix For: 2.2.6
>
>
> Backport HBASE-23805 and HBASE-24338



--
This message was sent by Atlassian Jira
(v8.3.4#803005)