[jira] [Updated] (HBASE-16730) Exclude junit as a transitive dependency from hadoop-common
[ https://issues.apache.org/jira/browse/HBASE-16730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16730: Fix Version/s: 2.0.0 > Exclude junit as a transitive dependency from hadoop-common > --- > > Key: HBASE-16730 > URL: https://issues.apache.org/jira/browse/HBASE-16730 > Project: HBase > Issue Type: Improvement > Components: hbase >Reporter: Nils Larsgård >Priority: Trivial > Labels: hbase-client, junit > Fix For: 2.0.0 > > Original Estimate: 20m > Remaining Estimate: 20m > > add exclusion to the hadoop-common dependency in hbase-client: exclude junit -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532969#comment-15532969 ] Ted Yu commented on HBASE-16179: 'To build two modules' should have been 'To build two modules simultaneously'. There're two profiles for Spark currently. Two sets of artifacts can be produced based on the profiles. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532946#comment-15532946
]
Sean Busbey commented on HBASE-16179:
-
{quote}
bq. can we work it so both modules build all the time
To build two modules we would need another module to use Spark 1.6 (the
non-default) compat.
{quote}
I'm a bit unclear on things. Could you walk me through how we expect a
downstream user who wants to run our storage adapter on Spark 1.6 would do so?
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533007#comment-15533007
]
Sean Busbey commented on HBASE-16179:
-
{quote}
Similar to what 0.98 offers:
https://mvnrepository.com/artifact/org.apache.hbase/hbase/0.98.20-hadoop1
https://mvnrepository.com/artifact/org.apache.hbase/hbase/0.98.20-hadoop2
{quote}
the hbase-spark module doesn't include a classifier or a variable in its
version number.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532994#comment-15532994 ] Ted Yu commented on HBASE-16179: Similar to what 0.98 offers: https://mvnrepository.com/artifact/org.apache.hbase/hbase/0.98.20-hadoop1 https://mvnrepository.com/artifact/org.apache.hbase/hbase/0.98.20-hadoop2 > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16723:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Pankaj.
Thanks for the review, Ashish.
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch,
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532977#comment-15532977
]
Sean Busbey commented on HBASE-16179:
-
{quote}
'To build two modules' should have been 'To build two modules simultaneously'.
There're two profiles for Spark currently.
Two sets of artifacts can be produced based on the profiles.
{quote}
Okay, so what does using them look like?
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16725:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the reviews.
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt,
> 16725.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533150#comment-15533150
]
Jonathan Hsieh commented on HBASE-16712:
On the new xerces dep -- here's my evidence from the build:
{code}
--
This product includes Xerces2 Java Parser licensed under the The Apache
Software License, Version 2.0.
${dep.licenses[0].comments}
Please check this License for acceptability here:
https://www.apache.org/legal/resolved
If it is okay, then update the list named 'non_aggregate_fine' in the
LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.
More info on the dependency:
xerces
xercesImpl
2.9.1
maven central search
g:xerces AND a:xercesImpl AND v:2.9.1
project website
http://xerces.apache.org/xerces2-j
project source
http://svn.apache.org/viewvc/maven/pom/tags/apache-4/xercesImpl
{code}
Let me do a 'mvn clean install -DskipTests' with the different profiles, and
then I'll attach the deps for h2 and h3. I recently learned that 'mvn install'
is required for dependency:tree to give the right information.
The Go license is brought in by re2j
{code}
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.0.0-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
..
[INFO] | +- com.google.re2j:re2j:jar:1.0:compile
{code}
For the go license can you be more specific about what you suggest we do?
Some of the failures wiht ${dep.licenses.isEmpty()} actually fail until the
next line which you didn't include which does a ${dep.license[0]...) and gets a
very cryptic array out of bounds error. I actually got the new list by
repeated going through looking at this output, "fixing it", and then trying
again until all were addressed. I'll leave more explicit comments in the next
rev.
We should have that (for LICENSE.vm and NOTICE.vm) error message googlable in
our docs so that others who run into it don't have to struggle to figure this
out as much.
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) The result retrieved from Get and Scan will be different if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() converts the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. > 1) The result retrieved from Get and Scan will be different if we use the > empty filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) The result retrieved from Get and Scan will be different if we use the empty filter. Scan doesn't return any data but Get does. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) The result retrieved from Get and Scan will be different if we use the empty filter. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() converts the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. > 1) The result retrieved from Get and Scan will be different if we use the > empty filter. Scan doesn't return any data but Get does. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533180#comment-15533180
]
Sean Busbey commented on HBASE-16712:
-
{quote}
On the new xerces dep – here's my evidence from the build:
{code}
--
This product includes Xerces2 Java Parser licensed under the The Apache
Software License, Version 2.0.
${dep.licenses[0].comments}
Please check this License for acceptability here:
https://www.apache.org/legal/resolved
If it is okay, then update the list named 'non_aggregate_fine' in the
LICENSE.vm file.
If it isn't okay, then revert the change that added the dependency.
More info on the dependency:
xerces
xercesImpl
2.9.1
maven central search
g:xerces AND a:xercesImpl AND v:2.9.1
project website
http://xerces.apache.org/xerces2-j
project source
http://svn.apache.org/viewvc/maven/pom/tags/apache-4/xercesImpl
{code}
Let me do a 'mvn clean install -DskipTests' with the different profiles, and
then I'll attach the deps for h2 and h3. I recently learned that 'mvn install'
is required for dependency:tree to give the right information.
{quote}
Thanks, that sounds good. Hadoop has been publishing an unneeded dependency on
Xerces since 2.6 or 2.7. This might mean they're just leaking it from a
different place, or it might mean that there is actually a need for downstream
clients now. I'd like us to know which it is before considering starting to
ship Xerces.
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16663:
-
Status: Patch Available (was: Open)
Patch for master branch, please review.
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23, 1.2.4
>
> Attachments: HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
> 2016-09-21 12:09:08,261 INFO
>
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() convert the Get to Scan without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() convert the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage. > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533167#comment-15533167
]
Sean Busbey edited comment on HBASE-16712 at 9/29/16 4:00 PM:
--
{quote}
We should have that (for LICENSE.vm and NOTICE.vm) error message googlable in
our docs so that others who run into it don't have to struggle to figure this
out as much.
{quote}
agreed. I thought we had an open JIRA for this, but it turns out I was thinking
of the one to redo things to rely on the enforcer plugin (HBASE-16351).
was (Author: busbey):
{quote}
We should have that (for LICENSE.vm and NOTICE.vm) error message googlable in
our docs so that others who run into it don't have to struggle to figure this
out as much.
{quote}
agreed. I thought we had an open JIRA for this, but it turns out I was thinking
of the one to redo things to rely on enforce (HBASE-16351).
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533167#comment-15533167
]
Sean Busbey commented on HBASE-16712:
-
{quote}
We should have that (for LICENSE.vm and NOTICE.vm) error message googlable in
our docs so that others who run into it don't have to struggle to figure this
out as much.
{quote}
agreed. I thought we had an open JIRA for this, but it turns out I was thinking
of the one to redo things to rely on enforce (HBASE-16351).
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16712: --- Attachment: h3-deps h2-deps h2-deps has dependency:tree with default hadoop2 profile, while h3-deps has hadoop.profile=3.0 > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15721) Optimization in cloning cells into MSLAB
[
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533307#comment-15533307
]
Anoop Sam John commented on HBASE-15721:
We are adding the write method to ExtendedCell now. So no concern of polluting
the Streamable interface. (we dont have it any more)
MSLAB responsibility is being increased. [~stack] ur concerns are addressed
now.. May I commit this?
> Optimization in cloning cells into MSLAB
>
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch,
> HBASE-15721_V3.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying
> it to MSLAB chunk area. This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output,
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
> pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
> pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
> }
> {code}
> Copied in 9 steps and we end up parsing all lengths. When the cell
> implementation is backed by a single byte[] (Like KeyValue) this can be done
> in single step copy.
> Also avoid lots of garbage while doing this MSLAB copy. With every cell, we
> used to create a ByteRange instance to pass the allocated chunk + offset from
> MSLAB to Segment. Now we can make the MSLAB impl to do the allocation and
> copy of the cell so that one object creation per cell addition can be avoided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533119#comment-15533119 ] Ted Yu commented on HBASE-16179: The version string can be defined at build time. [~apurtell]: Mind taking a look at the pom.xml changes in the patch and let me know what is missing ? Thanks > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from SCAN. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for GET operation. Any comments? Thanks. was: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) inconsistent behavior of Get and Scan which add the empty FilterList. No data is retrieved from SCAN. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for GET operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() convert the GET to SCAN without > scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from SCAN. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for GET operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for GET operation. Any comments? Thanks. was: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from SCAN. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for GET operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() convert the GET to SCAN without > scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for GET operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() convert the GET to SCAN without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for GET operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() convert the GET to SCAN without > scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage: > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533153#comment-15533153
]
Sean Busbey commented on HBASE-16179:
-
{quote}
The version string can be defined at build time.
{quote}
This sounds like it's putting a poorly defined additional burden on our release
managers. I'd like to see an update to the ref guide for building release
candidates for 2.0+ with this change in place so we can properly consider what
that burden will be.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) The result retrieved from Get and Scan will be different if we use the empty filter. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) The result retrieved from Get and Scan will be different if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() converts the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. > 1) The result retrieved from Get and Scan will be different if we use the > empty filter. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533176#comment-15533176
]
Sean Busbey commented on HBASE-16712:
-
{quote}
The Go license is brought in by re2j
[INFO] +- org.apache.hadoop:hadoop-common:jar:3.0.0-SNAPSHOT:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:3.0.0-SNAPSHOT:compile
..
[INFO] | +- com.google.re2j:re2j:jar:1.0:compile
For the go license can you be more specific about what you suggest we do?
{quote}
BSD-3 clause licensed dependencies are only supposed to be mentioned in the
LICENSE file, not hte NOTICE file. So either we can include a supplemental xml
update that says re2j is BSD-3 clause instead of 'the go license', or we should
update the processing in the NOTICE file to ensure we don't include any
information for dependencies that use the go license.
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533241#comment-15533241 ] Jonathan Hsieh commented on HBASE-16712: updated h3-deps to a dependency:tree that acutally has h3 in it. :) The strange thing is that there is no xerces in the tree! If you comment out the xerces bit in the supplemental, you end up with the error I posted earlier. Where else would this dependency get pulled in from? > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533292#comment-15533292 ] stack commented on HBASE-16264: --- New patch adds a hbase-endpoint module and moves all the bundled CPEPs to it, those that are non-core; i.e. bulk loading, aggregation, etc. This moves a bunch of code out of hbase-server module. Also moved the protos that are scoped to a module back to the module so rest now gets its protos back and so does rsgroup. This underlines the point we are trying to make about the difference between internal protobuf and external. Uploaded to review board a patch that is minus the generated files. > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, > HBASE-16264-Figure-how-to-deal-with-endpoints-and-sh.master.013.patch, > HBASE-16264.master.001.patch, HBASE-16264.master.002.patch, > HBASE-16264.master.003.patch, HBASE-16264.master.004.patch, > HBASE-16264.master.005.patch, HBASE-16264.master.006.patch, > HBASE-16264.master.007.patch, HBASE-16264.master.008.patch, > HBASE-16264.master.009.patch, HBASE-16264.master.010.patch, > HBASE-16264.master.011.patch, HBASE-16264.master.012.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15721) Optimization in cloning cells into MSLAB
[
https://issues.apache.org/jira/browse/HBASE-15721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-15721:
---
Description:
Before cells added to memstore CSLM, there is a clone of cell after copying it
to MSLAB chunk area. This is done not in an efficient way.
{code}
public static int appendToByteArray(final Cell cell, final byte[] output, final
int offset) {
int pos = offset;
pos = Bytes.putInt(output, pos, keyLength(cell));
pos = Bytes.putInt(output, pos, cell.getValueLength());
pos = appendKeyTo(cell, output, pos);
pos = CellUtil.copyValueTo(cell, output, pos);
if ((cell.getTagsLength() > 0)) {
pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
pos = CellUtil.copyTagTo(cell, output, pos);
}
return pos;
}
{code}
Copied in 9 steps and we end up parsing all lengths. When the cell
implementation is backed by a single byte[] (Like KeyValue) this can be done in
single step copy.
Also avoid lots of garbage while doing this MSLAB copy. With every cell, we
used to create a ByteRange instance to pass the allocated chunk + offset from
MSLAB to Segment. Now we can make the MSLAB impl to do the allocation and copy
of the cell so that one object creation per cell addition can be avoided.
was:
Before cells added to memstore CSLM, there is a clone of cell after copying it
to MSLAB chunk area. This is done not in an efficient way.
{code}
public static int appendToByteArray(final Cell cell, final byte[] output, final
int offset) {
int pos = offset;
pos = Bytes.putInt(output, pos, keyLength(cell));
pos = Bytes.putInt(output, pos, cell.getValueLength());
pos = appendKeyTo(cell, output, pos);
pos = CellUtil.copyValueTo(cell, output, pos);
if ((cell.getTagsLength() > 0)) {
pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
pos = CellUtil.copyTagTo(cell, output, pos);
}
return pos;
}
{code}
Copied in 9 steps and we end up parsing all lengths. When the cell
implementation is backed by a single byte[] (Like KeyValue) this can be done in
single step copy.
> Optimization in cloning cells into MSLAB
>
>
> Key: HBASE-15721
> URL: https://issues.apache.org/jira/browse/HBASE-15721
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-15721.patch, HBASE-15721_V2.patch,
> HBASE-15721_V3.patch
>
>
> Before cells added to memstore CSLM, there is a clone of cell after copying
> it to MSLAB chunk area. This is done not in an efficient way.
> {code}
> public static int appendToByteArray(final Cell cell, final byte[] output,
> final int offset) {
> int pos = offset;
> pos = Bytes.putInt(output, pos, keyLength(cell));
> pos = Bytes.putInt(output, pos, cell.getValueLength());
> pos = appendKeyTo(cell, output, pos);
> pos = CellUtil.copyValueTo(cell, output, pos);
> if ((cell.getTagsLength() > 0)) {
> pos = Bytes.putAsShort(output, pos, cell.getTagsLength());
> pos = CellUtil.copyTagTo(cell, output, pos);
> }
> return pos;
> }
> {code}
> Copied in 9 steps and we end up parsing all lengths. When the cell
> implementation is backed by a single byte[] (Like KeyValue) this can be done
> in single step copy.
> Also avoid lots of garbage while doing this MSLAB copy. With every cell, we
> used to create a ByteRange instance to pass the allocated chunk + offset from
> MSLAB to Segment. Now we can make the MSLAB impl to do the allocation and
> copy of the cell so that one object creation per cell addition can be avoided.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533142#comment-15533142
]
Ted Yu commented on HBASE-16663:
{code}
1244 private boolean execShutdown(final CoprocessorOperation ctx) throws
IOException {
{code}
Add javadoc for the return value.
Can the code in MasterCoprocessorHost and RegionCoprocessorHost be factored
into CoprocessorHost for re-use ?
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23, 1.2.4
>
> Attachments: HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
>
[jira] [Updated] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16712: --- Attachment: (was: h3-deps) > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16712: --- Attachment: h3-deps > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16663) JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
[
https://issues.apache.org/jira/browse/HBASE-16663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pankaj Kumar updated HBASE-16663:
-
Attachment: HBASE-16663.patch
> JMX ConnectorServer stopped when unauthorized user try to stop HM/RS/cluster
>
>
> Key: HBASE-16663
> URL: https://issues.apache.org/jira/browse/HBASE-16663
> Project: HBase
> Issue Type: Bug
> Components: metrics, security
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.23, 1.2.4
>
> Attachments: HBASE-16663.patch
>
>
> After HBASE-16284, unauthorized user will not able allowed to stop
> HM/RS/cluster, but while executing "cpHost.preStopMaster()", ConnectorServer
> will be stopped before AccessController validation.
> hbase-site.xml,
> {noformat}
>
> hbase.coprocessor.master.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
>
> hbase.coprocessor.regionserver.classes
>
> org.apache.hadoop.hbase.JMXListener,org.apache.hadoop.hbase.security.access.AccessController
>
> {noformat}
> HBaseAdmin.stopMaster(),
> {noformat}
> 2016-09-20 21:12:26,796 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 21:13:55,380 WARN
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 21:14:00,495 ERROR
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Exception occurred while stopping master
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopMaster(AccessController.java:1297)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost$68.call(MasterCoprocessorHost.java:821)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.execOperation(MasterCoprocessorHost.java:1188)
> at
> org.apache.hadoop.hbase.master.MasterCoprocessorHost.preStopMaster(MasterCoprocessorHost.java:817)
> at org.apache.hadoop.hbase.master.HMaster.stopMaster(HMaster.java:2352)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.stopMaster(MasterRpcServices.java:1364)
> {noformat}
> HBaseAdmin.stopRegionServer(rs-host-port),
> {noformat}
> 2016-09-20 20:59:01,234 INFO
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> hbase.JMXListener: ConnectorServer stopped!
> 2016-09-20 20:59:01,250 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> security.ShellBasedUnixGroupsMapping: got exception trying to get groups for
> user P72981
> ExitCodeException exitCode=1: id: P72981: No such user
> 2016-09-20 20:59:01,253 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=16020]
> regionserver.HRegionServer: The region server did not stop
> org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient
> permissions for user 'P72981' (global, action=ADMIN)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requireGlobalPermission(AccessController.java:546)
> at
> org.apache.hadoop.hbase.security.access.AccessController.requirePermission(AccessController.java:522)
> at
> org.apache.hadoop.hbase.security.access.AccessController.preStopRegionServer(AccessController.java:2501)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost$1.call(RegionServerCoprocessorHost.java:84)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.execOperation(RegionServerCoprocessorHost.java:256)
> at
> org.apache.hadoop.hbase.regionserver.RegionServerCoprocessorHost.preStop(RegionServerCoprocessorHost.java:80)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1905)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.stopServer(RSRpcServices.java:1961)
> {noformat}
> HBaseAdmin.shutdown(),
> {noformat}
> 2016-09-21 12:09:08,259 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> master.MasterRpcServices: Client=P72981//10.18.248.96 shutdown
> 2016-09-21 12:09:08,261 INFO
> [RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=16000]
> hbase.JMXListener: ConnectorServer stopped!
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() converts the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() convert the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() converts the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16731) Add Scan#setLoadColumnFamiliesOnDemand method to Get.
[ https://issues.apache.org/jira/browse/HBASE-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16731: -- Description: RSRpcServices#get() convert the Get to Scan without scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. was: RSRpcServices#get() convert the Get to Scan without scan#setLoadColumnFamiliesOnDemand. So it causes two disadvantage. 1) There are different result retrieved from Get and Scan if we use the empty filter. No data is retrieved from Scan. see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] 2) unable to read CF data lazily for Get operation. Any comments? Thanks. > Add Scan#setLoadColumnFamiliesOnDemand method to Get. > - > > Key: HBASE-16731 > URL: https://issues.apache.org/jira/browse/HBASE-16731 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Priority: Minor > > RSRpcServices#get() convert the Get to Scan without > scan#setLoadColumnFamiliesOnDemand. It causes two disadvantage. > 1) There are different result retrieved from Get and Scan if we use the empty > filter. No data is retrieved from Scan. > see [HBASE-16729 |https://issues.apache.org/jira/browse/HBASE-16729] > 2) unable to read CF data lazily for Get operation. > Any comments? Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533161#comment-15533161
]
Sean Busbey commented on HBASE-16712:
-
{quote}
actually fail until the next line which you didn't include which does a
${dep.license[0]...) and gets a very cryptic array out of bounds error. I
actually got the new list by repeated going through looking at this output,
"fixing it", and then trying again until all were addressed. I'll leave more
explicit comments in the next rev.
{quote}
Right, but those lines could change and all a person gets is a line number in
the NOTICE.vm file to look at. That's why the LICENSE.vm file has an explicit
failure condition as a part of the "something is wrong" check, with a comment
that tells folks they need to look at the partially generated file for the
cause.
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16712.v0.patch, hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533222#comment-15533222 ] Jonathan Hsieh commented on HBASE-16712: Oops, too quick - something is not right with the deps files -- they are nearly identical, let me try again. > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533256#comment-15533256 ] Sean Busbey commented on HBASE-16712: - or if you're unsure about what kind of information gets shred about your build environment with maven in debug mode, you could send me the log out of band. > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[
https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533253#comment-15533253
]
Sean Busbey commented on HBASE-16712:
-
{quote}The strange thing is that there is no xerces in the tree! If you comment
out the xerces bit in the supplemental, you end up with the error I posted
earlier. Where else would this dependency get pulled in from?
{quote}
yeah, this is very strange. Can you run the build that fails with a -X and then
post the log?
> fix hadoop-3.0 profile mvn install
> --
>
> Key: HBASE-16712
> URL: https://issues.apache.org/jira/browse/HBASE-16712
> Project: HBase
> Issue Type: Bug
> Components: build, hadoop3
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: h2-deps, h3-deps, hbase-16712.v0.patch,
> hbase-16712.v1.patch
>
>
> After the compile is fixed, mvn install fails due to transitive dependencies
> coming from hadoop3.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534901#comment-15534901 ] Hudson commented on HBASE-15984: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #34 (See [https://builds.apache.org/job/HBase-1.2-JDK7/34/]) HBASE-15984 Handle premature EOF treatment of WALs in replication. (busbey: rev 42dff8a58af02ec03fe97db34ab930defb79141f) * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationWALReaderManager.java > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534900#comment-15534900
]
Hudson commented on HBASE-16721:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #34 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/34/])
HBASE-16721 Concurrency issue in WAL unflushed seqId tracking (enis: rev
cf374af102f139a6176d05b97201bfa8d9f687be)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534899#comment-15534899
]
Hudson commented on HBASE-16732:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #34 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/34/])
HBASE-16732 Avoid possible NPE in MetaTableLocator (jerryjch: rev
bfb20c0c1fa40f0580d440747a16852d2deeb78e)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16653) Backport HBASE-11393 to all branches which support namespace
[ https://issues.apache.org/jira/browse/HBASE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534842#comment-15534842 ] Heng Chen commented on HBASE-16653: --- As [~enis] mentioned, we should leave the deprecated method here for branch-1 due to our compatibility contact. And HBASE-11393 has some compatibility issues, if we do backport to branch-1, we should take care of this. Please upload the patch to review board. [~zghaobac] > Backport HBASE-11393 to all branches which support namespace > > > Key: HBASE-16653 > URL: https://issues.apache.org/jira/browse/HBASE-16653 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.0.5, 1.3.1, 0.98.22, 1.1.7, 1.2.4 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16653-branch-1-v1.patch, > HBASE-16653-branch-1-v2.patch > > > As HBASE-11386 mentioned, the parse code about replication table-cfs config > will be wrong when table name contains namespace and we can only config the > default namespace's tables in the peer. It is a bug for all branches which > support namespace. HBASE-11393 resolved this by use a pb object but it was > only merged to master branch. Other branches still have this problem. I > thought we should fix this bug in all branches which support namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16736: --- Attachment: 16736.v1.txt > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16736.v1.txt > > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16736: --- Status: Patch Available (was: Open) > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16736.v1.txt > > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf-3.1.x
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535002#comment-15535002 ] Hudson commented on HBASE-16567: FAILURE: Integrated in Jenkins build HBASE-16264 #3 (See [https://builds.apache.org/job/HBASE-16264/3/]) HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this (stack: rev b4a729ed027621a062c4a1ad9c50d81e6fdd8758) * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/EncryptionProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/util/ByteStringer.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/ipc/protobuf/generated/TestRpcServiceProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClusterIdProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RPCProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/AdminProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MapReduceProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RegionNormalizerProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClusterStatusProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ProcedureProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RegionServerStatusProtos.java * (delete) hbase-protocol-shaded/src/main/java/com/google/protobuf/HBaseZeroCopyByteString.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ErrorHandlingProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/LoadBalancerProtos.java * (edit) hbase-protocol-shaded/pom.xml * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/TracingProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/FilterProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProcedureProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/HBaseProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ZooKeeperProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ClientProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/ipc/protobuf/generated/TestProcedureProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/CellProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/SnapshotProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/HFileProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/ipc/protobuf/generated/TestProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/WALProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/ComparatorProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/FSProtos.java > Upgrade to protobuf-3.1.x > - > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 16567.patch, HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- though read-time is still a TODO (this means pb3 is not > enough; we'll have to patch it -- or patch pb2.5). > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534855#comment-15534855
]
Hudson commented on HBASE-16721:
FAILURE: Integrated in Jenkins build HBase-1.4 #438 (See
[https://builds.apache.org/job/HBase-1.4/438/])
HBASE-16721 Concurrency issue in WAL unflushed seqId tracking (enis: rev
bf5a7aba5c0c83874f52cbd775dd280cb4a1cd49)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16735) Procedure v2 - Fix yield while holding locks
[ https://issues.apache.org/jira/browse/HBASE-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16735: Attachment: HBASE-16735-v0.patch > Procedure v2 - Fix yield while holding locks > > > Key: HBASE-16735 > URL: https://issues.apache.org/jira/browse/HBASE-16735 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16735-v0.patch > > > Sched fix for proc holding locks. the proc was added back to the queue, but > the queue was not re-added to the runq -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16735) Procedure v2 - Fix yield while holding locks
[ https://issues.apache.org/jira/browse/HBASE-16735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16735: Status: Patch Available (was: Open) > Procedure v2 - Fix yield while holding locks > > > Key: HBASE-16735 > URL: https://issues.apache.org/jira/browse/HBASE-16735 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16735-v0.patch > > > Sched fix for proc holding locks. the proc was added back to the queue, but > the queue was not re-added to the runq -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16735) Procedure v2 - Fix yield while holding locks
Matteo Bertozzi created HBASE-16735: --- Summary: Procedure v2 - Fix yield while holding locks Key: HBASE-16735 URL: https://issues.apache.org/jira/browse/HBASE-16735 Project: HBase Issue Type: Sub-task Components: proc-v2 Affects Versions: 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 Attachments: HBASE-16735-v0.patch Sched fix for proc holding locks. the proc was added back to the queue, but the queue was not re-added to the runq -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16736: --- Attachment: 16736.v1.txt > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16736.v1.txt > > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16736) Add getter to ResizableBlockCache for max size
[
https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534977#comment-15534977
]
Hadoop QA commented on HBASE-16736:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 12s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red} 9m 7s {color}
| {color:red} Docker failed to build yetus/hbase:7bda515. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831057/16736.v1.txt |
| JIRA Issue | HBASE-16736 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3778/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add getter to ResizableBlockCache for max size
> --
>
> Key: HBASE-16736
> URL: https://issues.apache.org/jira/browse/HBASE-16736
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
>
> Currently ResizableBlockCache only has one method for setting max size.
> As more first level cache type is added, we need the ability to retrieve the
> max size.
> This issue is to add getter to ResizableBlockCache for retrieving max size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534976#comment-15534976 ] Anoop Sam John commented on HBASE-16736: bq.long setMaxSize(); Typo. U want getMaxSize. Does it make sense to add the getter in the BC itself? Even if the cache is not resizable, getting the max capacity of a cache make sense. What is the need for this getter now? Want use in some reporting place? > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16736) Add getter to ResizableBlockCache for max size
Ted Yu created HBASE-16736: -- Summary: Add getter to ResizableBlockCache for max size Key: HBASE-16736 URL: https://issues.apache.org/jira/browse/HBASE-16736 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Currently ResizableBlockCache only has one method for setting max size. As more first level cache type is added, we need the ability to retrieve the max size. This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15535000#comment-15535000 ] Hudson commented on HBASE-15984: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #26 (See [https://builds.apache.org/job/HBase-1.3-JDK8/26/]) HBASE-15984 Handle premature EOF treatment of WALs in replication. (busbey: rev 39a79d50f1bde8ec54e08e7c249ba07562a30f63) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationWALReaderManager.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534998#comment-15534998
]
Hudson commented on HBASE-16721:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #26 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/26/])
HBASE-16721 Concurrency issue in WAL unflushed seqId tracking (enis: rev
f77f1530d4cebd1679bc1c27782bc283638dbd5f)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534997#comment-15534997
]
Hudson commented on HBASE-16732:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #26 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/26/])
HBASE-16732 Avoid possible NPE in MetaTableLocator (jerryjch: rev
728f58ad5f1e52264df58161fcbcea4ce8527a9d)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15984) Given failure to parse a given WAL that was closed cleanly, replay the WAL.
[ https://issues.apache.org/jira/browse/HBASE-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534850#comment-15534850 ] Hudson commented on HBASE-15984: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #31 (See [https://builds.apache.org/job/HBase-1.2-JDK8/31/]) HBASE-15984 Handle premature EOF treatment of WALs in replication. (busbey: rev 42dff8a58af02ec03fe97db34ab930defb79141f) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationGlobalSourceSource.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/ProtobufLogReader.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSourceImpl.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsReplicationSourceSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationWALReaderManager.java > Given failure to parse a given WAL that was closed cleanly, replay the WAL. > --- > > Key: HBASE-15984 > URL: https://issues.apache.org/jira/browse/HBASE-15984 > Project: HBase > Issue Type: Sub-task > Components: Replication >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0, 1.0.4, 1.4.0, 1.3.1, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-15984.1.patch, HBASE-15984.2.patch > > > subtask for a general work around for "underlying reader failed / is in a bad > state" just for the case where a WAL 1) was closed cleanly and 2) we can tell > that our current offset ought not be the end of parseable entries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534854#comment-15534854
]
Hudson commented on HBASE-16732:
FAILURE: Integrated in Jenkins build HBase-1.4 #438 (See
[https://builds.apache.org/job/HBase-1.4/438/])
HBASE-16732 Avoid possible NPE in MetaTableLocator (jerryjch: rev
5ac2776d2394c339f4bfee99de1150387e0d92e4)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534984#comment-15534984 ] Ted Yu commented on HBASE-16736: I found the typo as well. BucketCache doesn't extend ResizableBlockCache so I don't see the need to add the getter there. The new method can be used in reporting and in assertion(s) of tests. > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16736.v1.txt > > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16727) Backup refactoring: move MR dependencies from HMaster
[ https://issues.apache.org/jira/browse/HBASE-16727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534771#comment-15534771 ] Vladimir Rodionov commented on HBASE-16727: --- Review board https://reviews.apache.org/r/52413/ Unit test now run 70% slower (47 vs 28 min). No explanation yet. > Backup refactoring: move MR dependencies from HMaster > - > > Key: HBASE-16727 > URL: https://issues.apache.org/jira/browse/HBASE-16727 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16727-v1.patch > > > * No MR jobs in HMaster > * No proc2 implementation > * client-driven backup-restore > * basic security: only super user is allowed to run backup/restore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16736) Add getter to ResizableBlockCache for max size
[ https://issues.apache.org/jira/browse/HBASE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16736: --- Attachment: (was: 16736.v1.txt) > Add getter to ResizableBlockCache for max size > -- > > Key: HBASE-16736 > URL: https://issues.apache.org/jira/browse/HBASE-16736 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > > Currently ResizableBlockCache only has one method for setting max size. > As more first level cache type is added, we need the ability to retrieve the > max size. > This issue is to add getter to ResizableBlockCache for retrieving max size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534848#comment-15534848
]
Hudson commented on HBASE-16732:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #31 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/31/])
HBASE-16732 Avoid possible NPE in MetaTableLocator (jerryjch: rev
bfb20c0c1fa40f0580d440747a16852d2deeb78e)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/MetaTableLocator.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534849#comment-15534849
]
Hudson commented on HBASE-16721:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #31 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/31/])
HBASE-16721 Concurrency issue in WAL unflushed seqId tracking (enis: rev
cf374af102f139a6176d05b97201bfa8d9f687be)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestFSHLog.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v11.txt Patch v11 adds Spark version to hbase-spark jar filename. As for generating both jars in one go, that can be handled in separate JIRA. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533522#comment-15533522
]
Sean Busbey commented on HBASE-16179:
-
{quote}
As for generating both jars in one go, that can be handled in separate JIRA.
{quote}
we're supposed to be getting close to 2.0 alphas. If you want this in a follow
on, please retarget this change to a feature branch.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt,
> 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533576#comment-15533576
]
Ted Yu commented on HBASE-16732:
+1
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16732) Avoid possible NPE in MetaTableLocator
Jerry He created HBASE-16732:
Summary: Avoid possible NPE in MetaTableLocator
Key: HBASE-16732
URL: https://issues.apache.org/jira/browse/HBASE-16732
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Priority: Minor
{noformat}
java.lang.NullPointerException: null
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
at
org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at
org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at
org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at
org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at
org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
at
org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
at
org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
at
org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
at
com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
at
com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
{noformat}
It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533372#comment-15533372 ] Sean Busbey commented on HBASE-16179: - that would be great. I'd vastly prefer it if we could set up the build to make such jars on every build rather than rely on making our release managers hand poke them. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1551#comment-1551 ] Ted Yu commented on HBASE-16179: If Spark version is included in the filename of the jar, it would look like: hbase-spark-1.6.0-2.0.0-SNAPSHOT.jar hbase-spark-2.0.0-2.0.0-SNAPSHOT.jar > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533535#comment-15533535 ] Ted Yu commented on HBASE-16179: bq. If you want this in a follow on, please retarget this change to a feature branch. Can you clarify w.r.t. the change ? Do you prefer patch v10 for hbase 2.0 ? > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533672#comment-15533672
]
stack commented on HBASE-16721:
---
Patch is excellent. +1.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533401#comment-15533401 ] Ted Yu commented on HBASE-16179: Clarification: One build of master branch would generate only one of the above jars. A Jenkins matrix job can be setup to produce both. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533551#comment-15533551
]
Hudson commented on HBASE-16723:
FAILURE: Integrated in Jenkins build HBase-1.4 #436 (See
[https://builds.apache.org/job/HBase-1.4/436/])
HBASE-16723 RMI registry is not destroyed after stopping JMX Connector (tedyu:
rev d4b5645a4ea5674fa5be7f870d9d14d596077d40)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch,
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16725) Don't let flushThread hang in TestHRegion
[
https://issues.apache.org/jira/browse/HBASE-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533550#comment-15533550
]
Hudson commented on HBASE-16725:
FAILURE: Integrated in Jenkins build HBase-1.4 #436 (See
[https://builds.apache.org/job/HBase-1.4/436/])
HBASE-16725 Don't let flushThread hang in TestHRegion (tedyu: rev
df578592582059e08fcbbb8a0bc6b950a9768323)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Don't let flushThread hang in TestHRegion
> -
>
> Key: HBASE-16725
> URL: https://issues.apache.org/jira/browse/HBASE-16725
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Ted Yu
>Priority: Minor
> Attachments: 16725.branch-1.v1.txt, 16725.branch-1.v2.txt,
> 16725.v2.txt
>
>
> I was running TestHRegion locally and observed the following in test output:
> {code}
> 2016-09-28 16:29:36,836 INFO [main] hbase.ResourceChecker(171): after:
> regionserver.TestHRegion#testFlushCacheWhileScanning Thread=50 (was 49)
> Potentially hanging thread: FlushThread
> java.lang.Object.wait(Native Method)
> java.lang.Object.wait(Object.java:502)
>
> org.apache.hadoop.hbase.regionserver.TestHRegion$FlushThread.run(TestHRegion.java:3834)
> {code}
> Call to flushThread.done() etc should be placed in the finally block.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16721) Concurrency issue in WAL unflushed seqId tracking
[
https://issues.apache.org/jira/browse/HBASE-16721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533650#comment-15533650
]
Enis Soztutar commented on HBASE-16721:
---
bq. I'd like to see this land in time for inclusion in 1.2.4. How is it looking
for ~sunday?
The patches are ready. Need a +1 for commit actually. UT failures are not
related.
> Concurrency issue in WAL unflushed seqId tracking
> -
>
> Key: HBASE-16721
> URL: https://issues.apache.org/jira/browse/HBASE-16721
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16721_v1.branch-1.patch,
> hbase-16721_v2.branch-1.patch, hbase-16721_v2.master.patch
>
>
> I'm inspecting an interesting case where in a production cluster, some
> regionservers ends up accumulating hundreds of WAL files, even with force
> flushes going on due to max logs. This happened multiple times on the
> cluster, but not on other clusters. The cluster has periodic memstore flusher
> disabled, however, this still does not explain why the force flush of regions
> due to max limit is not working. I think the periodic memstore flusher just
> masks the underlying problem, which is why we do not see this in other
> clusters.
> The problem starts like this:
> {code}
> 2016-09-21 17:49:18,272 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=33, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-21 17:49:18,273 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> then, it continues until the RS is restarted:
> {code}
> 2016-09-23 17:43:49,356 INFO [regionserver//10.2.0.55:16020.logRoller]
> wal.FSHLog: Too many wals: logs=721, maxlogs=32; forcing flush of 1
> regions(s): d4cf39dc40ea79f5da4d0cf66d03cb1f
> 2016-09-23 17:43:49,357 WARN [regionserver//10.2.0.55:16020.logRoller]
> regionserver.LogRoller: Failed to schedule flush of
> d4cf39dc40ea79f5da4d0cf66d03cb1f, region=null, requester=null
> {code}
> The problem is that region {{d4cf39dc40ea79f5da4d0cf66d03cb1f}} is already
> split some time ago, and was able to flush its data and split without any
> problems. However, the FSHLog still thinks that there is some unflushed data
> for this region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533421#comment-15533421
]
Sean Busbey commented on HBASE-16179:
-
{quote}
Clarification:
One build of master branch would generate only one of the above jars.
A Jenkins matrix job can be setup to produce both.
{quote}
Right, this is my concern. a jenkins job doesn't help our release managers.
> Fix compilation errors when building hbase-spark against Spark 2.0
> --
>
> Key: HBASE-16179
> URL: https://issues.apache.org/jira/browse/HBASE-16179
> Project: HBase
> Issue Type: Bug
> Components: spark
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 2.0.0
>
> Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt,
> 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt
>
>
> I tried building hbase-spark module against Spark-2.0 snapshot and got the
> following compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer
> accessible to downstream projects.
> hbase-spark module should not depend on such classes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533774#comment-15533774 ] stack commented on HBASE-16264: --- Hmmm... I can't attach my patch. It is too big for JIRA. 21MB. Let me fix. > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, > HBASE-16264-Figure-how-to-deal-with-endpoints-and-sh.master.013.patch, > HBASE-16264.master.001.patch, HBASE-16264.master.002.patch, > HBASE-16264.master.003.patch, HBASE-16264.master.004.patch, > HBASE-16264.master.005.patch, HBASE-16264.master.006.patch, > HBASE-16264.master.007.patch, HBASE-16264.master.008.patch, > HBASE-16264.master.009.patch, HBASE-16264.master.010.patch, > HBASE-16264.master.011.patch, HBASE-16264.master.012.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533704#comment-15533704 ] Sean Busbey commented on HBASE-16179: - That's the advantage of a feature branch. We can wait until we find someone who is interested in doing that part of the work before we impact downstream folks. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533785#comment-15533785
]
Hadoop QA commented on HBASE-16732:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 23s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 58s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 3s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830951/HBASE-16732-v2.patch |
| JIRA Issue | HBASE-16732 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d39b61da5f7a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7639671 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3772/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3772/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
[jira] [Created] (HBASE-16733) add hadoop 3.0.0-alpha1 to precommit checks
Jonathan Hsieh created HBASE-16733: -- Summary: add hadoop 3.0.0-alpha1 to precommit checks Key: HBASE-16733 URL: https://issues.apache.org/jira/browse/HBASE-16733 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jonathan Hsieh Fix For: 2.0.0 Been working on getting hadoop3 related build up and running and woudl ike to add a precommit check so that new commits don't break the mvn compile/install. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16567) Upgrade to protobuf-3.1.x
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534177#comment-15534177 ] stack commented on HBASE-16567: --- pb3.1.0 just went out. On top of the current set of unsafe additions added in pb3, they have added: bq. Experimental API: Added UnsafeByteOperations.unsafeWrap(byte[]) to wrap a byte array into ByteString without copy. This latter makes it so we can purge our HBaseZeroCopyByteString class. > Upgrade to protobuf-3.1.x > - > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- though read-time is still a TODO (this means pb3 is not > enough; we'll have to patch it -- or patch pb2.5). > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run against our .protos works fine except pb3 breaks our > HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly > recast using new pb3 types) > * Starting up a cluster that is all pb3 seems to work fine. > * A pb2 branch-1 can read and write against the pb3 master cluster. > What will break if we just upgrade to pb3? > * We should be able to write HDFS messages on our AsyncWAL using pb3; the > pb2 HDFS should be able to read them (not tested). Or maybe not. See policy > here: https://github.com/google/protobuf/issues/1852 which seems to indicate > pb3s will not be able to write compatible pb2 Messages. TODO. > * Core Coprocessor Endpoints such as AccessControl seem to just work (their > protos will have been protoc3'd). I did simple test with a server from master > branch up on pb3 and then going against it with a branch-1 client on pb2. I > was able to add grants. > * For non-core CPEPs where the protos are pb2 still, it might just work. To > test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16733) add hadoop 3.0.0-alpha1 to precommit checks
[ https://issues.apache.org/jira/browse/HBASE-16733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-16733: --- Attachment: hbase-16733.v0.patch v0 - testing via submitting a patch. > add hadoop 3.0.0-alpha1 to precommit checks > --- > > Key: HBASE-16733 > URL: https://issues.apache.org/jira/browse/HBASE-16733 > Project: HBase > Issue Type: Bug > Components: build >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: hbase-16733.v0.patch > > > Been working on getting hadoop3 related build up and running and woudl ike to > add a precommit check so that new commits don't break the mvn compile/install. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[
https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534202#comment-15534202
]
Hadoop QA commented on HBASE-16179:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 9s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 1m
22s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 4m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 8s
{color} | {color:red} root generated 1 new + 19 unchanged - 1 fixed = 20 total
(was 20) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 11s
{color} | {color:red} hbase-spark generated 1 new + 17 unchanged - 1 fixed = 18
total (was 18) {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 1m 1s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} scaladoc {color} | {color:red} 0m 27s
{color} | {color:red} hbase-spark in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s
{color} | {color:green} hbase-spark2.0-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s
{color} | {color:green} hbase-spark1.6-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 29s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 12s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
50s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 156m 55s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
[jira] [Updated] (HBASE-16567) Upgrade to protobuf-3.1.x
[ https://issues.apache.org/jira/browse/HBASE-16567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16567: -- Attachment: 16567.patch Patch goes in nicely over the top of HBASE-16264 Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString > Upgrade to protobuf-3.1.x > - > > Key: HBASE-16567 > URL: https://issues.apache.org/jira/browse/HBASE-16567 > Project: HBase > Issue Type: Task > Components: Protobufs >Affects Versions: 2.0.0 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: 16567.patch, HBASE-16567.master.001.patch > > > Move master branch on to protobuf3. See > https://github.com/google/protobuf/releases We'd do it because pb3 saves some > on byte copies can work with offheap buffers -- needed for the off-heap write > path project -- though read-time is still a TODO (this means pb3 is not > enough; we'll have to patch it -- or patch pb2.5). > HBASE-15638 has us first shading protobufs before upgrading. Let us list here > issues just going to pb3 without shading if only for completeness sake; i.e. > do we have to shade? > * pb3 is by default wire compatible with pb2. > * protoc3 run against our .protos works fine except pb3 breaks our > HBaseZeroCopyLiteralByteString hack so this has to be removed (possibly > recast using new pb3 types) > * Starting up a cluster that is all pb3 seems to work fine. > * A pb2 branch-1 can read and write against the pb3 master cluster. > What will break if we just upgrade to pb3? > * We should be able to write HDFS messages on our AsyncWAL using pb3; the > pb2 HDFS should be able to read them (not tested). Or maybe not. See policy > here: https://github.com/google/protobuf/issues/1852 which seems to indicate > pb3s will not be able to write compatible pb2 Messages. TODO. > * Core Coprocessor Endpoints such as AccessControl seem to just work (their > protos will have been protoc3'd). I did simple test with a server from master > branch up on pb3 and then going against it with a branch-1 client on pb2. I > was able to add grants. > * For non-core CPEPs where the protos are pb2 still, it might just work. To > test. It would not be the end-of-the-world if they did not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16653) Backport HBASE-11393 to all branches which support namespace
[ https://issues.apache.org/jira/browse/HBASE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534204#comment-15534204 ] Enis Soztutar commented on HBASE-16653: --- Thanks for taking on this work. How different is the patch for branch-1 vs the master patch? In branch-1 and below, we cannot remove deprecated methods for public interfaces or interfaces exposed to the ReplicationEndpoints (like ReplicationPeerConfig, etc). So, the ReplicationAdmin changes should be undone. [~chenheng] you want to take a look at this? > Backport HBASE-11393 to all branches which support namespace > > > Key: HBASE-16653 > URL: https://issues.apache.org/jira/browse/HBASE-16653 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.0.5, 1.3.1, 0.98.22, 1.1.7, 1.2.4 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16653-branch-1-v1.patch, > HBASE-16653-branch-1-v2.patch > > > As HBASE-11386 mentioned, the parse code about replication table-cfs config > will be wrong when table name contains namespace and we can only config the > default namespace's tables in the peer. It is a bug for all branches which > support namespace. HBASE-11393 resolved this by use a pb object but it was > only merged to master branch. Other branches still have this problem. I > thought we should fix this bug in all branches which support namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15534053#comment-15534053 ] Sean Busbey commented on HBASE-16712: - hurm. looking at the logs, some stuff looks messed up because of the SNAPSHOT version. Any reason we can't switch from 3.0.0-SNAPSHOT to 3.0.0-alpha1? > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16732:
-
Fix Version/s: (was: 1.2.4)
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16732:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the review, Ted.
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16732:
-
Issue Type: Bug (was: Improvement)
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Bug
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533693#comment-15533693 ] Ted Yu commented on HBASE-16179: I don't plan to work on that (I just follow the current practice). > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16264) Figure how to deal with endpoints and shaded pb
[ https://issues.apache.org/jira/browse/HBASE-16264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533855#comment-15533855 ] stack commented on HBASE-16264: --- Pushed a dev branch HBASE-16264 Added a jenkins job HBASE-16264 > Figure how to deal with endpoints and shaded pb > --- > > Key: HBASE-16264 > URL: https://issues.apache.org/jira/browse/HBASE-16264 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors, Protobufs >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 16264.tactic2.patch, > HBASE-16264-Figure-how-to-deal-with-endpoints-and-sh.master.013.patch, > HBASE-16264.master.001.patch, HBASE-16264.master.002.patch, > HBASE-16264.master.003.patch, HBASE-16264.master.004.patch, > HBASE-16264.master.005.patch, HBASE-16264.master.006.patch, > HBASE-16264.master.007.patch, HBASE-16264.master.008.patch, > HBASE-16264.master.009.patch, HBASE-16264.master.010.patch, > HBASE-16264.master.011.patch, HBASE-16264.master.012.patch > > > Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. > Would be sweet if could make it so all just worked. At worst, come up w/ a > prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16712) fix hadoop-3.0 profile mvn install
[ https://issues.apache.org/jira/browse/HBASE-16712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533939#comment-15533939 ] Jonathan Hsieh commented on HBASE-16712: [~busbey] check out the last commit here -- I've included the h?-deps and a log file with a -X of a 'mvn install -Dhadoop.profile=3.0. https://github.com/jmhsieh/hbase/commits/hbase-16712 > fix hadoop-3.0 profile mvn install > -- > > Key: HBASE-16712 > URL: https://issues.apache.org/jira/browse/HBASE-16712 > Project: HBase > Issue Type: Bug > Components: build, hadoop3 >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: 2.0.0 > > Attachments: h2-deps, h3-deps, hbase-16712.v0.patch, > hbase-16712.v1.patch > > > After the compile is fixed, mvn install fails due to transitive dependencies > coming from hadoop3. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-16732:
-
Attachment: HBASE-16732-v2.patch
v2 updates with Ted's comment.
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
> Attachments: HBASE-16732-v2.patch, HBASE-16732.patch
>
>
> {noformat}
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.getMetaReplicaNodes(ZooKeeperWatcher.java:489)
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:549)
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1211)
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1178)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:305)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:156)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
> at
> org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
> at
> org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
> at
> org.apache.hadoop.hbase.client.ClientScanner.(ClientScanner.java:155)
> at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:804)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:602)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:366)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:396)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseAdmin1_2.clearTable(HBaseAdmin1_2.java:38)
> at
> com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.clearStorage(HBaseStoreManager.java:483)
> {noformat}
> It happens when hbase is not fully up, and the client comes in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533686#comment-15533686 ] Sean Busbey commented on HBASE-16179: - if you'd like "generate both downstream facing jars for spark use during a single normal build" to be a follow on, then I'd like the work on this jira to go into a feature branch until that follow on is complete. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug > Components: spark >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 2.0.0 > > Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, > 16179.v11.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, > 16179.v9.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16732) Avoid possible NPE in MetaTableLocator
[
https://issues.apache.org/jira/browse/HBASE-16732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15533956#comment-15533956
]
Hadoop QA commented on HBASE-16732:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 2s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12830951/HBASE-16732-v2.patch |
| JIRA Issue | HBASE-16732 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d1535241d3a2 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 7639671 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3774/testReport/ |
| modules | C: hbase-client U: hbase-client |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3774/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Avoid possible NPE in MetaTableLocator
> --
>
> Key: HBASE-16732
> URL: https://issues.apache.org/jira/browse/HBASE-16732
> Project: HBase
> Issue Type: Improvement
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4
>
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[
https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531893#comment-15531893
]
ramkrishna.s.vasudevan commented on HBASE-16608:
Yet to fully understand the issue, but am very sure it is because of the merge
and snapshot that happens parallely. Previously there was a wait() that got
removed.
How does merge and snapshot work together? Can you check that part once again.
{code}
org.apache.hadoop.hbase.DroppedSnapshotException: region:
TestTable,0010695414,1475147500030.2c0a4161a4462d9070921ceb0fe22390.
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2538)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2223)
at
org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2192)
at
org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2083)
at org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:2009)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:502)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:472)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.IllegalArgumentException: offset (808910118) + length (8)
exceed the capacity of the array: 2097152
at
org.apache.hadoop.hbase.util.Bytes.explainWrongLengthOrOffset(Bytes.java:840)
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:814)
at org.apache.hadoop.hbase.util.Bytes.toLong(Bytes.java:799)
at org.apache.hadoop.hbase.KeyValue.getTimestamp(KeyValue.java:1511)
at org.apache.hadoop.hbase.KeyValue.getTimestamp(KeyValue.java:1502)
at
org.apache.hadoop.hbase.regionserver.querymatcher.ScanQueryMatcher.preCheck(ScanQueryMatcher.java:192)
at
org.apache.hadoop.hbase.regionserver.querymatcher.MinorCompactionScanQueryMatcher.match(MinorCompactionScanQueryMatcher.java:40)
at
org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:564)
at
org.apache.hadoop.hbase.regionserver.StoreFlusher.performFlush(StoreFlusher.java:132)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:75)
at
org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:880)
{code}
Able to get this quite easily. I infact tried running with no inmemory
compaction and I did not get this issue. I am still in the process of debugging
will get back here once I am clear on how this is caused. Just updating in case
you have time to see the problem.
Also addIntoPooledChunks now adds from all the segments into one segment. From
each segment there is a chance the pooledChunk is full and you add all those
full chunkQueue to another chunkQueue which at the max can hold
chunkPool.getMaxCount().
> Introducing the ability to merge ImmutableSegments without copy-compaction or
> SQM usage
> ---
>
> Key: HBASE-16608
> URL: https://issues.apache.org/jira/browse/HBASE-16608
> Project: HBase
> Issue Type: Sub-task
>Reporter: Anastasia Braginsky
>Assignee: Anastasia Braginsky
> Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch,
> HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch,
> HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch,
> HBASE-16608-V04.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531951#comment-15531951 ] ramkrishna.s.vasudevan commented on HBASE-16608: The queue return has got nothing to do with the issue that I agree. Am just asking how is that logic working there. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531944#comment-15531944 ] Anoop Sam John commented on HBASE-16608: Thanks for testing Ram. bq.Also addIntoPooledChunks now adds from all the segments into one segment. From each segment there is a chance the pooledChunk is full and you add all those full chunkQueue to another chunkQueue which at the max can hold chunkPool.getMaxCount(). Ya this is doing nothing with the pool as such. Each of the merged segments might have N pooled chunks in its Q. Now we have merged all these segments into one. We dont refer to those segments any more. So when the result segment is getting closed some time later, we will have to release all of these chunks to pool.. This op just put all PooledChunks from all these segments into one Q. That is the Q in the result Segment. I dont think some thing wrong there. > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15532138#comment-15532138 ] Anastasia Braginsky commented on HBASE-16608: - Hey Guys! I see we have the problematic exception there. First, I am going to try to recreate and resolve this issue. This have nothing to do with the memory reclamation. Second, I will publish some document explaining how the memory reclamation works together with merge. Hope to get back to you soon... > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-V01.patch, HBASE-16608-V03.patch, > HBASE-16608-V04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16723) RMI registry is not destroyed after stopping JMX Connector Server
[
https://issues.apache.org/jira/browse/HBASE-16723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16723:
---
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
> RMI registry is not destroyed after stopping JMX Connector Server
> -
>
> Key: HBASE-16723
> URL: https://issues.apache.org/jira/browse/HBASE-16723
> Project: HBase
> Issue Type: Bug
> Components: metrics
>Affects Versions: 2.0.0, 1.4.0, 1.3.1, 1.2.3, 0.98.22
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16723-V2.patch, HBASE-16723-V3.patch,
> HBASE-16723.patch
>
>
> We are creating RMI registry in JMXListener.startConnectorServer() ,
> {code}
> // Create the RMI registry
> LocateRegistry.createRegistry(rmiRegistryPort);
> {code}
> This registry is never deregistered, should be destoyed after stopping JMX
> Connector server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
