[jira] [Commented] (HBASE-15158) Change order in which we do write pipeline operations; do all under row locks!
[
https://issues.apache.org/jira/browse/HBASE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136686#comment-15136686
]
Anoop Sam John commented on HBASE-15158:
Great ! Now we can get rid of Memstore rollback stuff..:-)
> Change order in which we do write pipeline operations; do all under row locks!
> --
>
> Key: HBASE-15158
> URL: https://issues.apache.org/jira/browse/HBASE-15158
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15158.patch, 15158v2.patch, 15158v3.patch,
> 15158v4.patch, 15158v4.patch, measurements.tgz
>
>
> Change how we do our write pipeline. I want to do all write pipeline ops
> under row lock so I lean on this fact fixing performance regression in
> check-and-set type operations like increment, append, and checkAnd* (see
> sibling issue HBASE-15082).
> To be specific, we write like this now:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # add to memstore
> # let go of rowlock
> # sync WAL
> # in case of error: rollback memstore
> {code}
> Instead, write like this:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # sync WAL
> # add to memstore
> # let go of rowlock
> ... no need to do rollback.
> {code}
> The old ordering was put in place because it got better performance in a time
> when WAL was different and before row locks were read/write (HBASE-12751).
> Testing in branch-1 shows that a reordering and skipping mvcc waits gets us
> back to the performance we had before we unified mvcc and sequenceid
> (HBASE-8763). Tests in HBASE-15046 show that at the macro level using our
> usual perf tools, reordering pipeline seems to cause no slowdown (see
> HBASE-15046). A rough compare of increments with reordered write pipeline
> seems to have us getting back a bunch of our performance (see tail of
> https://issues.apache.org/jira/browse/HBASE-15082?focusedCommentId=15111703=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15111703
> and subsequent comment).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137184#comment-15137184
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 13s
{color} | {color:red} Patch generated 1 new checkstyle issues in hbase-server
(total was 216, now 216). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 25s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 6s
{color} | {color:red} hbase-server introduced 1 new FindBugs issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 99m 24s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 103m 38s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 247m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Dead store to walEdit in
org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation)
At
HRegion.java:org.apache.hadoop.hbase.regionserver.HRegion.doMiniBatchMutate(HRegion$BatchOperation)
At HRegion.java:[line 2954] |
\\
\\
|| Subsystem ||
[jira] [Commented] (HBASE-15219) Canary tool does not return non-zero exit when one of region stuck state
[
https://issues.apache.org/jira/browse/HBASE-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137071#comment-15137071
]
Vishal Khandelwal commented on HBASE-15219:
---
If we all the errors at once then multiple iterations can be avoided.
Corrective action can be taken tp fix HW/SW.
> Canary tool does not return non-zero exit when one of region stuck state
> -
>
> Key: HBASE-15219
> URL: https://issues.apache.org/jira/browse/HBASE-15219
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15219.v1.patch, HBASE-15219.v3.patch,
> HBASE-15219.v4.patch
>
>
> {code}
> 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from
> region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> column family 0 failed
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Fri Feb 05 12:24:15 GMT 2016,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> -bash-4.1$ echo $?
> 0
> {code}
> Below code prints the error but it does sets/returns the exit code. Due to
> this tool can't be integrated with nagios or other alerting.
> Ideally it should return error for failures. as pre the documentation:
>
> This tool will return non zero error codes to user for collaborating with
> other monitoring tools, such as Nagios. The error code definitions are:
> private static final int USAGE_EXIT_CODE = 1;
> private static final int INIT_ERROR_EXIT_CODE = 2;
> private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
> private static final int ERROR_EXIT_CODE = 4;
>
> {code}
> org.apache.hadoop.hbase.tool.Canary.RegionTask
> public Void read() {
>
> try {
> table = connection.getTable(region.getTable());
> tableDesc = table.getTableDescriptor();
> } catch (IOException e) {
> LOG.debug("sniffRegion failed", e);
> sink.publishReadFailure(region, e);
>...
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137141#comment-15137141
]
Sean Busbey commented on HBASE-15122:
-
I kicked off a new precommit test run.
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137172#comment-15137172 ] Josh Elser commented on HBASE-15221: I'm tentatively dropping the 98.x fixVersion. It looks like this was already (attempted to be?) fixed in HBASE-12198. There's a clearCaches call in HTableMultiplexer in 0.98 that isn't present in the newer branches. Sadly, there *is* a call in AsyncProcess which looks like it should be triggering a location cache clear. I need to step back again and look at this some more... > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser updated HBASE-15221: --- Fix Version/s: (was: 0.98.18) > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137104#comment-15137104 ] Ted Yu commented on HBASE-15211: For a cluster with many tables, there may be region(s) constantly in transition. I think we can relax the constraint a bit: CatalogJanitor can run for table none of whose regions is in transition. What do you think ? > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Status: Patch Available (was: Open)
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
> Attachments: HBASE-15229.v1.patch
>
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136809#comment-15136809 ] Hudson commented on HBASE-15221: SUCCESS: Integrated in HBase-1.3-IT #487 (See [https://builds.apache.org/job/HBase-1.3-IT/487/]) HBASE-15221 Reload the cache on re-tried puts in HTableMultiplexer and (busbey: rev 01b73e9877ccac6faed941fc83351955b642fc09) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerViaMocks.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * hbase-client/pom.xml > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.18 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136853#comment-15136853 ] Hudson commented on HBASE-15221: FAILURE: Integrated in HBase-Trunk_matrix #690 (See [https://builds.apache.org/job/HBase-Trunk_matrix/690/]) HBASE-15221 Reload the cache on re-tried puts in HTableMultiplexer and (busbey: rev dfd8a31a130ffdad382a5a6923035b1142ccdb0c) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java * hbase-client/pom.xml * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerViaMocks.java > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.18 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[ https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Khandelwal updated HBASE-15229: -- Description: run method in canary tool calls system.Exit on failure. Due to this it can't be integrated as an API as jvm will stop on failure. It should only return (exit code) was:run method in canary tool calls system.Exit on failure. Due to this it can't be integrated as an API as jvm will stop on failure. It should only return (exit code) > Canary Tools should not call System.Exit on error > - > > Key: HBASE-15229 > URL: https://issues.apache.org/jira/browse/HBASE-15229 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 0.98.17 >Reporter: Vishal Khandelwal >Assignee: Vishal Khandelwal >Priority: Critical > > run method in canary tool calls system.Exit on failure. Due to this it can't > be integrated as an API as jvm will stop on failure. It should only return > (exit code) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Description:
run method in canary tool calls system.Exit on failure. Due to this it can't be
integrated as an API as jvm will stop on failure. It should only return (exit
code)
So any integration with unit test also fail with
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
project <>: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked VM
terminated without saying properly goodbye. VM crash or System.exit called ?
{code}
was:
run method in canary tool calls system.Exit on failure. Due to this it can't be
integrated as an API as jvm will stop on failure. It should only return (exit
code)
So any integration with unit test also fail with
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
project <>: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked VM
terminated without saying properly goodbye. VM crash or System.exit called ?
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Description:
run method in canary tool calls system.Exit on failure. Due to this it can't be
integrated as an API as jvm will stop on failure. It should only return (exit
code)
So any integration with unit test also fail with
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
project <>: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked VM
terminated without saying properly goodbye. VM crash or System.exit called ?
was:
run method in canary tool calls system.Exit on failure. Due to this it can't be
integrated as an API as jvm will stop on failure. It should only return (exit
code)
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136842#comment-15136842
]
Hadoop QA commented on HBASE-15198:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 5m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 51s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 28s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 13s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 105m 58s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 286m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.client.TestAdmin1 |
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster |
| |
[jira] [Commented] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[
https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136852#comment-15136852
]
Hadoop QA commented on HBASE-15204:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 3s {color}
| {color:red} HBASE-15204 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/latest/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786792/HBASE-15204_3.patch |
| JIRA Issue | HBASE-15204 |
| Powered by | Apache Yetus 0.1.0 http://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/476/console |
This message was automatically generated.
> Try to estimate the cell count for adding into WALEdit
> --
>
> Key: HBASE-15204
> URL: https://issues.apache.org/jira/browse/HBASE-15204
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0, 1.3.0
>
> Attachments: HBASE-15204.patch, HBASE-15204_1.patch,
> HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch,
> WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg
>
>
> The write path profiling shows that when we try to add Cells to WALEdits we
> try to do a lot of Array copy inorder to grow the Arraylist backing the
> WALEdits. In a simple one min profiling of the write path with 50 YCSB
> threads shows around 261MB of allocation done for the Array copy to happen.
> We can try to avoid that.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136795#comment-15136795 ] Hudson commented on HBASE-15221: SUCCESS: Integrated in HBase-1.3 #544 (See [https://builds.apache.org/job/HBase-1.3/544/]) HBASE-15221 Reload the cache on re-tried puts in HTableMultiplexer and (busbey: rev 01b73e9877ccac6faed941fc83351955b642fc09) * hbase-client/pom.xml * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerViaMocks.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.18 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15158) Change order in which we do write pipeline operations; do all under row locks!
[
https://issues.apache.org/jira/browse/HBASE-15158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136854#comment-15136854
]
Hudson commented on HBASE-15158:
FAILURE: Integrated in HBase-Trunk_matrix #690 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/690/])
HBASE-15158 Change order in which we do write pipeline operations; do (stack:
rev ec92a8a705dfec076a93454e1042645d466758f0)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSWALEntry.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALKey.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALUtil.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogKey.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Change order in which we do write pipeline operations; do all under row locks!
> --
>
> Key: HBASE-15158
> URL: https://issues.apache.org/jira/browse/HBASE-15158
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: stack
>Assignee: stack
> Fix For: 2.0.0
>
> Attachments: 15158.patch, 15158v2.patch, 15158v3.patch,
> 15158v4.patch, 15158v4.patch, measurements.tgz
>
>
> Change how we do our write pipeline. I want to do all write pipeline ops
> under row lock so I lean on this fact fixing performance regression in
> check-and-set type operations like increment, append, and checkAnd* (see
> sibling issue HBASE-15082).
> To be specific, we write like this now:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # add to memstore
> # let go of rowlock
> # sync WAL
> # in case of error: rollback memstore
> {code}
> Instead, write like this:
> {code}
> # take rowlock
> # start mvcc
> # append to WAL
> # sync WAL
> # add to memstore
> # let go of rowlock
> ... no need to do rollback.
> {code}
> The old ordering was put in place because it got better performance in a time
> when WAL was different and before row locks were read/write (HBASE-12751).
> Testing in branch-1 shows that a reordering and skipping mvcc waits gets us
> back to the performance we had before we unified mvcc and sequenceid
> (HBASE-8763). Tests in HBASE-15046 show that at the macro level using our
> usual perf tools, reordering pipeline seems to cause no slowdown (see
> HBASE-15046). A rough compare of increments with reordered write pipeline
> seems to have us getting back a bunch of our performance (see tail of
> https://issues.apache.org/jira/browse/HBASE-15082?focusedCommentId=15111703=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15111703
> and subsequent comment).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_3.patch Rebased patch. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > HBASE-15204_3.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14025) Update CHANGES.txt for 1.2
[ https://issues.apache.org/jira/browse/HBASE-14025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136768#comment-15136768 ] Hudson commented on HBASE-14025: SUCCESS: Integrated in HBase-1.2 #534 (See [https://builds.apache.org/job/HBase-1.2/534/]) HBASE-14025 update CHANGES.txt for 1.2 RC2 (busbey: rev d568db8372a3fbc6b93c5854749c30276ba19ba4) * CHANGES.txt > Update CHANGES.txt for 1.2 > -- > > Key: HBASE-14025 > URL: https://issues.apache.org/jira/browse/HBASE-14025 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 1.2.0 >Reporter: Sean Busbey >Assignee: Sean Busbey > Fix For: 1.2.0 > > > Since it's more effort than I expected, making a ticket to track actually > updating CHANGES.txt so that new RMs have an idea what to expect. > Maybe will make doc changes if there's enough here. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136767#comment-15136767 ] Hudson commented on HBASE-15221: SUCCESS: Integrated in HBase-1.2 #534 (See [https://builds.apache.org/job/HBase-1.2/534/]) HBASE-15221 Reload the cache on re-tried puts in HTableMultiplexer and (busbey: rev 25878e1a97ab3d1009ecf799bfa5dabb205ee2fc) * hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestHTableMultiplexerViaMocks.java * hbase-client/pom.xml * hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableMultiplexer.java > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.18 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[
https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136811#comment-15136811
]
Duo Zhang commented on HBASE-14949:
---
{quote}
It will be better if we use a switch which store in System Table, after upgrade
finished, we can switch on manually to use new WAL logic.
{quote}
This is a good idea but how do we know upgrade is finished? For HDFS, there is
a rolling upgrade state. We call "hdfs dfsadmin -rollingUpgrade prepare" when
starting and call "hdfs dfsadmin -rollingUpgrade finalize" when done. Is there
a similar tool in HBase? I think our rolling upgrade is just kill the old
server and restart it with the new code...
Thanks.
> Skip duplicate entries when replay WAL.
> ---
>
> Key: HBASE-14949
> URL: https://issues.apache.org/jira/browse/HBASE-14949
> Project: HBase
> Issue Type: Sub-task
>Reporter: Heng Chen
> Attachments: HBASE-14949.patch, HBASE-14949_v1.patch,
> HBASE-14949_v2.patch
>
>
> As HBASE-14004 design, there will be duplicate entries in different WAL. It
> happens when one hflush failed, we will close old WAL with 'acked hflushed'
> length, then open a new WAL and write the unacked hlushed entries into it.
> So there maybe some overlap between old WAL and new WAL.
> We should skip the duplicate entries when replay. I think it has no harm to
> current logic, maybe we do it first.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Patch Available (was: Open) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Attachment: HBASE-15204_3.patch Attaching patch addressing comments. > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, HBASE-15204_3.patch, > WAlEdit_add_allocation.jpg, WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
[
https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136872#comment-15136872
]
ramkrishna.s.vasudevan commented on HBASE-15205:
Let me tell with the code here. Since replication is enabled by default. For
every append we try to scope the WALEdits with the replication scope. So even
if there is one CF and there is no replication at all enabled even then we try
to iterate and find the scope associated with that CF for every append.
{code}
} else {
family = CellUtil.cloneFamily(cell);
// Unexpected, has a tendency to happen in unit tests
assert htd.getFamily(family) != null;
if (!scopes.containsKey(family)) {
int scope = htd.getFamily(family).getScope();
if (scope != REPLICATION_SCOPE_LOCAL) {
scopes.put(family, scope);
}
}
{code}
This code ' int scope = htd.getFamily(family).getScope();' generates lot of
garbage as we do some new String() operation.
In case of Multi Cf case in this same piece of code we define a local map
{code}
NavigableMap scopes = new TreeMap(Bytes.BYTES_COMPARATOR);
{code}
to which we copy all the CFs and their scopes which has NON default scope
associated. So for every append we iterate thro all the cells, find the scope
of each CF in the cell (if it is not already added to the 'scopes') map. This
map is then serialized in the 'pb'.
The above logic for multiCF makes sense because if among all the cF if only one
is with GLOBAL scope then only that information is added to that WALKey.
So first thing that we can avoid is reduce the garbage created by doing this
new String() by actually getting the scope once in the HRegion and use that in
the append(). This avoids all the garbage created by new String() and the UTF8
encoder for every append.
But the other thing is that if we can just add all the non default CF and
serialize it for every WAL key we can even avoid the local map getting created
and the check that we perform on these maps etc. But at the cost of serializing
more information per WAL.
> Do not find the replication scope for every WAL#append()
>
>
> Key: HBASE-15205
> URL: https://issues.apache.org/jira/browse/HBASE-15205
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-15205.patch, ScopeWALEdits.jpg,
> ScopeWALEdits_afterpatch.jpg
>
>
> After the byte[] and char[] the other top contributor for lot of GC (though
> it is only 2.86%) is the UTF_8.newDecoder.
> This happens because for every WAL append we try to calculate the replication
> scope associate with the families associated with the TableDescriptor. I
> think per WAL append doing this is very costly and creates lot of garbage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15229) Canary Tools should not call System.Exit on error
Vishal Khandelwal created HBASE-15229: - Summary: Canary Tools should not call System.Exit on error Key: HBASE-15229 URL: https://issues.apache.org/jira/browse/HBASE-15229 Project: HBase Issue Type: Bug Components: canary Affects Versions: 0.98.17 Reporter: Vishal Khandelwal Assignee: Vishal Khandelwal Priority: Critical run method in canary tool calls system.Exit on failure. Due to this it can't be integrated as an API as jvm will stop on failure. It should only return (exit code) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15204) Try to estimate the cell count for adding into WALEdit
[ https://issues.apache.org/jira/browse/HBASE-15204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-15204: --- Status: Open (was: Patch Available) > Try to estimate the cell count for adding into WALEdit > -- > > Key: HBASE-15204 > URL: https://issues.apache.org/jira/browse/HBASE-15204 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15204.patch, HBASE-15204_1.patch, > HBASE-15204_1.patch, HBASE-15204_2.patch, WAlEdit_add_allocation.jpg, > WAlEdit_add_allocation_after_patch.jpg > > > The write path profiling shows that when we try to add Cells to WALEdits we > try to do a lot of Array copy inorder to grow the Arraylist backing the > WALEdits. In a simple one min profiling of the write path with 50 YCSB > threads shows around 261MB of allocation done for the Array copy to happen. > We can try to avoid that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15228) Add the methods to RegionObserver to trigger start/complete restoring WALs
[ https://issues.apache.org/jira/browse/HBASE-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136845#comment-15136845 ] Anoop Sam John commented on HBASE-15228: We have some CP hooks around restore WAL already right? > Add the methods to RegionObserver to trigger start/complete restoring WALs > -- > > Key: HBASE-15228 > URL: https://issues.apache.org/jira/browse/HBASE-15228 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki > Attachments: HBASE-15228.patch > > > In our use case, we write indexes to a index table when restoring WALs. We > want to trigger start/complete restoring WALs for initial and end processing > for writing indexes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Attachment: HBASE-15229.v1.patch
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
> Attachments: HBASE-15229.v1.patch
>
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138353#comment-15138353 ] Vladimir Rodionov commented on HBASE-14030: --- Fixed, thanks, Ted. v35 in RB. > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: (was: HBASE-15229.v3.patch)
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: (was: HBASE-15216.v2.patch)
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Attachment: HBASE-15229.v3.patch
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
> Attachments: HBASE-15229.v1.patch, HBASE-15229.v2.patch,
> HBASE-15229.v3.patch
>
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15232) Exceptions returned over multi RPC don't automatically trigger region location reloads
[
https://issues.apache.org/jira/browse/HBASE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-15232:
---
Attachment: (was: HBASE-15232.001.patch)
> Exceptions returned over multi RPC don't automatically trigger region
> location reloads
> --
>
> Key: HBASE-15232
> URL: https://issues.apache.org/jira/browse/HBASE-15232
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
>
> Attachments: HBASE-15232.001.patch
>
>
> Follow-on for HBASE-15221:
> A work-around was added in HTableMultiplexer to work around an issue that
> AsyncProcess wasn't clearing the region location cache on Exception. This was
> stemming from the issue that the {{tableName}} is {{null}} because
> HTableMultiplexer is using the {{multi}} RPC. This causes an error that looks
> like:
> {noformat}
> [WARN] Coding error, see method javadoc. row=[B@1673eff, tableName=null
> {noformat}
> HBASE-15221 should fix HTableMultiplexer, but it would be good to push the
> fix down into AsyncProcess instead of using higher-level workarounds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
[ https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138363#comment-15138363 ] ramkrishna.s.vasudevan commented on HBASE-15205: bq.When then is the scope checked? When we go to replicate after reading the WAL? We'd be trading CPU for i/o? Actually in the current code we check the scope both on writes and reads. So when we try to replicate we once again try to iterate the scope that was persisted and check against the families associated with every cell and only then decide on which one to be replicated. bq.We'd be trading CPU for i/o? Yes but on the write path only. But now may be more data is written per WAL Key for a multiCF case. > Do not find the replication scope for every WAL#append() > > > Key: HBASE-15205 > URL: https://issues.apache.org/jira/browse/HBASE-15205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15205.patch, ScopeWALEdits.jpg, > ScopeWALEdits_afterpatch.jpg > > > After the byte[] and char[] the other top contributor for lot of GC (though > it is only 2.86%) is the UTF_8.newDecoder. > This happens because for every WAL append we try to calculate the replication > scope associate with the families associated with the TableDescriptor. I > think per WAL append doing this is very costly and creates lot of garbage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[ https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138316#comment-15138316 ] ramkrishna.s.vasudevan commented on HBASE-15236: bq.In such a case, depending on file size (because we take it into account when sorting hfiles internally), Where do we do this? > Inconsistent cell reads over multiple bulk-loaded HFiles > > > Key: HBASE-15236 > URL: https://issues.apache.org/jira/browse/HBASE-15236 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > > If there are two bulkloaded hfiles in a region with same seqID and duplicate > keys*, get and scan may return different values for a key. > More details: > - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' > and column qualifiers are 1 to 1000. > - hfiles were split somewhere along that row, but there were a range of > columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and > hfile2: [r:cf:40, ). > - Between columns 40 to 70, some (not all) columns were in both the files > with different values. Whereas other were only in one of the files. > In such a case, depending on file size (because we take it into account when > sorting hfiles internally), we may get different values for the same cell > (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" > "cf:". > I have been able to replicate this issue, will post the instructions shortly. > * not sure how this would happen. These files are small ~50M, nor could i > find any setting for max file size that could lead to splits. Need to > investigate more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15237) NumberFormatException in create table if HBase version is of form X.Y.Z-BBB
[
https://issues.apache.org/jira/browse/HBASE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138319#comment-15138319
]
Enis Soztutar commented on HBASE-15237:
---
According to semver, the version can be of form {{1.0.0-alpha+001}}.
> NumberFormatException in create table if HBase version is of form X.Y.Z-BBB
> ---
>
> Key: HBASE-15237
> URL: https://issues.apache.org/jira/browse/HBASE-15237
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Just ran into this while testing with a custom build with made up version
> {{1.1.1-dal}}
> {code}
> 2016-02-08 20:10:38,494 ERROR
> [B.defaultRpcServer.handler=1,queue=1,port=52622] ipc.RpcServer: Unexpected
> throwable object·
> java.lang.NumberFormatException: For input string: "1-dal"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:492)
> at java.lang.Integer.parseInt(Integer.java:527)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.currentClientHasMinimumVersion(ProcedurePrepareLatch.java:61)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.hasProcedureSupport(ProcedurePrepareLatch.java:47)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.createLatch(ProcedurePrepareLatch.java:43)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1530)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:449)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51097)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15232) Exceptions returned over multi RPC don't automatically trigger region location reloads
[
https://issues.apache.org/jira/browse/HBASE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Elser updated HBASE-15232:
---
Attachment: HBASE-15232.001.patch
Reattaching for yetus. The tests that failed passed for me locally. Let's see
what a re-run does.
> Exceptions returned over multi RPC don't automatically trigger region
> location reloads
> --
>
> Key: HBASE-15232
> URL: https://issues.apache.org/jira/browse/HBASE-15232
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
>
> Attachments: HBASE-15232.001.patch
>
>
> Follow-on for HBASE-15221:
> A work-around was added in HTableMultiplexer to work around an issue that
> AsyncProcess wasn't clearing the region location cache on Exception. This was
> stemming from the issue that the {{tableName}} is {{null}} because
> HTableMultiplexer is using the {{multi}} RPC. This causes an error that looks
> like:
> {noformat}
> [WARN] Coding error, see method javadoc. row=[B@1673eff, tableName=null
> {noformat}
> HBASE-15221 should fix HTableMultiplexer, but it would be good to push the
> fix down into AsyncProcess instead of using higher-level workarounds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15237) NumberFormatException in create table if HBase version is of form X.Y.Z-BBB
Enis Soztutar created HBASE-15237:
-
Summary: NumberFormatException in create table if HBase version is
of form X.Y.Z-BBB
Key: HBASE-15237
URL: https://issues.apache.org/jira/browse/HBASE-15237
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Just ran into this while testing with a custom build with made up version
{{1.1.1-dal}}
{code}
2016-02-08 20:10:38,494 ERROR [B.defaultRpcServer.handler=1,queue=1,port=52622]
ipc.RpcServer: Unexpected throwable object·
java.lang.NumberFormatException: For input string: "1-dal"
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:492)
at java.lang.Integer.parseInt(Integer.java:527)
at
org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.currentClientHasMinimumVersion(ProcedurePrepareLatch.java:61)
at
org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.hasProcedureSupport(ProcedurePrepareLatch.java:47)
at
org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.createLatch(ProcedurePrepareLatch.java:43)
at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1530)
at
org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:449)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51097)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15233) Bytes.toBytes() methods should allow arrays to be re-used
[
https://issues.apache.org/jira/browse/HBASE-15233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138346#comment-15138346
]
Anoop Sam John commented on HBASE-15233:
In which area of code, you think we can make use of this new method? If there
are, it will be nice to add.
> Bytes.toBytes() methods should allow arrays to be re-used
> --
>
> Key: HBASE-15233
> URL: https://issues.apache.org/jira/browse/HBASE-15233
> Project: HBase
> Issue Type: Improvement
> Components: API
>Affects Versions: 1.1.3
>Reporter: Jean-Marc Spaggiari
>Assignee: Rachel Asher Silver
>Priority: Minor
> Labels: beginner
>
> Today we have this:
> {code}
> public static byte[] toBytes(long val) {
> byte [] b = new byte[8];
> for (int i = 7; i > 0; i--) {
> b[i] = (byte) val;
> val >>>= 8;
> }
> b[0] = (byte) val;
> return b;
> }
> {code}
> might be nice to also have this:
> {code}
> public static byte[] toBytes(long val, byte[] reuse) {
> for (int i = 7; i > 0; i--) {
> reuse[i] = (byte) val;
> val >>>= 8;
> }
> reuse[0] = (byte) val;
> return reuse;
> }
> {code}
> Same for all the other Bytes.toBytes() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: HBASE-15216.v2.patch
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch, HBASE-15216.v2.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15235) Inconsistent cell reads over multiple bulk-loaded HFiles
[ https://issues.apache.org/jira/browse/HBASE-15235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved HBASE-15235. - Resolution: Duplicate Dupe of HBASE-15236 > Inconsistent cell reads over multiple bulk-loaded HFiles > > > Key: HBASE-15235 > URL: https://issues.apache.org/jira/browse/HBASE-15235 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > > If there are two bulkloaded hfiles in a region with same seqID and duplicate > keys*, get and scan may return different values for a key. > More details: > - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' > and column qualifiers are 1 to 1000. > - hfiles were split somewhere along that row, but there were a range of > columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and > hfile2: [r:cf:40, ). > - Between columns 40 to 70, some (not all) columns were in both the files > with different values. Whereas other were only in one of the files. > In such a case, depending on file size (because we take it into account when > sorting hfiles internally), we may get different values for the same cell > (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" > "cf:". > I have been able to replicate this issue, will post the instructions shortly. > * not sure how this would happen. These files are small ~50M, nor could i > find any setting for max file size that could lead to splits. Need to > investigate more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[ https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-15236: - Description: If there are two bulkloaded hfiles in a region with same seqID and duplicate keys*, get and scan may return different values for a key. More details: - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' and column qualifiers are 1 to 1000. - hfiles were split somewhere along that row, but there were a range of columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and hfile2: [r:cf:40, ). - Between columns 40 to 70, some (not all) columns were in both the files with different values. Whereas other were only in one of the files. In such a case, depending on file size (because we take it into account when sorting hfiles internally), we may get different values for the same cell (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" "cf:". I have been able to replicate this issue, will post the instructions shortly. --- \* not sure how this would happen. These files are small ~50M, nor could i find any setting for max file size that could lead to splits. Need to investigate more. was: If there are two bulkloaded hfiles in a region with same seqID and duplicate keys*, get and scan may return different values for a key. More details: - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' and column qualifiers are 1 to 1000. - hfiles were split somewhere along that row, but there were a range of columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and hfile2: [r:cf:40, ). - Between columns 40 to 70, some (not all) columns were in both the files with different values. Whereas other were only in one of the files. In such a case, depending on file size (because we take it into account when sorting hfiles internally), we may get different values for the same cell (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" "cf:". I have been able to replicate this issue, will post the instructions shortly. * not sure how this would happen. These files are small ~50M, nor could i find any setting for max file size that could lead to splits. Need to investigate more. > Inconsistent cell reads over multiple bulk-loaded HFiles > > > Key: HBASE-15236 > URL: https://issues.apache.org/jira/browse/HBASE-15236 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > > If there are two bulkloaded hfiles in a region with same seqID and duplicate > keys*, get and scan may return different values for a key. > More details: > - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' > and column qualifiers are 1 to 1000. > - hfiles were split somewhere along that row, but there were a range of > columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and > hfile2: [r:cf:40, ). > - Between columns 40 to 70, some (not all) columns were in both the files > with different values. Whereas other were only in one of the files. > In such a case, depending on file size (because we take it into account when > sorting hfiles internally), we may get different values for the same cell > (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" > "cf:". > I have been able to replicate this issue, will post the instructions shortly. > --- > \* not sure how this would happen. These files are small ~50M, nor could i > find any setting for max file size that could lead to splits. Need to > investigate more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15198) RPC client not using Codec and CellBlock for puts by default
[
https://issues.apache.org/jira/browse/HBASE-15198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138379#comment-15138379
]
Anoop Sam John commented on HBASE-15198:
Ping for reviews..
> RPC client not using Codec and CellBlock for puts by default
>
>
> Key: HBASE-15198
> URL: https://issues.apache.org/jira/browse/HBASE-15198
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15198.patch, HBASE-15198_V2.patch,
> HBASE-15198_V3.patch, HBASE-15198_V4.patch, HBASE-15198_V5.patch
>
>
> For puts we use MultiServerCallable. Here to decide whether to use cellBlock
> we have
> {code}
> private boolean isCellBlock() {
> // This is not exact -- the configuration could have changed on us after
> connection was set up
> // but it will do for now.
> HConnection connection = getConnection();
> if (connection == null) return true; // Default is to do cellblocks.
> Configuration configuration = connection.getConfiguration();
> if (configuration == null) return true;
> String codec = configuration.get(HConstants.RPC_CODEC_CONF_KEY, "");
> return codec != null && codec.length() > 0;
> }
> {code}
> By default in hbase-default.xml, we dont have any Codec being specified.
> Where as in AbstractRpcClient we have
> {code}
> Codec getCodec() {
> // For NO CODEC, "hbase.client.rpc.codec" must be configured with empty
> string AND
> // "hbase.client.default.rpc.codec" also -- because default is to do cell
> block encoding.
> String className = conf.get(HConstants.RPC_CODEC_CONF_KEY,
> getDefaultCodec(this.conf));
> if (className == null || className.length() == 0) return null;
> try {
> return (Codec)Class.forName(className).newInstance();
> } catch (Exception e) {
> throw new RuntimeException("Failed getting codec " + className, e);
> }
> }
> .
> public static String getDefaultCodec(final Configuration c) {
> // If "hbase.client.default.rpc.codec" is empty string -- you can't set
> it to null because
> // Configuration will complain -- then no default codec (and we'll pb
> everything). Else
> // default is KeyValueCodec
> return c.get(DEFAULT_CODEC_CLASS, KeyValueCodec.class.getCanonicalName());
> }
> {code}
> Our aim is to by def use Codec and it is KeyValueCodec.
> The codec finding in MultiServerCallable to be same way as in
> AbstractRpcClient and then only we will be doing cellblock stuff.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15230) ReplicationSource is replicating existing WAL data in a newly added peer
[ https://issues.apache.org/jira/browse/HBASE-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi resolved HBASE-15230. --- Resolution: Duplicate Assignee: (was: Ashish Singhi) Duplicate of HBASE-9888. > ReplicationSource is replicating existing WAL data in a newly added peer > > > Key: HBASE-15230 > URL: https://issues.apache.org/jira/browse/HBASE-15230 > Project: HBase > Issue Type: Bug >Reporter: Ashish Singhi > > Scenario: > 1. Add a peer 'p1' and enable table replication for a table 't1' > 2. Put some data in the table 't1' and its gets replicated to peer 'p1' as > expected. > 3. Remove peer 'p1' and truncate the table 't1' in both source and peer > cluster. > 4. Now add peer 'p2' , there is no data in source cluster in table 't1' but > in peer cluster 'p2' where table 't1' already exists, existing WAL data of > table 't1' is getting replicated in 'p2'. > Expectation: Table 't1' in peer cluster 'p2' should only have data which is > inserted in source cluster table 't1' after adding peer 'p2' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14919) Infrastructure refactoring for In-Memory Flush
[
https://issues.apache.org/jira/browse/HBASE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138218#comment-15138218
]
Hadoop QA commented on HBASE-14919:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
23m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 97m 28s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 37s
{color} | {color:green} hbase-server in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 249m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestRowCounter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-08 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786886/HBASE-14919-V12.patch
|
| JIRA Issue | HBASE-14919 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux a2a41f70a728 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64
[jira] [Commented] (HBASE-14030) HBase Backup/Restore Phase 1
[ https://issues.apache.org/jira/browse/HBASE-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138308#comment-15138308 ] Ted Yu commented on HBASE-14030: TestBackupBoundaryTests fails in patch v34. Can you check ? > HBase Backup/Restore Phase 1 > > > Key: HBASE-14030 > URL: https://issues.apache.org/jira/browse/HBASE-14030 > Project: HBase > Issue Type: Umbrella >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Fix For: 2.0.0 > > Attachments: HBASE-14030-v0.patch, HBASE-14030-v1.patch, > HBASE-14030-v10.patch, HBASE-14030-v11.patch, HBASE-14030-v12.patch, > HBASE-14030-v13.patch, HBASE-14030-v14.patch, HBASE-14030-v15.patch, > HBASE-14030-v17.patch, HBASE-14030-v18.patch, HBASE-14030-v2.patch, > HBASE-14030-v20.patch, HBASE-14030-v21.patch, HBASE-14030-v22.patch, > HBASE-14030-v23.patch, HBASE-14030-v24.patch, HBASE-14030-v25.patch, > HBASE-14030-v26.patch, HBASE-14030-v27.patch, HBASE-14030-v28.patch, > HBASE-14030-v3.patch, HBASE-14030-v30.patch, HBASE-14030-v4.patch, > HBASE-14030-v5.patch, HBASE-14030-v6.patch, HBASE-14030-v7.patch, > HBASE-14030-v8.patch > > > This is the umbrella ticket for Backup/Restore Phase 1. See HBASE-7912 design > doc for the phase description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12751) Allow RowLock to be reader writer
[ https://issues.apache.org/jira/browse/HBASE-12751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138355#comment-15138355 ] stack commented on HBASE-12751: --- [~eclark] Nice side effect Regards HBASE-15047, as said before... radical. HBASE-15082 is in now (smile). > Allow RowLock to be reader writer > - > > Key: HBASE-12751 > URL: https://issues.apache.org/jira/browse/HBASE-12751 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 2.0.0, 1.3.0 >Reporter: Elliott Clark >Assignee: stack > Fix For: 2.0.0 > > Attachments: 12751.rebased.v25.txt, 12751.rebased.v26.txt, > 12751.rebased.v26.txt, 12751.rebased.v27.txt, 12751.rebased.v29.txt, > 12751.rebased.v31.txt, 12751.rebased.v32.txt, 12751.rebased.v32.txt, > 12751.rebased.v33.txt, 12751.rebased.v34.txt, 12751.rebased.v35.txt, > 12751.rebased.v35.txt, 12751.rebased.v35.txt, 12751.v37.txt, 12751.v38.txt, > 12751.v39.txt, 12751.v40.txt, 12751.v41.txt, 12751v22.txt, 12751v23.txt, > 12751v23.txt, 12751v23.txt, 12751v23.txt, 12751v36.txt, HBASE-12751-v1.patch, > HBASE-12751-v10.patch, HBASE-12751-v10.patch, HBASE-12751-v11.patch, > HBASE-12751-v12.patch, HBASE-12751-v13.patch, HBASE-12751-v14.patch, > HBASE-12751-v15.patch, HBASE-12751-v16.patch, HBASE-12751-v17.patch, > HBASE-12751-v18.patch, HBASE-12751-v19 (1).patch, HBASE-12751-v19.patch, > HBASE-12751-v2.patch, HBASE-12751-v20.patch, HBASE-12751-v20.patch, > HBASE-12751-v21.patch, HBASE-12751-v3.patch, HBASE-12751-v4.patch, > HBASE-12751-v5.patch, HBASE-12751-v6.patch, HBASE-12751-v7.patch, > HBASE-12751-v8.patch, HBASE-12751-v9.patch, HBASE-12751.patch > > > Right now every write operation grabs a row lock. This is to prevent values > from changing during a read modify write operation (increment or check and > put). However it limits parallelism in several different scenarios. > If there are several puts to the same row but different columns or stores > then this is very limiting. > If there are puts to the same column then mvcc number should ensure a > consistent ordering. So locking is not needed. > However locking for check and put or increment is still needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138417#comment-15138417
]
Vishal Khandelwal commented on HBASE-15229:
---
Submitted the patch by fixing formatting errors. Other failures are not related
to the Canary
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
> Attachments: HBASE-15229.v1.patch, HBASE-15229.v2.patch
>
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15232) Exceptions returned over multi RPC don't automatically trigger region location reloads
[
https://issues.apache.org/jira/browse/HBASE-15232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138209#comment-15138209
]
Hadoop QA commented on HBASE-15232:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 16s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 17s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
30m 9s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 132m 49s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 23s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 132m 49s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 352m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| JDK v1.7.0_95 Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| |
[jira] [Commented] (HBASE-15228) Add the methods to RegionObserver to trigger start/complete restoring WALs
[ https://issues.apache.org/jira/browse/HBASE-15228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138291#comment-15138291 ] Toshihiro Suzuki commented on HBASE-15228: -- Yes, we have preWALRestore and postWALRestore in RegionObserver already. But those are called per WAL. The methods I added are called when start/complete restoring WALs. > Add the methods to RegionObserver to trigger start/complete restoring WALs > -- > > Key: HBASE-15228 > URL: https://issues.apache.org/jira/browse/HBASE-15228 > Project: HBase > Issue Type: New Feature > Components: Coprocessors >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki > Attachments: HBASE-15228.patch > > > In our use case, we write indexes to a index table when restoring WALs. We > want to trigger start/complete restoring WALs for initial and end processing > for writing indexes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15223) Make convertScanToString public for Spark
[
https://issues.apache.org/jira/browse/HBASE-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138338#comment-15138338
]
Hadoop QA commented on HBASE-15223:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
38s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 50s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} branch-1 passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} branch-1 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red} Patch generated 2 new checkstyle issues in hbase-server
(total was 9, now 11). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green} 4m
12s {color} | {color:green} Patch does not cause any errors with Hadoop 2.4.1
2.5.2 2.6.0. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 81m 30s
{color} | {color:green} hbase-server in the patch passed with JDK v1.8.0_72.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 86m 3s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 183m 48s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_95 Failed junit tests | hadoop.hbase.mapreduce.TestImportExport |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-09 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786969/HBASE-15223-branch-1.patch
|
| JIRA Issue | HBASE-15223 |
| Optional Tests |
[jira] [Commented] (HBASE-13259) mmap() based BucketCache IOEngine
[ https://issues.apache.org/jira/browse/HBASE-13259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138356#comment-15138356 ] ramkrishna.s.vasudevan commented on HBASE-13259: Completed the testing. Here are the findings Using YCSB scans and gets were performed with 75 threads and the throughput was measured. I used to a server with 50GB of RAM. Measured the throughput diff between a setup where the mmap() file mode was configured with a cache of 100G. In one of the setup only 10G of data was loaded and that was cached and in another loaded around 75G of data and the whole 75G was cached in the file mode BC. With 10G of cache ||Scans||Gets|| |13697.46 ops/sec|69085.88 ops/sec|| With 75G of cache ||Scans||Gets|| |8745.08 ops/sec|66221.93 ops/sec|| The same 75G of cache setup was run with the current File Mode impl of BC ||Scans||Gets|| |12107.92 ops/sec|42725.07 ops/sec|| Also my Filemode BC impl is backed with *PCIe SSD*. So the test clearly shows that the mmap based file mode is best suited for gets rather than scans because when the data does not fit in the RAM there may lot of page faults and we do a lot of read operations like compare on the BB that is retrieved out of this mmap buffers. Whereas in the current way of File Mode BC since the BB is copied to the onheap there is no page faults. > mmap() based BucketCache IOEngine > - > > Key: HBASE-13259 > URL: https://issues.apache.org/jira/browse/HBASE-13259 > Project: HBase > Issue Type: New Feature > Components: BlockCache >Affects Versions: 0.98.10 >Reporter: Zee Chen >Assignee: Zee Chen >Priority: Critical > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-13259-v2.patch, HBASE-13259.patch, > HBASE-13259_v3.patch, ioread-1.svg, mmap-0.98-v1.patch, mmap-1.svg, > mmap-trunk-v1.patch > > > Of the existing BucketCache IOEngines, FileIOEngine uses pread() to copy data > from kernel space to user space. This is a good choice when the total working > set size is much bigger than the available RAM and the latency is dominated > by IO access. However, when the entire working set is small enough to fit in > the RAM, using mmap() (and subsequent memcpy()) to move data from kernel > space to user space is faster. I have run some short keyval gets tests and > the results indicate a reduction of 2%-7% of kernel CPU on my system, > depending on the load. On the gets, the latency histograms from mmap() are > identical to those from pread(), but peak throughput is close to 40% higher. > This patch modifies ByteByfferArray to allow it to specify a backing file. > Example for using this feature: set hbase.bucketcache.ioengine to > mmap:/dev/shm/bucketcache.0 in hbase-site.xml. > Attached perf measured CPU usage breakdown in flames graph. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: HBASE-15216.v2.patch
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch, HBASE-15216.v2.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138402#comment-15138402
]
Vishal Khandelwal commented on HBASE-15216:
---
fixed the formatting and findbugs issue.
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch, HBASE-15216.v2.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15201) Add hbase-spark to hbase assembly
[ https://issues.apache.org/jira/browse/HBASE-15201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138304#comment-15138304 ] Hudson commented on HBASE-15201: SUCCESS: Integrated in HBase-Trunk_matrix #692 (See [https://builds.apache.org/job/HBase-Trunk_matrix/692/]) HBASE-15201 Add hbase-spark to hbase assembly (jerryjch: rev 3aff98c75b5e23a5010be17eecef3140d2bf70bb) * hbase-spark/pom.xml * hbase-assembly/src/main/assembly/hadoop-two-compat.xml * hbase-assembly/pom.xml > Add hbase-spark to hbase assembly > - > > Key: HBASE-15201 > URL: https://issues.apache.org/jira/browse/HBASE-15201 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15201.patch > > > hbase-spark currently is missing from hbase assembly. > We should add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15231) Make TableState.State private
[ https://issues.apache.org/jira/browse/HBASE-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138305#comment-15138305 ] Hudson commented on HBASE-15231: SUCCESS: Integrated in HBase-Trunk_matrix #692 (See [https://builds.apache.org/job/HBase-Trunk_matrix/692/]) HBASE-15231 Make TableState.State private (Misty Stanley-Jones) (tedyu: rev 7bb68b9031591cf378954a0eb8f71a8b9be01f9c) * hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableState.java > Make TableState.State private > - > > Key: HBASE-15231 > URL: https://issues.apache.org/jira/browse/HBASE-15231 > Project: HBase > Issue Type: Bug > Components: API >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-15231.patch > > > TableState is private but TableState.State is evolving. This means that > Javadoc links from TableState.State to TableState are broken. [~stack] says > that TableState.State should be private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15237) NumberFormatException in create table if HBase version is of form X.Y.Z-BBB
[
https://issues.apache.org/jira/browse/HBASE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar resolved HBASE-15237.
---
Resolution: Not A Problem
Reading the code, I think the version was 1.1-dal, not 1.1.1-dal. Resolving for
now.
> NumberFormatException in create table if HBase version is of form X.Y.Z-BBB
> ---
>
> Key: HBASE-15237
> URL: https://issues.apache.org/jira/browse/HBASE-15237
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
>
> Just ran into this while testing with a custom build with made up version
> {{1.1.1-dal}}
> {code}
> 2016-02-08 20:10:38,494 ERROR
> [B.defaultRpcServer.handler=1,queue=1,port=52622] ipc.RpcServer: Unexpected
> throwable object·
> java.lang.NumberFormatException: For input string: "1-dal"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:492)
> at java.lang.Integer.parseInt(Integer.java:527)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.currentClientHasMinimumVersion(ProcedurePrepareLatch.java:61)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.hasProcedureSupport(ProcedurePrepareLatch.java:47)
> at
> org.apache.hadoop.hbase.master.procedure.ProcedurePrepareLatch.createLatch(ProcedurePrepareLatch.java:43)
> at org.apache.hadoop.hbase.master.HMaster.createTable(HMaster.java:1530)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.createTable(MasterRpcServices.java:449)
> at
> org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:51097)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2114)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:101)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:130)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:107)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
[ https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138352#comment-15138352 ] stack commented on HBASE-15205: --- bq. But the other thing is that if we can just add all the non default CF and serialize it for every WAL key we can even avoid the local map getting created and the check that we perform on these maps etc. But at the cost of serializing more information per WAL. When then is the scope checked? When we go to replicate after reading the WAL? We'd be trading CPU for i/o? If we calculate once on open of the region, we would have to rolling restart when scopes are changed (that is probably fine). > Do not find the replication scope for every WAL#append() > > > Key: HBASE-15205 > URL: https://issues.apache.org/jira/browse/HBASE-15205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15205.patch, ScopeWALEdits.jpg, > ScopeWALEdits_afterpatch.jpg > > > After the byte[] and char[] the other top contributor for lot of GC (though > it is only 2.86%) is the UTF_8.newDecoder. > This happens because for every WAL append we try to calculate the replication > scope associate with the families associated with the TableDescriptor. I > think per WAL append doing this is very costly and creates lot of garbage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15229:
--
Attachment: HBASE-15229.v2.patch
> Canary Tools should not call System.Exit on error
> -
>
> Key: HBASE-15229
> URL: https://issues.apache.org/jira/browse/HBASE-15229
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.17
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
>Priority: Critical
> Attachments: HBASE-15229.v1.patch, HBASE-15229.v2.patch
>
>
> run method in canary tool calls system.Exit on failure. Due to this it can't
> be integrated as an API as jvm will stop on failure. It should only return
> (exit code)
> So any integration with unit test also fail with
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test (default-test) on
> project <>: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:2.14:test failed: The forked
> VM terminated without saying properly goodbye. VM crash or System.exit called
> ?
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15205) Do not find the replication scope for every WAL#append()
[ https://issues.apache.org/jira/browse/HBASE-15205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138300#comment-15138300 ] Enis Soztutar commented on HBASE-15205: --- bq. If we are ok with the suggestion of adding all the families that has non default scope per WAL Key then we can make things much faster in the direct write path since what ever we extract from the HRegion can directly be persisted per WALKey. No need for any checks in the write path to see what the families with each cell and then creating another local map that can be persisted in PBs. I was reading the patch without looking further down the comments and was thinking the same thing. I think it is worth doing it. Although it is not just the WAL overhead, but the replicate RPC also encodes the scopes. > Do not find the replication scope for every WAL#append() > > > Key: HBASE-15205 > URL: https://issues.apache.org/jira/browse/HBASE-15205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-15205.patch, ScopeWALEdits.jpg, > ScopeWALEdits_afterpatch.jpg > > > After the byte[] and char[] the other top contributor for lot of GC (though > it is only 2.86%) is the UTF_8.newDecoder. > This happens because for every WAL append we try to calculate the replication > scope associate with the families associated with the TableDescriptor. I > think per WAL append doing this is very costly and creates lot of garbage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15236) Inconsistent cell reads over multiple bulk-loaded HFiles
[ https://issues.apache.org/jira/browse/HBASE-15236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138342#comment-15138342 ] Appy commented on HBASE-15236: -- https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java#L1751 > Inconsistent cell reads over multiple bulk-loaded HFiles > > > Key: HBASE-15236 > URL: https://issues.apache.org/jira/browse/HBASE-15236 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > > If there are two bulkloaded hfiles in a region with same seqID and duplicate > keys*, get and scan may return different values for a key. > More details: > - one of the rows had 200k+ columns. say row is 'r', column family is 'cf' > and column qualifiers are 1 to 1000. > - hfiles were split somewhere along that row, but there were a range of > columns in both hfiles. For eg, something like - hfile1: ["", r:cf:70) and > hfile2: [r:cf:40, ). > - Between columns 40 to 70, some (not all) columns were in both the files > with different values. Whereas other were only in one of the files. > In such a case, depending on file size (because we take it into account when > sorting hfiles internally), we may get different values for the same cell > (say "r", "cf:50") depending on what we call: get "r" "cf:50" or get "r" > "cf:". > I have been able to replicate this issue, will post the instructions shortly. > --- > \* not sure how this would happen. These files are small ~50M, nor could i > find any setting for max file size that could lead to splits. Need to > investigate more. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15223) Make convertScanToString public for Spark
[ https://issues.apache.org/jira/browse/HBASE-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15138343#comment-15138343 ] stack commented on HBASE-15223: --- TestImportExport is flakey. You passed all 1.8 tests. That is good enough. Try and fix the findbugs and the checkstyle on commit if you cant [~jerryhe]? Patch seems good to me. > Make convertScanToString public for Spark > - > > Key: HBASE-15223 > URL: https://issues.apache.org/jira/browse/HBASE-15223 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15223-branch-1.patch, HBASE-15223-master.patch > > > One way to access HBase from Spark is to use newAPIHadoopRDD, which can take > a TableInputFormat as class name. But we are not able to set a Scan object > in there, for example to set a HBase filter. > In MR, the public API TableMapReduceUtil.initTableMapperJob() or equivalent > is used which can take a Scan object. But this call is not used in Spark > conveniently. > We need to make the TableMapReduceUtil.convertScanToString() public. > So that a Scan object can be created, populated and then convert to the > property and used by Spark. They are now package private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: HBASE-15229.v3.patch
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch, HBASE-15216.v2.patch,
> HBASE-15229.v3.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-15230) ReplicationSource is replicating existing WAL data in a newly added peer
Ashish Singhi created HBASE-15230: - Summary: ReplicationSource is replicating existing WAL data in a newly added peer Key: HBASE-15230 URL: https://issues.apache.org/jira/browse/HBASE-15230 Project: HBase Issue Type: Bug Reporter: Ashish Singhi Assignee: Ashish Singhi Scenario: 1. Add a peer 'p1' and enable table replication for a table 't1' 2. Put some data in the table 't1' and its gets replicated to peer 'p1' as expected. 3. Remove peer 'p1' and truncate the table 't1' in both source and peer cluster. 4. Now add peer 'p2' , there is no data in source cluster in table 't1' but in peer cluster 'p2' where table 't1' already exists, existing WAL data of table 't1' is getting replicated in 'p2'. Expectation: Table 't1' in peer cluster 'p2' should only have data which is inserted in source cluster table 't1' after adding peer 'p2' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Status: Patch Available (was: Open)
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15122) Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
[
https://issues.apache.org/jira/browse/HBASE-15122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136950#comment-15136950
]
Samir Ahmic commented on HBASE-15122:
-
[~busbey], [~stack] do we need any additional work around this issue ? I have
test v3 on master branch. Looks good.
> Servlets generate XSS_REQUEST_PARAMETER_TO_SERVLET_WRITER findbugs warnings
> ---
>
> Key: HBASE-15122
> URL: https://issues.apache.org/jira/browse/HBASE-15122
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Priority: Critical
> Attachments: HBASE-15122-v0-master, HBASE-15122.patch,
> HBASE-15122_v1.patch, HBASE-15122_v2.patch, HBASE-15122_v3.patch
>
>
> In our JMXJsonServlet we are doing this:
> jsonpcb = request.getParameter(CALLBACK_PARAM);
> if (jsonpcb != null) {
> response.setContentType("application/javascript; charset=utf8");
> writer.write(jsonpcb + "(");
> ...
> Findbugs complains rightly. There are other instances in our servlets and
> then there are the pages generated by jamon excluded from findbugs checking
> (and findbugs volunteers that it is dumb in this regard finding only the most
> egregious of violations).
> We have no sanitizing tooling in hbase that I know of (correct me if I am
> wrong). I started to pull on this thread and it runs deep. Our Jamon
> templating (last updated in 2013 and before that, in 2011) engine doesn't
> seem to have sanitizing means either and there seems to be outstanding XSS
> complaint against jamon that goes unaddressed.
> Could pull in something like
> https://www.owasp.org/index.php/OWASP_Java_Encoder_Project and run all
> emissions via it or get a templating engine that has sanitizing built in.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136951#comment-15136951
]
Vishal Khandelwal commented on HBASE-15216:
---
Submitted the patch for main, aftee the review i can provide the pacth for 0.98
version as well.
FYI [~apurtell]
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136961#comment-15136961
]
Elliott Clark commented on HBASE-14790:
---
For any data that HBase will ever consider to be durable we will have called
hsync (excluding edits that explicitly have it turned off). That turns
DFSOutputStream isSync = true. That will be mean that syncBlock is true in the
above code. That will mean that all data before the sync is in the kernel's
page cache. Any data that's only appended but not synced will never send a
result to the client for HBase.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136985#comment-15136985
]
Duo Zhang commented on HBASE-14790:
---
[~eclark] This is the implementation of {{ProtobufLogWriter.sync}}
{code:title=ProtobufLogWriter.java}
@Override
public void sync() throws IOException {
FSDataOutputStream fsdos = this.output;
if (fsdos == null) return; // Presume closed
fsdos.flush();
fsdos.hflush();
}
{code}
We only call hflush here? And yes, hsync has its benefits, but it is really
slow...
Thanks.
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15211) Don't run the CatalogJanitor if there are regions in transition
[ https://issues.apache.org/jira/browse/HBASE-15211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-15211: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Don't run the CatalogJanitor if there are regions in transition > --- > > Key: HBASE-15211 > URL: https://issues.apache.org/jira/browse/HBASE-15211 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4 > > Attachments: HBASE-15211.patch > > > This could make things a little safer to make sure that CatalogJanitor > doesn't do something weird while a cluster is starting up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15216) Canary does not accept few client config param from commandline( it should only be present in hbase-site.xml)
[
https://issues.apache.org/jira/browse/HBASE-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vishal Khandelwal updated HBASE-15216:
--
Attachment: HBASE-15216.v1.patch
> Canary does not accept few client config param from commandline( it should
> only be present in hbase-site.xml)
> -
>
> Key: HBASE-15216
> URL: https://issues.apache.org/jira/browse/HBASE-15216
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Vishal Khandelwal
> Attachments: HBASE-15216.v1.patch
>
>
> At present there are few configs which needs to be present in hbase-site or
> default xml for it to work. following are the list.
> hbase.canary.threads.num
> hbase.canary.sink.class
> hbase.client.keytab.file
> hbase.client.kerberos.principal
> Execution in secure expects keytab and princ to be present
> {noformat}
> 2016-02-05 05:58:44,024 ERROR [main] hbase.AuthUtil - Error while trying to
> perform the initial login: Running in secure mode, but config doesn't have a
> keytab
> java.io.IOException: Running in secure mode, but config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> Exception in thread "main" java.io.IOException: Running in secure mode, but
> config doesn't have a keytab
> at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:236)
> at
> org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:392)
> at org.apache.hadoop.hbase.security.User.login(User.java:259)
> at
> org.apache.hadoop.hbase.security.UserProvider.login(UserProvider.java:116)
> at org.apache.hadoop.hbase.AuthUtil.launchAuthChore(AuthUtil.java:64)
> at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1146)
> {noformat}
> {code}
> public static void main(String[] args) throws Exception {
> final Configuration conf = HBaseConfiguration.create();
> AuthUtil.launchAuthChore(conf);
> int numThreads = conf.getInt("hbase.canary.threads.num", MAX_THREADS_NUM);
> ExecutorService executor = new ScheduledThreadPoolExecutor(numThreads);
> Class sinkClass =
> conf.getClass("hbase.canary.sink.class", StdOutSink.class,
> Sink.class);
> Sink sink = ReflectionUtils.newInstance(sinkClass);
> int exitCode = ToolRunner.run(conf, new Canary(executor, sink), args);
> executor.shutdown();
> System.exit(exitCode);
> }
> {code}
> In main class these params should be parsed and updated. else for any change
> to these value hbase-stie.xml needs to be updated
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136991#comment-15136991
]
Elliott Clark commented on HBASE-14790:
---
I was looking at WALProcedure rather than protobuf.
Yeah we really need something between hflush and hsync. The current behavior is
just wrong and broken :-/
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15229) Canary Tools should not call System.Exit on error
[
https://issues.apache.org/jira/browse/HBASE-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137054#comment-15137054
]
Hadoop QA commented on HBASE-15229:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
3s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m 3s
{color} | {color:red} Patch generated 10 new checkstyle issues in hbase-server
(total was 15, now 25). {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 9 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
22m 41s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 99m 48s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0_72. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 146m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_72 Failed junit tests | hadoop.hbase.mapreduce.TestRowCounter |
| JDK v1.8.0_72 Timed out junit tests |
[jira] [Commented] (HBASE-14949) Skip duplicate entries when replay WAL.
[ https://issues.apache.org/jira/browse/HBASE-14949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137089#comment-15137089 ] Phil Yang commented on HBASE-14949: --- I think we needn't wait for upgrading finished. If we solve this issue in version A, and use new WAL logic in version B. We can only support rolling upgrade to version B if the old version >=A. So if there is a RS with version B, all RSs in the cluster will contain this patch. > Skip duplicate entries when replay WAL. > --- > > Key: HBASE-14949 > URL: https://issues.apache.org/jira/browse/HBASE-14949 > Project: HBase > Issue Type: Sub-task >Reporter: Heng Chen > Attachments: HBASE-14949.patch, HBASE-14949_v1.patch, > HBASE-14949_v2.patch > > > As HBASE-14004 design, there will be duplicate entries in different WAL. It > happens when one hflush failed, we will close old WAL with 'acked hflushed' > length, then open a new WAL and write the unacked hlushed entries into it. > So there maybe some overlap between old WAL and new WAL. > We should skip the duplicate entries when replay. I think it has no harm to > current logic, maybe we do it first. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15230) ReplicationSource is replicating existing WAL data in a newly added peer
[ https://issues.apache.org/jira/browse/HBASE-15230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15136952#comment-15136952 ] Ashish Singhi commented on HBASE-15230: --- As per the current design the replication of WAL data starts from the beginning of the current WAL file. So in this case the table 't1' entries already present in the current WAL will get replicated to peer 'p2'. Didn't find any way to set the current WAL writer offset while adding the peer in RS. One work around we think is to roll the WAL while adding the peer in each RS. Any other suggestions ? > ReplicationSource is replicating existing WAL data in a newly added peer > > > Key: HBASE-15230 > URL: https://issues.apache.org/jira/browse/HBASE-15230 > Project: HBase > Issue Type: Bug >Reporter: Ashish Singhi >Assignee: Ashish Singhi > > Scenario: > 1. Add a peer 'p1' and enable table replication for a table 't1' > 2. Put some data in the table 't1' and its gets replicated to peer 'p1' as > expected. > 3. Remove peer 'p1' and truncate the table 't1' in both source and peer > cluster. > 4. Now add peer 'p2' , there is no data in source cluster in table 't1' but > in peer cluster 'p2' where table 't1' already exists, existing WAL data of > table 't1' is getting replicated in 'p2'. > Expectation: Table 't1' in peer cluster 'p2' should only have data which is > inserted in source cluster table 't1' after adding peer 'p2' -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15219) Canary tool does not return non-zero exit when one of region stuck state
[
https://issues.apache.org/jira/browse/HBASE-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137008#comment-15137008
]
Ted Yu commented on HBASE-15219:
bq. sniff to all the regions
What's the difference between several of the regions having failures vs. half
of the regions having failures ?
How about introducing a threshold (expressed either in count or in percentage
of the number of regions) beyond which failures are treated as error ?
> Canary tool does not return non-zero exit when one of region stuck state
> -
>
> Key: HBASE-15219
> URL: https://issues.apache.org/jira/browse/HBASE-15219
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15219.v1.patch, HBASE-15219.v3.patch,
> HBASE-15219.v4.patch
>
>
> {code}
> 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from
> region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> column family 0 failed
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Fri Feb 05 12:24:15 GMT 2016,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> -bash-4.1$ echo $?
> 0
> {code}
> Below code prints the error but it does sets/returns the exit code. Due to
> this tool can't be integrated with nagios or other alerting.
> Ideally it should return error for failures. as pre the documentation:
>
> This tool will return non zero error codes to user for collaborating with
> other monitoring tools, such as Nagios. The error code definitions are:
> private static final int USAGE_EXIT_CODE = 1;
> private static final int INIT_ERROR_EXIT_CODE = 2;
> private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
> private static final int ERROR_EXIT_CODE = 4;
>
> {code}
> org.apache.hadoop.hbase.tool.Canary.RegionTask
> public Void read() {
>
> try {
> table = connection.getTable(region.getTable());
> tableDesc = table.getTableDescriptor();
> } catch (IOException e) {
> LOG.debug("sniffRegion failed", e);
> sink.publishReadFailure(region, e);
>...
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137067#comment-15137067 ] Josh Elser commented on HBASE-15221: bq. Presuming we still want this fixed on 0.98, could you make an appropriate patch? Sure thing. bq. Before pushing to branch-1.1 I wanted to confirm that we're fine with adding a method to the public API in a patch release? Per the guidance last I read it, such a method falls is acceptable to add in a patch-release. I am happy to be out-of-date on what the guidance is, though. > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4, 0.98.18 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14790) Implement a new DFSOutputStream for logging WAL only
[
https://issues.apache.org/jira/browse/HBASE-14790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137286#comment-15137286
]
Yu Li commented on HBASE-14790:
---
bq. Yeah we really need something between hflush and hsync. The current
behavior is just wrong and broken
I guess HBASE-5954 is following this topic? I believe the real hsync will be
used when hardware improves to some extent (actually we already see promising
result with PCIe SSD), but not sure how much longer we still need to wait. On
the other hand, IMHO a fan-out DFSOutputStream is nice to have for write
performance, no matter using hflush or hsync. :-)
> Implement a new DFSOutputStream for logging WAL only
>
>
> Key: HBASE-14790
> URL: https://issues.apache.org/jira/browse/HBASE-14790
> Project: HBase
> Issue Type: Improvement
>Reporter: Duo Zhang
>
> The original {{DFSOutputStream}} is very powerful and aims to serve all
> purposes. But in fact, we do not need most of the features if we only want to
> log WAL. For example, we do not need pipeline recovery since we could just
> close the old logger and open a new one. And also, we do not need to write
> multiple blocks since we could also open a new logger if the old file is too
> large.
> And the most important thing is that, it is hard to handle all the corner
> cases to avoid data loss or data inconsistency(such as HBASE-14004) when
> using original DFSOutputStream due to its complicated logic. And the
> complicated logic also force us to use some magical tricks to increase
> performance. For example, we need to use multiple threads to call {{hflush}}
> when logging, and now we use 5 threads. But why 5 not 10 or 100?
> So here, I propose we should implement our own {{DFSOutputStream}} when
> logging WAL. For correctness, and also for performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15231) Make TableState.State private
[ https://issues.apache.org/jira/browse/HBASE-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-15231: Attachment: HBASE-15231.patch > Make TableState.State private > - > > Key: HBASE-15231 > URL: https://issues.apache.org/jira/browse/HBASE-15231 > Project: HBase > Issue Type: Bug > Components: API >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-15231.patch > > > TableState is private but TableState.State is evolving. This means that > Javadoc links from TableState.State to TableState are broken. [~stack] says > that TableState.State should be private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15231) Make TableState.State private
[ https://issues.apache.org/jira/browse/HBASE-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-15231: Status: Patch Available (was: Open) > Make TableState.State private > - > > Key: HBASE-15231 > URL: https://issues.apache.org/jira/browse/HBASE-15231 > Project: HBase > Issue Type: Bug > Components: API >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Attachments: HBASE-15231.patch > > > TableState is private but TableState.State is evolving. This means that > Javadoc links from TableState.State to TableState are broken. [~stack] says > that TableState.State should be private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15082) Fix merge of MVCC and SequenceID performance regression
[ https://issues.apache.org/jira/browse/HBASE-15082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15082: -- Resolution: Not A Problem Release Note: As it happens, the performance regression in increment, append, etc., was fixed as a side effect of HBASE-12751. See HBASE-15213 for detail. Status: Resolved (was: Patch Available) Resolving as not a problem. See release note. > Fix merge of MVCC and SequenceID performance regression > --- > > Key: HBASE-15082 > URL: https://issues.apache.org/jira/browse/HBASE-15082 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0 > > Attachments: 15082.patch, 15082v10.patch, 15082v12.patch, > 15082v13.patch, 15082v14.patch, 15082v14.patch, 15082v15.patch, > 15082v16.patch, 15082v2.patch, 15082v2.txt, 15082v3.txt, 15082v4.patch, > 15082v5.patch, 15082v6.patch, 15082v7.patch, 15082v8.patch > > > This is general fix for increments (appends, checkAnd*) perf-regression > identified in the parent issue. HBASE-15031 has a narrow fix for branch-1.1 > and branch-1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[ https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15224: -- Resolution: Fixed Fix Version/s: 1.0.4 1.1.4 1.2.1 Status: Resolved (was: Patch Available) Pushed to branch-1.0+ Not needed in 2.0. > Undo "hbase.increment.fast.but.narrow.consistency" option; it is not > necessary since HBASE-15213 > -- > > Key: HBASE-15224 > URL: https://issues.apache.org/jira/browse/HBASE-15224 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Fix For: 1.2.1, 1.1.4, 1.0.4 > > Attachments: > 0001-HBASE-15224-Undo-hbase.increment.fast.branch-1.2.patch, > 0001-HBASE-15224-branch-1.2.patch, 15224.branch-1.2.patch, > HBASE-15224.branch-1.2.patch, HBASE-15224v2-branch-1.2.patch > > > Remove an escape valve no longer needed after HBASE-15213. Remove so folks > don't ever have to worry their pretty-little heads about it. > Let the bulk of HBASE-15031 remain, the issue that added > "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we > want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15223) Make convertScanToString public for Spark
[ https://issues.apache.org/jira/browse/HBASE-15223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137404#comment-15137404 ] Jerry He commented on HBASE-15223: -- Tests org.apache.hadoop.hbase.mapred.* org.apache.hadoop.hbase.mapreduce.*, org.apache.hadoop.hbase.snapshot.*, org.apache.hadoop.hbase.coprocessor.TestRegionObserverScannerOpenHook, org.apache.hadoop.hbase.TestIOFencing all pass cleanly on my local env. Will trigger a new QA run again. > Make convertScanToString public for Spark > - > > Key: HBASE-15223 > URL: https://issues.apache.org/jira/browse/HBASE-15223 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15223-master.patch > > > One way to access HBase from Spark is to use newAPIHadoopRDD, which can take > a TableInputFormat as class name. But we are not able to set a Scan object > in there, for example to set a HBase filter. > In MR, the public API TableMapReduceUtil.initTableMapperJob() or equivalent > is used which can take a Scan object. But this call is not used in Spark > conveniently. > We need to make the TableMapReduceUtil.convertScanToString() public. > So that a Scan object can be created, populated and then convert to the > property and used by Spark. They are now package private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14460) [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed Increments, CheckAndPuts, batch operations
[ https://issues.apache.org/jira/browse/HBASE-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-14460. --- Resolution: Fixed Fix Version/s: 1.0.4 1.1.4 1.2.1 1.3.0 2.0.0 Release Note: This release note tries to tell the general story. Dive into sub-tasks for more specific release noting. Increments, appends, checkAnd* have been slow since hbase-.1.0.0. The unification of mvcc and sequence id done by HBASE-8763 was responsible. A ‘fast-path’ workaround was added by HBASE-15031 “Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments”. It became available in 1.0.3 and 1.1.3. To enable the fast path, set "hbase.increment.fast.but.narrow.consistency" and then rolling restart. The workaround was for increments only (appends, checkAndPut, etc., were not addressed. See HBASE-15031 release note for more detail). Subsequently, the regression was properly identified and fixed in HBASE-15213 and the fix applied to branch-1.0 and branch-1.1. As it happens, hbase-1.2.0 does not suffer from the performance regression (though the thought was that it did -- and so it got the fast-path patch too via HBASE-15092) nor does the master branch. HBASE-15213 identified that HBASE-12751 (as a side effect) had cured the regression. hbase-1.0.4 (if it is ever released -- 1.0 has been end-of-lifed) and hbase-1.1.4 will have the HBASE-15213 fix. If you are suffering from the increment regression and you are on 1.0.3 or 1.1.3, you can enable the work around to get back your increment performance but you should upgrade. was:Regression addressed narrowly on branch-1 (1.0.3+, 1.1.3+, and 1.2.0). You need to set a "hbase.increment.fast.but.narrow.consistency" and rolling restart to enable a 'fast path' for increments only (appends, checkAndPut, etc., are not address in branch-1). See the sub-task release notes for more detail. On master branch, a more substantial change is done changing write path so all check-and-put type operations -- i.e. increment, append, checkAnd*, etc. -- have the regression addressed. > [Perf Regression] Merge of MVCC and SequenceId (HBASE-8763) slowed > Increments, CheckAndPuts, batch operations > - > > Key: HBASE-14460 > URL: https://issues.apache.org/jira/browse/HBASE-14460 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4 > > Attachments: 0.94.test.patch, 0.98.test.patch, > 1.0.80.flamegraph-7932.svg, 14460.txt, 14460.v0.branch-1.0.patch, > 98.80.flamegraph-11428.svg, HBASE-14460-discussion.patch, client.test.patch, > flamegraph-13120.svg.master.singlecell.svg, flamegraph-26636.094.100.svg, > flamegraph-28066.098.singlecell.svg, flamegraph-28767.098.100.svg, > flamegraph-31647.master.100.svg, flamegraph-9466.094.singlecell.svg, > hack.flamegraph-16593.svg, hack.uncommitted.patch, m.test.patch, > region_lock.png, testincrement.094.patch, testincrement.098.patch, > testincrement.master.patch > > > As reported by 鈴木俊裕 up on the mailing list -- see "Performance degradation > between CDH5.3.1(HBase0.98.6) and CDH5.4.5(HBase1.0.0)" -- our unification of > sequenceid and MVCC slows Increments (and other ops) as the mvcc needs to > 'catch up' to our current point before we can read the last Increment value > that we need to update. > We can say that our Increment is just done wrong, we should just be writing > Increments and summing on read, but checkAndPut as well as batching > operations have the same issue. Fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[ https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137469#comment-15137469 ] Nick Dimiduk commented on HBASE-15221: -- Skimmed patch. Adding new constructor is okay re: our compat guidelines. +1 for branch-1.1. > HTableMultiplexer improvements (stale region locations and resource leaks) > -- > > Key: HBASE-15221 > URL: https://issues.apache.org/jira/browse/HBASE-15221 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Critical > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4 > > Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch, > HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch > > > It looks like HTableMultiplexer has a couple of issues. > Upon failing to send a Put to the appropriate RS, the Put is re-queued back > into the system. Normally this is fine as such an exception is transient and > the Put would eventually succeed. However, in the case where the Put was > rejected because of a NotServingRegionException (e.g. split, balance, merge), > the re-queuing of the Put will end up using the same cached HRegionLocation. > This means that the Put will just be repeatedly sent back to the same RS over > and over again, eventually being dropped on the floor. Need to invalidate the > location cache (or make sure we refresh it) when we re-queue the Put. > The internal ClusterConnection is also leaked. If a user creates many > HTableMultiplexers, they'll eventually run into issues (memory, zk > connections, etc) because they'll never get cleaned up. HTableMultiplexer > needs a close method. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15224) Undo "hbase.increment.fast.but.narrow.consistency" option; it is not necessary since HBASE-15213
[ https://issues.apache.org/jira/browse/HBASE-15224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-15224: -- Release Note: HBASE-15031 “Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments” and HBASE-15092 ‘Forward-port to 1.2+ HBASE-15031 "Fix merge of MVCC and SequenceID performance regression in branch-1.0 for Increments"’ added a workaround ‘fast-path’ to restore an increment performance regression that came into hbase 1.0 when we unified mvcc and sequence id in HBASE-8763. The workaround became available in hbase-1.0.3 and hbase-1.1.3. The workaround involved setting the flag "hbase.increment.fast.but.narrow.consistency" in your configuration and restarting. Subsequently, the regression was fixed in HBASE-15213. The fix will be available in hbase-1.0.4 and hbase-1.1.4 when they are released. hbase-1.2.0 has the flag but as it turns out, it is not needed; there is no regression in 1.2.0. This issue removes the fast-path flag. If set, it will just be ignored. > Undo "hbase.increment.fast.but.narrow.consistency" option; it is not > necessary since HBASE-15213 > -- > > Key: HBASE-15224 > URL: https://issues.apache.org/jira/browse/HBASE-15224 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Attachments: > 0001-HBASE-15224-Undo-hbase.increment.fast.branch-1.2.patch, > 0001-HBASE-15224-branch-1.2.patch, 15224.branch-1.2.patch, > HBASE-15224.branch-1.2.patch, HBASE-15224v2-branch-1.2.patch > > > Remove an escape valve no longer needed after HBASE-15213. Remove so folks > don't ever have to worry their pretty-little heads about it. > Let the bulk of HBASE-15031 remain, the issue that added > "hbase.increment.fast.but.narrow.consistency" because it mostly cleanup we > want to keep. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[
https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137380#comment-15137380
]
Sean Busbey commented on HBASE-15221:
-
Yeah, I have it tee'd up on branch-1.1 and branch 1.0. Just waiting in case any
of the west-er timezones have input on this:
{quote}
bq. Before pushing to branch-1.1 I wanted to confirm that we're fine with
adding a method to the public API in a patch release?
Per the guidance last I read it, such a method falls is acceptable to add in a
patch-release. I am happy to be out-of-date on what the guidance is, though.
{quote}
> HTableMultiplexer improvements (stale region locations and resource leaks)
> --
>
> Key: HBASE-15221
> URL: https://issues.apache.org/jira/browse/HBASE-15221
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch,
> HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch
>
>
> It looks like HTableMultiplexer has a couple of issues.
> Upon failing to send a Put to the appropriate RS, the Put is re-queued back
> into the system. Normally this is fine as such an exception is transient and
> the Put would eventually succeed. However, in the case where the Put was
> rejected because of a NotServingRegionException (e.g. split, balance, merge),
> the re-queuing of the Put will end up using the same cached HRegionLocation.
> This means that the Put will just be repeatedly sent back to the same RS over
> and over again, eventually being dropped on the floor. Need to invalidate the
> location cache (or make sure we refresh it) when we re-queue the Put.
> The internal ClusterConnection is also leaked. If a user creates many
> HTableMultiplexers, they'll eventually run into issues (memory, zk
> connections, etc) because they'll never get cleaned up. HTableMultiplexer
> needs a close method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-15219) Canary tool does not return non-zero exit when one of region stuck state
[
https://issues.apache.org/jira/browse/HBASE-15219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137397#comment-15137397
]
Andrew Purtell edited comment on HBASE-15219 at 2/8/16 6:21 PM:
Can we report all errors and then exit? No need for a threshold, IMHO
Edit: The difference between a handful and half the regions not available would
be a different degree of partial availability, which might be measured and
reported. Where multitenant applications access different parts of the keyspace
with known prefixes, knowing which regions exactly in total are offline when
the canary runs gives us a precise snapshot of the number and identity of
applications/users possibly having availability issues at that time.
was (Author: apurtell):
Can we report all errors and then exit? No need for a threshold, IMHO
> Canary tool does not return non-zero exit when one of region stuck state
> -
>
> Key: HBASE-15219
> URL: https://issues.apache.org/jira/browse/HBASE-15219
> Project: HBase
> Issue Type: Bug
> Components: canary
>Affects Versions: 0.98.16
>Reporter: Vishal Khandelwal
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 0.98.18
>
> Attachments: HBASE-15219.v1.patch, HBASE-15219.v3.patch,
> HBASE-15219.v4.patch
>
>
> {code}
> 2016-02-05 12:24:18,571 ERROR [pool-2-thread-7] tool.Canary - read from
> region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> column family 0 failed
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
> attempts=2, exceptions:
> Fri Feb 05 12:24:15 GMT 2016,
> org.apache.hadoop.hbase.client.RpcRetryingCaller@54c9fea0,
> org.apache.hadoop.hbase.NotServingRegionException:
> org.apache.hadoop.hbase.NotServingRegionException: Region
> CAN_1,\x08\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00,1454667477865.00e77d07b8defe10704417fb99aa0418.
> is not online on isthbase02-dnds1-3-crd.eng.sfdc.net,60020,1454669984738
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2852)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4468)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2984)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:31186)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2149)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
> at java.lang.Thread.run(Thread.java:745)
>
> -bash-4.1$ echo $?
> 0
> {code}
> Below code prints the error but it does sets/returns the exit code. Due to
> this tool can't be integrated with nagios or other alerting.
> Ideally it should return error for failures. as pre the documentation:
>
> This tool will return non zero error codes to user for collaborating with
> other monitoring tools, such as Nagios. The error code definitions are:
> private static final int USAGE_EXIT_CODE = 1;
> private static final int INIT_ERROR_EXIT_CODE = 2;
> private static final int TIMEOUT_ERROR_EXIT_CODE = 3;
> private static final int ERROR_EXIT_CODE = 4;
>
> {code}
> org.apache.hadoop.hbase.tool.Canary.RegionTask
> public Void read() {
>
> try {
> table = connection.getTable(region.getTable());
> tableDesc = table.getTableDescriptor();
> } catch (IOException e) {
> LOG.debug("sniffRegion failed", e);
> sink.publishReadFailure(region, e);
>...
> return null;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[
https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137306#comment-15137306
]
Josh Elser commented on HBASE-15221:
Alright, I think I see what's happening now. One detail that I had omitted
(because I wasn't sure how it was relevant) is the following log message that
also accompanied the error:
{noformat}
2016-01-22 08:16:37.939 o.a.h.h.c.ConnectionManager$HConnectionImplementation
[WARN] Coding error, see method javadoc. row=[B@33364882, tableName=null
{noformat}
Turns out, this was critical as to why the region location cache wasn't getting
reloaded. Tracing through AsyncProcess, we can see the following path (starting
in HTableMultiplexer.FlushWorker):
{{AsyncProcess.submitMultiActions}}
{{AsyncRequestFutureImpl.sendMultiAction}}
{{SingleServerRequestRunnable.run}}
{{AsyncRequestFutureImpl.receiveMultiAction}}
{{ClusterConnection.updateCachedLocations}}
So, because HTableMultiplexer/AsyncProcess is sending around {{null}} for the
table name (since we don't necessarily have only one table), when AsyncProcess
goes to manage the failure, the implementation of {{updateCachedLocations}}
short-circuits because the table name is null.
Let me see if it would make sense to write an addendum that would be more in
line with how this is supposed to work from the AsyncProcess POV.
> HTableMultiplexer improvements (stale region locations and resource leaks)
> --
>
> Key: HBASE-15221
> URL: https://issues.apache.org/jira/browse/HBASE-15221
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch,
> HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch
>
>
> It looks like HTableMultiplexer has a couple of issues.
> Upon failing to send a Put to the appropriate RS, the Put is re-queued back
> into the system. Normally this is fine as such an exception is transient and
> the Put would eventually succeed. However, in the case where the Put was
> rejected because of a NotServingRegionException (e.g. split, balance, merge),
> the re-queuing of the Put will end up using the same cached HRegionLocation.
> This means that the Put will just be repeatedly sent back to the same RS over
> and over again, eventually being dropped on the floor. Need to invalidate the
> location cache (or make sure we refresh it) when we re-queue the Put.
> The internal ClusterConnection is also leaked. If a user creates many
> HTableMultiplexers, they'll eventually run into issues (memory, zk
> connections, etc) because they'll never get cleaned up. HTableMultiplexer
> needs a close method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15221) HTableMultiplexer improvements (stale region locations and resource leaks)
[
https://issues.apache.org/jira/browse/HBASE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137339#comment-15137339
]
Josh Elser commented on HBASE-15221:
I think the root cause is that {{AsyncProcess.receiveGlobalFailures}} isn't
getting invoked in the normal path (we fall into {{receiveMultiAction()}} --
the exceptions aren't raised, only passed along in the MultiResponse). For
consistency with what 0.98 is doing, we should clear the cache for the server
in AsyncProcess and not do this custom handling up in HTableMultiplexer.
Let me try to get an addendum on what was already applied (preemptive thanks
[~tedyu] and [~busbey] for getting pulled along by my sleuthing)
> HTableMultiplexer improvements (stale region locations and resource leaks)
> --
>
> Key: HBASE-15221
> URL: https://issues.apache.org/jira/browse/HBASE-15221
> Project: HBase
> Issue Type: Bug
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Critical
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.1.4
>
> Attachments: HBASE-15221.001.patch, HBASE-15221.002.patch,
> HBASE-15221.003.patch, HBASE-15221.branch-1.patch, HBASE-15221.v4.patch
>
>
> It looks like HTableMultiplexer has a couple of issues.
> Upon failing to send a Put to the appropriate RS, the Put is re-queued back
> into the system. Normally this is fine as such an exception is transient and
> the Put would eventually succeed. However, in the case where the Put was
> rejected because of a NotServingRegionException (e.g. split, balance, merge),
> the re-queuing of the Put will end up using the same cached HRegionLocation.
> This means that the Put will just be repeatedly sent back to the same RS over
> and over again, eventually being dropped on the floor. Need to invalidate the
> location cache (or make sure we refresh it) when we re-queue the Put.
> The internal ClusterConnection is also leaked. If a user creates many
> HTableMultiplexers, they'll eventually run into issues (memory, zk
> connections, etc) because they'll never get cleaned up. HTableMultiplexer
> needs a close method.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-15046) Perf test doing all mutation steps under row lock
[ https://issues.apache.org/jira/browse/HBASE-15046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15046. --- Resolution: Fixed Fix Version/s: 2.0.0 Release Note: In here we perf tested a realignment of the write pipeline and mvcc handling. Thought was that this work was a predicate for a general fix of HBASE-14460 (turns out, realignment of write path was not needed to fix the increment perf regression). The perf testing here made it so we were able to simplify writing. HBASE-15158 was just committed. This work is done. > Perf test doing all mutation steps under row lock > - > > Key: HBASE-15046 > URL: https://issues.apache.org/jira/browse/HBASE-15046 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack >Assignee: stack > Fix For: 2.0.0 > > Attachments: 1.2.svg, 1.2.v2.svg, 50threads_ycsb.png, > all_under_lock.patch, all_under_lock.svg, call_times_0-1_and_1-3_ycsb.png, > compare.png, gc.png, latencies.png, nopatch.png, > notpatched.workloadw.80percentwrites.json, ops.png, patched.png, > patched.workloadw.80percentwrites.json, writes.png > > > This issue is about perf testing a redo of the write pipeline so that rather > than: > * take rowlock > * start mvcc > * append to WAL > * add to memstore > * sync WAL > * let go of rowlock > * finish up mvcc > instead try... > * take rowlock > * start mvcc > * append to WAL > * sync WAL > * add to memstore > * finish up mvcc > * let go of rowlock > The latter is more straight-forward undoing need of rolling back memstore if > all does not succeed. > It might be slower though. This issue is a look-see/try it. > The redo will also help address the parent issue in a more general way so we > can do without the special-casing done for branch-1.0 and branch-1.1 done in > a sibling subtask. > Other benefits are that the current write pipeline is copy/pasted in a few > places -- in append, increment and checkand* -- and a refactor will allow us > to fix this duplication. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-15159) Fix merge of MVCC and SequenceID performance regression in branch-1.0 for checkAnd* and Append
[ https://issues.apache.org/jira/browse/HBASE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-15159. --- Resolution: Not A Problem Resolving as addressed by HBASE-15213 (good thing we didn't start in on this one [~chenheng]) > Fix merge of MVCC and SequenceID performance regression in branch-1.0 for > checkAnd* and Append > -- > > Key: HBASE-15159 > URL: https://issues.apache.org/jira/browse/HBASE-15159 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: stack > > Do the HBASE-15031 trick -- loosen reliance on mvcc for increments -- for > checkAnd* too in branch-1. Should be quick enough to do. Only prereq would > be HBASE-15157, tooling we have to do anyways, so we can show we've made > improvement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15201) Add hbase-spark to hbase assembly
[ https://issues.apache.org/jira/browse/HBASE-15201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137433#comment-15137433 ] Jerry He commented on HBASE-15201: -- Hi, folks If there is no more comments or objection, I'd like to get this in. Will the hbase-spark jar in the hbase lib, there are more choices to manipulate the spark-submit classpath to use the module. > Add hbase-spark to hbase assembly > - > > Key: HBASE-15201 > URL: https://issues.apache.org/jira/browse/HBASE-15201 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Attachments: HBASE-15201.patch > > > hbase-spark currently is missing from hbase assembly. > We should add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15201) Add hbase-spark to hbase assembly
[ https://issues.apache.org/jira/browse/HBASE-15201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-15201: Fix Version/s: 1.3.0 2.0.0 > Add hbase-spark to hbase assembly > - > > Key: HBASE-15201 > URL: https://issues.apache.org/jira/browse/HBASE-15201 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He >Priority: Minor > Fix For: 2.0.0, 1.3.0 > > Attachments: HBASE-15201.patch > > > hbase-spark currently is missing from hbase assembly. > We should add it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15231) Make TableState.State private
Misty Stanley-Jones created HBASE-15231: --- Summary: Make TableState.State private Key: HBASE-15231 URL: https://issues.apache.org/jira/browse/HBASE-15231 Project: HBase Issue Type: Bug Components: API Affects Versions: 2.0.0 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones TableState is private but TableState.State is evolving. This means that Javadoc links from TableState.State to TableState are broken. [~stack] says that TableState.State should be private. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15217) Canary -regionserver fails with cast exception
[ https://issues.apache.org/jira/browse/HBASE-15217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137264#comment-15137264 ] Hudson commented on HBASE-15217: FAILURE: Integrated in HBase-0.98-matrix #294 (See [https://builds.apache.org/job/HBase-0.98-matrix/294/]) HBASE-15217 Canary -regionserver fails with cast exception (Ted Yu) (apurtell: rev 2c126026cb7c735ad300e1363486791a330af18b) * hbase-server/src/main/java/org/apache/hadoop/hbase/tool/Canary.java > Canary -regionserver fails with cast exception > -- > > Key: HBASE-15217 > URL: https://issues.apache.org/jira/browse/HBASE-15217 > Project: HBase > Issue Type: Bug > Components: canary >Affects Versions: 0.98.16 >Reporter: Vishal Khandelwal >Assignee: Ted Yu >Priority: Critical > Fix For: 0.98.18 > > Attachments: HBASE-15217-0.98.v1.patch > > > ./bin/hbase org.apache.hadoop.hbase.tool.Canary -regionserver > Exception in thread "main" java.lang.ClassCastException: > org.apache.hadoop.hbase.tool.Canary$StdOutSink cannot be cast to > org.apache.hadoop.hbase.tool.Canary$ExtendedSink > at org.apache.hadoop.hbase.tool.Canary.newMonitor(Canary.java:622) > at org.apache.hadoop.hbase.tool.Canary.run(Canary.java:536) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.hbase.tool.Canary.main(Canary.java:1154) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-15232) Exceptions returned over multi RPC don't trigger region location reloads
Josh Elser created HBASE-15232:
--
Summary: Exceptions returned over multi RPC don't trigger region
location reloads
Key: HBASE-15232
URL: https://issues.apache.org/jira/browse/HBASE-15232
Project: HBase
Issue Type: Sub-task
Components: Client
Reporter: Josh Elser
Assignee: Josh Elser
Fix For: 2.0.0, 1.3.0, 1.2.1, 1.1.4
Follow-on for HBASE-15221:
A work-around was added in HTableMultiplexer to work around an issue that
AsyncProcess wasn't clearing the region location cache on Exception. This was
stemming from the issue that the {{tableName}} is {{null}} because
HTableMultiplexer is using the {{multi}} RPC. This causes an error that looks
like:
{noformat}
[WARN] Coding error, see method javadoc. row=[B@1673eff, tableName=null
{noformat}
HBASE-15221 should fix HTableMultiplexer, but it would be good to push the fix
down into AsyncProcess instead of using higher-level workarounds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15231) Make TableState.State private
[
https://issues.apache.org/jira/browse/HBASE-15231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15137390#comment-15137390
]
Hadoop QA commented on HBASE-15231:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} master passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} master passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
21m 50s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_72 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 45s
{color} | {color:green} hbase-client in the patch passed with JDK v1.8.0_72.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 56s
{color} | {color:green} hbase-client in the patch passed with JDK v1.7.0_95.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 23s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.9.1 Server=1.9.1 Image:yetus/hbase:date2016-02-08 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12786844/HBASE-15231.patch |
| JIRA Issue | HBASE-15231 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8bbcab979836 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3
