[jira] [Commented] (HBASE-11590) use a specific ThreadPoolExecutor
[ https://issues.apache.org/jira/browse/HBASE-11590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952237#comment-14952237 ] Nicolas Liochon commented on HBASE-11590: - > maybe just because it is more parsimonious in its thread use? That's the magic part: even of there is a single thread in the pool it's faster than the others. I didn't check if it consumes more CPU or not however. I will do the patch to use ForkJoin soon (hopefully today, if not next week). > use a specific ThreadPoolExecutor > - > > Key: HBASE-11590 > URL: https://issues.apache.org/jira/browse/HBASE-11590 > Project: HBase > Issue Type: Bug > Components: Client, Performance >Affects Versions: 1.0.0, 2.0.0 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Fix For: 2.0.0 > > Attachments: ExecutorServiceTest.java, > LifoThreadPoolExecutorSQP.java, UnitQueueP.java, UnitQueuePU.java, tp.patch > > > The JDK TPE creates all the threads in the pool. As a consequence, we create > (by default) 256 threads even if we just need a few. > The attached TPE create threads only if we have something in the queue. > On a PE test with replica on, it improved the 99 latency percentile by 5%. > Warning: there are likely some race conditions, but I'm posting it here > because there is may be an implementation available somewhere we can use, or > a good reason not to do that. So feedback welcome as usual. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-14580:
Attachment: hbase-14580.v2.patch
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0
>
> Attachments: hbase-14580.v2.patch, patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-14580:
Status: Patch Available (was: Open)
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0
>
> Attachments: hbase-14580.v2.patch, patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14579) Users authenticated with KERBEROS are recorded as being authenticated with SIMPLE
[
https://issues.apache.org/jira/browse/HBASE-14579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952250#comment-14952250
]
Nicolas Liochon commented on HBASE-14579:
-
I'm not sure I understand correctly the test patch script. Can I just change
the property
{code}
# All supported Hadoop versions that we want to test the compilation with
HADOOP2_VERSIONS="2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1"
{code}
to
{code}
HADOOP2_VERSIONS="2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0 2.7.1"
{code}
Or is there is a risk to hide a problem a patch could cause to the 0.98 release
(and even the 0.94)?
We will need to update the matrix in the hbase book as well...
> Users authenticated with KERBEROS are recorded as being authenticated with
> SIMPLE
> -
>
> Key: HBASE-14579
> URL: https://issues.apache.org/jira/browse/HBASE-14579
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 1.0.0, 1.2.0, 0.98.15
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: hbase-14579.patch
>
>
> That's the HBase version of HADOOP-10683.
> We see:
> ??hbase.Server (RpcServer.java:saslReadAndProcess(1446)) - Auth successful
> for securedUser/localh...@example.com (auth:SIMPLE)??
> while we would like to see:
> ??hbase.Server (RpcServer.java:saslReadAndProcess(1446)) - Auth successful
> for securedUser/localh...@example.com (auth:KERBEROS)??
> The fix is simple, but it means we need hadoop 2.5+.
> There is also a lot of cases where HBase calls "createUser" w/o specifying
> the authentication method... I don"'t have the solution for these ones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Liochon updated HBASE-14580:
Status: Open (was: Patch Available)
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0
>
> Attachments: patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952241#comment-14952241
]
Nicolas Liochon commented on HBASE-14580:
-
> I think it's just a log spam issue – see HADOOP-12450.
Oh, yeah. Thanks for the pointer.
> Instead of hard-coding the config values, can you use
> User.isHBaseSecurityEnabled(c)?
Yes, you're right. I updated the patch.
> The username suffixes were fed into the data dirs used by each DN/RS's for a
> "distributed" minicluster setup
> So, as I understand it, that would not be an issue here are Kerberos would
> only be supported with a single node setup?
I'm not sure here: I don't see why the user name is needed in the data dirs.
But in any case, this patch does not break anything, as the suffix approach
clashes with the kerberos realm...
> As Gary said, this is harmless unless you're test depends on a valid group
> membership.
Thanks.
Thanks for the reviews, all. I will commit the v2 if hadoop-qa passes with the
v2.
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0
>
> Attachments: patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC Server handler 9 on 61366]
> security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1521))
> - No groups available for user securedUser' I'm not sure of its impact.
> [~apurtell], what do you think?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14479) Apply the Leader/Followers pattern to RpcServer's Reader
[
https://issues.apache.org/jira/browse/HBASE-14479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952235#comment-14952235
]
Nicolas Liochon commented on HBASE-14479:
-
Yeah, I tried to get rid of this array of readers a while back, but I didn't
push the patch because I didn't get any significant result. Nice work, [~ikeda]
> Apply the Leader/Followers pattern to RpcServer's Reader
>
>
> Key: HBASE-14479
> URL: https://issues.apache.org/jira/browse/HBASE-14479
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-14479-V2 (1).patch, HBASE-14479-V2.patch,
> HBASE-14479-V2.patch, HBASE-14479.patch, gc.png, gets.png, io.png, median.png
>
>
> {{RpcServer}} uses multiple selectors to read data for load distribution, but
> the distribution is just done by round-robin. It is uncertain, especially for
> long run, whether load is equally divided and resources are used without
> being wasted.
> Moreover, multiple selectors may cause excessive context switches which give
> priority to low latency (while we just add the requests to queues), and it is
> possible to reduce throughput of the whole server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14580) Make the HBaseMiniCluster compliant with Kerberos
[
https://issues.apache.org/jira/browse/HBASE-14580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952278#comment-14952278
]
Hadoop QA commented on HBASE-14580:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766017/hbase-14580.v2.patch
against master branch at commit 587f5bc11f9d5d37557baf36c7df110af860a95c.
ATTACHMENT ID: 12766017
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestWALLockup
{color:red}-1 core zombie tests{color}. There are 5 zombie test(s):
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:292)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:266)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:198)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:292)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:266)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemStateWithSkipTmp(TestExportSnapshot.java:204)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:292)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testExportFileSystemState(TestExportSnapshot.java:266)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:260)
at
org.apache.hadoop.hbase.snapshot.TestExportSnapshot.testSnapshotWithRefsExportFileSystemState(TestExportSnapshot.java:240)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15954//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15954//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15954//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15954//console
This message is automatically generated.
> Make the HBaseMiniCluster compliant with Kerberos
> -
>
> Key: HBASE-14580
> URL: https://issues.apache.org/jira/browse/HBASE-14580
> Project: HBase
> Issue Type: Improvement
> Components: security, test
>Affects Versions: 2.0.0
>Reporter: Nicolas Liochon
>Assignee: Nicolas Liochon
> Fix For: 2.0.0
>
> Attachments: hbase-14580.v2.patch, patch-14580.v1.patch
>
>
> Whne using MiniKDC and the minicluster in a unit test, there is a conflict
> causeed by HBaseTestingUtility:
> {code}
> public static User getDifferentUser(final Configuration c,
> final String differentiatingSuffix)
> throws IOException {
>// snip
> String username = User.getCurrent().getName() +
> differentiatingSuffix; < problem here
> User user = User.createUserForTesting(c, username,
> new String[]{"supergroup"});
> return user;
> }
> {code}
> This creates users like securedUser/localh...@example.com.hfs.0, and this
> does not work.
> My fix is to return the current user when Kerberos is set. I don't think that
> there is another option (any other opinion?). However this user is not in a
> group so we have logs like 'WARN [IPC
[jira] [Commented] (HBASE-7755) Experiment with LAB in BlockEndcoding
[ https://issues.apache.org/jira/browse/HBASE-7755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952406#comment-14952406 ] Lars Hofhansl commented on HBASE-7755: -- Just thought about this again. It's interesting that when I cannot find a good slab size here, how come we can find a good slab size for the memstore? I want to think about this again. > Experiment with LAB in BlockEndcoding > - > > Key: HBASE-7755 > URL: https://issues.apache.org/jira/browse/HBASE-7755 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 7755-0.94-WORK_IN_PROGRESS.txt, 7755-0.94-W_I_P_v1.txt > > > I was looking at and profiling the BlockEncoding code to figure out how to > make it faster. One issue that jumped out was we call > ByteBuffer.allocate(...) for each single KV. > As an experiment I tried using the MemStoreLAB code to allocate those buffers. > Here are some preliminary numbers, all scanning 10m rows (all in cache): > * no encoding: 5.2s > * FAST_DIFF without patch: 7.3s > * FAST_DIFF with patch and small LAB: 4.1s > * FAST_DIFF with patch and large LAB: 11s > So this is very sensitive to the right sizing of the LAB. > Need to do a bit more testing, but it seems that there is a chance to > actually make scanning with block encoding faster than without! -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14584) TestNamespacesInstanceModel fails on jdk8
[
https://issues.apache.org/jira/browse/HBASE-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Warhaftig updated HBASE-14584:
---
Attachment: hbase-14584-v1.patch
Attached patch 'hbase-14584-v1.patch' does the following:
- Remove alphabetic XML accessor ordering from NamespacesInstanceModel because
it will have no effect with only one element name used for all properties.
- Modify TestModelBase#testToXML() to reconvert to the model for verification
since XML element ordering is not guaranteed.
- Reenable TestNamespacesInstanceModel#testFromXML() as it wouldn't be affected
by XML element ordering and never seemed to have failed.
> TestNamespacesInstanceModel fails on jdk8
> -
>
> Key: HBASE-14584
> URL: https://issues.apache.org/jira/browse/HBASE-14584
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Reporter: Nick Dimiduk
>Assignee: Matt Warhaftig
> Attachments: 14584.00.patch, disable.txt, hbase-14584-v1.patch
>
>
> Noticed this in the [build
> output|https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/235/consoleFull]
> of HBASE-12911. Seems this test has been failing for a long time, got all
> the way back to {{6534583}}, {{master~44}} and it's still failing there. I
> guess tests usually fail in {{hbase-server}}, so we don't often get to
> {{hbase-rest}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14560) TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
[
https://issues.apache.org/jira/browse/HBASE-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Warhaftig reassigned HBASE-14560:
--
Assignee: Matt Warhaftig
> TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
>
>
> Key: HBASE-14560
> URL: https://issues.apache.org/jira/browse/HBASE-14560
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Matt Warhaftig
>Priority: Minor
>
> From
> https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/237/consoleFull
> :
> {code}
> org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel
> testToXML(org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel)
> Time elapsed: 0.017 sec <<< FAILURE!
> junit.framework.ComparisonFailure:
> expected:<...perties>[NAMEtestNamespaceKEY_2VALUE_2KEY_1VALUE_1]
> but
> was:<...perties>[KEY_1VALUE_1KEY_2VALUE_2NAMEtestNamespace]
> at junit.framework.Assert.assertEquals(Assert.java:100)
> at junit.framework.Assert.assertEquals(Assert.java:107)
> at junit.framework.TestCase.assertEquals(TestCase.java:269)
> at
> org.apache.hadoop.hbase.rest.model.TestModelBase.testToXML(TestModelBase.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> The above test failure can be reproduced locally.
> It was likely caused by the different behavior w.r.t. JAXBContext between JDK
> 1.7 and 1.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14560) TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
[
https://issues.apache.org/jira/browse/HBASE-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952501#comment-14952501
]
Matt Warhaftig commented on HBASE-14560:
Thanks Ted for finding and reporting this issue.
A later ticket, HBASE-14584, got more love and a potential patch even though
your ticket is earlier.
> TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
>
>
> Key: HBASE-14560
> URL: https://issues.apache.org/jira/browse/HBASE-14560
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Matt Warhaftig
>Priority: Minor
>
> From
> https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/237/consoleFull
> :
> {code}
> org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel
> testToXML(org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel)
> Time elapsed: 0.017 sec <<< FAILURE!
> junit.framework.ComparisonFailure:
> expected:<...perties>[NAMEtestNamespaceKEY_2VALUE_2KEY_1VALUE_1]
> but
> was:<...perties>[KEY_1VALUE_1KEY_2VALUE_2NAMEtestNamespace]
> at junit.framework.Assert.assertEquals(Assert.java:100)
> at junit.framework.Assert.assertEquals(Assert.java:107)
> at junit.framework.TestCase.assertEquals(TestCase.java:269)
> at
> org.apache.hadoop.hbase.rest.model.TestModelBase.testToXML(TestModelBase.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> The above test failure can be reproduced locally.
> It was likely caused by the different behavior w.r.t. JAXBContext between JDK
> 1.7 and 1.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14560) TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
[
https://issues.apache.org/jira/browse/HBASE-14560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu resolved HBASE-14560.
Resolution: Duplicate
> TestNamespacesInstanceModel#testToXML fails when JDK 1.8 is used
>
>
> Key: HBASE-14560
> URL: https://issues.apache.org/jira/browse/HBASE-14560
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Matt Warhaftig
>Priority: Minor
>
> From
> https://builds.apache.org/job/HBase-1.3/jdk=latest1.8,label=Hadoop/237/consoleFull
> :
> {code}
> org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel
> testToXML(org.apache.hadoop.hbase.rest.model.TestNamespacesInstanceModel)
> Time elapsed: 0.017 sec <<< FAILURE!
> junit.framework.ComparisonFailure:
> expected:<...perties>[NAMEtestNamespaceKEY_2VALUE_2KEY_1VALUE_1]
> but
> was:<...perties>[KEY_1VALUE_1KEY_2VALUE_2NAMEtestNamespace]
> at junit.framework.Assert.assertEquals(Assert.java:100)
> at junit.framework.Assert.assertEquals(Assert.java:107)
> at junit.framework.TestCase.assertEquals(TestCase.java:269)
> at
> org.apache.hadoop.hbase.rest.model.TestModelBase.testToXML(TestModelBase.java:115)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> The above test failure can be reproduced locally.
> It was likely caused by the different behavior w.r.t. JAXBContext between JDK
> 1.7 and 1.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14584) TestNamespacesInstanceModel fails on jdk8
[
https://issues.apache.org/jira/browse/HBASE-14584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952523#comment-14952523
]
Hadoop QA commented on HBASE-14584:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766039/hbase-14584-v1.patch
against master branch at commit 587f5bc11f9d5d37557baf36c7df110af860a95c.
ATTACHMENT ID: 12766039
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15955//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15955//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15955//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15955//console
This message is automatically generated.
> TestNamespacesInstanceModel fails on jdk8
> -
>
> Key: HBASE-14584
> URL: https://issues.apache.org/jira/browse/HBASE-14584
> Project: HBase
> Issue Type: Test
> Components: REST, test
>Reporter: Nick Dimiduk
>Assignee: Matt Warhaftig
> Attachments: 14584.00.patch, disable.txt, hbase-14584-v1.patch
>
>
> Noticed this in the [build
> output|https://builds.apache.org/job/HBase-1.2/jdk=latest1.8,label=Hadoop/235/consoleFull]
> of HBASE-12911. Seems this test has been failing for a long time, got all
> the way back to {{6534583}}, {{master~44}} and it's still failing there. I
> guess tests usually fail in {{hbase-server}}, so we don't often get to
> {{hbase-rest}} module.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Priority: Blocker (was: Major)
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952609#comment-14952609
]
ramkrishna.s.vasudevan commented on HBASE-14221:
Thanks for the review. Yes that is how it has to be - but there is one
difference in how this works. Due to various conditions and cases I thought of
having 3 cases - INIT, NOT_IN_NEXT_ROW and NEXT_ROW.
So in case there are 2 CFs and the 2 cells are 'row1:cf1:q1' and 'row1:cf2:q1'.
(assume we have row2:cf1:q1 and row2:cf2:q1).
So once the first cell with cf1 is fetched that storeScanner would be moved to
nextRow as it is currently at row2. But for fetching row1:cf2 that
storeSCanner is still at INIT state and so we will be doing the matchingRows
there and that will also be moved to nextRow. But at the KVHeap having these
StoreScanners will next fetch row2:cf1 but since this scanner is the 'current'
StoreScanner and already in nextRow it will avoid the compare this time.
Internally at the StoreScanner level also we are avoiding the matchingrows to
set the SQM with the current row.
Why this INIT state is also needed is that, some times though there are two CFs
but the rows in the 2nd CF may always be lesser than the row fetched out from
the CF1. So the KVHeap holding the StoreScanners may not actually use the 2nd
CF at all so that point we really don know if that CF has crossed the current
row it is pointing to.
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952610#comment-14952610
]
Anoop Sam John commented on HBASE-14557:
This issue is there in branch-1.2 also and must get fixed before the release.
Just making it blocker. Will attach patch version fixes soon.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14590) Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name
Anoop Sam John created HBASE-14590: -- Summary: Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name Key: HBASE-14590 URL: https://issues.apache.org/jira/browse/HBASE-14590 Project: HBase Issue Type: Sub-task Affects Versions: 2.0.0 Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0 As per the discussion under HBASE-14398, shorten these big method names to getXXXPosition -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14326) HBase book: fix definition of max min size to compact
[
https://issues.apache.org/jira/browse/HBASE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-14326:
Attachment: HBASE-14326.patch
Hopefully this patch clarifies the docs.
> HBase book: fix definition of max min size to compact
> -
>
> Key: HBASE-14326
> URL: https://issues.apache.org/jira/browse/HBASE-14326
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Vladimir Rodionov
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14326.patch
>
>
> I think we need to change wording/definition of these config parameters in
> HBase book, they are misleading:
> {quote}
> hbase.hstore.compaction.min.size
> Description
> A StoreFile smaller than this size will always be eligible for minor
> compaction. HFiles this size or larger are evaluated by
> hbase.hstore.compaction.ratio to determine if they are eligible. Because this
> limit represents the "automatic include"limit for all StoreFiles smaller than
> this value, this value may need to be reduced in write-heavy environments
> where many StoreFiles in the 1-2 MB range are being flushed, because every
> StoreFile will be targeted for compaction and the resulting StoreFiles may
> still be under the minimum size and require further compaction. If this
> parameter is lowered, the ratio check is triggered more quickly. This
> addressed some issues seen in earlier versions of HBase but changing this
> parameter is no longer necessary in most situations. Default: 128 MB
> expressed in bytes.
> Default
> 134217728
> hbase.hstore.compaction.max.size
> Description
> A StoreFile larger than this size will be excluded from compaction. The
> effect of raising hbase.hstore.compaction.max.size is fewer, larger
> StoreFiles that do not get compacted often. If you feel that compaction is
> happening too often without much benefit, you can try raising this value.
> Default: the value of LONG.MAX_VALUE, expressed in bytes.
> hbase.hstore.compaction.ratio
> Description
> For minor compaction, this ratio is used to determine whether a given
> StoreFile which is larger than hbase.hstore.compaction.min.size is eligible
> for compaction. Its effect is to limit compaction of large StoreFiles. The
> value of hbase.hstore.compaction.ratio is expressed as a floating-point
> decimal. A large ratio, such as 10, will produce a single giant StoreFile.
> Conversely, a low value, such as .25, will produce behavior similar to the
> BigTable compaction algorithm, producing four StoreFiles. A moderate value of
> between 1.0 and 1.4 is recommended. When tuning this value, you are balancing
> write costs with read costs. Raising the value (to something like 1.4) will
> have more write costs, because you will compact larger StoreFiles. However,
> during reads, HBase will need to seek through fewer StoreFiles to accomplish
> the read. Consider this approach if you cannot take advantage of Bloom
> filters. Otherwise, you can lower this value to something like 1.0 to reduce
> the background cost of writes, and use Bloom filters to control the number of
> StoreFiles touched during reads. For most cases, the default value is
> appropriate.
> Default
> 1.2F
> {quote}
> For details, see HBASE-14263.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14326) HBase book: fix definition of max min size to compact
[
https://issues.apache.org/jira/browse/HBASE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones updated HBASE-14326:
Status: Patch Available (was: Open)
> HBase book: fix definition of max min size to compact
> -
>
> Key: HBASE-14326
> URL: https://issues.apache.org/jira/browse/HBASE-14326
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Vladimir Rodionov
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
> Attachments: HBASE-14326.patch
>
>
> I think we need to change wording/definition of these config parameters in
> HBase book, they are misleading:
> {quote}
> hbase.hstore.compaction.min.size
> Description
> A StoreFile smaller than this size will always be eligible for minor
> compaction. HFiles this size or larger are evaluated by
> hbase.hstore.compaction.ratio to determine if they are eligible. Because this
> limit represents the "automatic include"limit for all StoreFiles smaller than
> this value, this value may need to be reduced in write-heavy environments
> where many StoreFiles in the 1-2 MB range are being flushed, because every
> StoreFile will be targeted for compaction and the resulting StoreFiles may
> still be under the minimum size and require further compaction. If this
> parameter is lowered, the ratio check is triggered more quickly. This
> addressed some issues seen in earlier versions of HBase but changing this
> parameter is no longer necessary in most situations. Default: 128 MB
> expressed in bytes.
> Default
> 134217728
> hbase.hstore.compaction.max.size
> Description
> A StoreFile larger than this size will be excluded from compaction. The
> effect of raising hbase.hstore.compaction.max.size is fewer, larger
> StoreFiles that do not get compacted often. If you feel that compaction is
> happening too often without much benefit, you can try raising this value.
> Default: the value of LONG.MAX_VALUE, expressed in bytes.
> hbase.hstore.compaction.ratio
> Description
> For minor compaction, this ratio is used to determine whether a given
> StoreFile which is larger than hbase.hstore.compaction.min.size is eligible
> for compaction. Its effect is to limit compaction of large StoreFiles. The
> value of hbase.hstore.compaction.ratio is expressed as a floating-point
> decimal. A large ratio, such as 10, will produce a single giant StoreFile.
> Conversely, a low value, such as .25, will produce behavior similar to the
> BigTable compaction algorithm, producing four StoreFiles. A moderate value of
> between 1.0 and 1.4 is recommended. When tuning this value, you are balancing
> write costs with read costs. Raising the value (to something like 1.4) will
> have more write costs, because you will compact larger StoreFiles. However,
> during reads, HBase will need to seek through fewer StoreFiles to accomplish
> the read. Consider this approach if you cannot take advantage of Bloom
> filters. Otherwise, you can lower this value to something like 1.0 to reduce
> the background cost of writes, and use Bloom filters to control the number of
> StoreFiles touched during reads. For most cases, the default value is
> appropriate.
> Default
> 1.2F
> {quote}
> For details, see HBASE-14263.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952649#comment-14952649
]
ramkrishna.s.vasudevan commented on HBASE-14221:
Stack, not getting this
bq.Comment from left field: ?
You mean the left hand side compare?
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14582) Regionserver status webpage bucketcache list can become huge
[ https://issues.apache.org/jira/browse/HBASE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-14582: --- Priority: Major (was: Trivial) > Regionserver status webpage bucketcache list can become huge > > > Key: HBASE-14582 > URL: https://issues.apache.org/jira/browse/HBASE-14582 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: James Hartshorn >Assignee: stack > Attachments: 14582.txt, Screen Shot 2015-10-09 at 8.00.37 PM.png, > Screen Shot 2015-10-09 at 8.00.37 PM.png > > > The regionserver status page, such as http://127.0.0.1:60030/rs-status always > downloads information about every bucket in the cache. In some cases this > can be hundreds of thousands of buckets, causing megabytes of info to need to > be downloaded and significant browser instability and memory usage: > wc -l HBase-Region-Server-hb22.html > 2010116 HBase-Region-Server-hb22.html > ls -lah HBase-Region-Server-hb22.html > 32M Oct 6 19:23 HBase-Region-Server-hb22.html > Firefox "about:memory": > 1,330.18 MB (48.22%) -- top(http://hb22:60030/rs-status#bc_l2, id=2010) > 1,329.61 MB (48.20%) -- active/window(http://hb22:60030/rs-status#bc_l2) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14532) [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config and not described in section 7
[ https://issues.apache.org/jira/browse/HBASE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-14532: --- Assignee: Misty Stanley-Jones > [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config > and not described in section 7 > - > > Key: HBASE-14532 > URL: https://issues.apache.org/jira/browse/HBASE-14532 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Rick Moritz >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14532.patch > > > After trying to figure out whether shortcircuit reads would work on my > system, I studied the book and found conflicting information. > It's suggested in section 92.2, that dfs.client.read.shortcircuit is an > option in hbase-site.xml, but the supposedly complete default configuration > in section 7 does not include this setting. This leads to confusion on > whether it's sufficient to enable this setting in hdfs-site.xml, or whether > it needs to be added to both configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14532) [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config and not described in section 7
[ https://issues.apache.org/jira/browse/HBASE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14532: Fix Version/s: 2.0.0 Status: Patch Available (was: Open) > [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config > and not described in section 7 > - > > Key: HBASE-14532 > URL: https://issues.apache.org/jira/browse/HBASE-14532 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Rick Moritz >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14532.patch > > > After trying to figure out whether shortcircuit reads would work on my > system, I studied the book and found conflicting information. > It's suggested in section 92.2, that dfs.client.read.shortcircuit is an > option in hbase-site.xml, but the supposedly complete default configuration > in section 7 does not include this setting. This leads to confusion on > whether it's sufficient to enable this setting in hdfs-site.xml, or whether > it needs to be added to both configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14532) [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config and not described in section 7
[ https://issues.apache.org/jira/browse/HBASE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14532: Attachment: HBASE-14532.patch Thanks for these reports, [~RPCMoritz]. Hopefully this patch will address your concerns. > [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config > and not described in section 7 > - > > Key: HBASE-14532 > URL: https://issues.apache.org/jira/browse/HBASE-14532 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Rick Moritz >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14532.patch > > > After trying to figure out whether shortcircuit reads would work on my > system, I studied the book and found conflicting information. > It's suggested in section 92.2, that dfs.client.read.shortcircuit is an > option in hbase-site.xml, but the supposedly complete default configuration > in section 7 does not include this setting. This leads to confusion on > whether it's sufficient to enable this setting in hdfs-site.xml, or whether > it needs to be added to both configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14532) [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config and not described in section 7
[ https://issues.apache.org/jira/browse/HBASE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952623#comment-14952623 ] Rick Moritz commented on HBASE-14532: - Looks good, many thanks. > [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config > and not described in section 7 > - > > Key: HBASE-14532 > URL: https://issues.apache.org/jira/browse/HBASE-14532 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Rick Moritz >Assignee: Misty Stanley-Jones >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14532.patch > > > After trying to figure out whether shortcircuit reads would work on my > system, I studied the book and found conflicting information. > It's suggested in section 92.2, that dfs.client.read.shortcircuit is an > option in hbase-site.xml, but the supposedly complete default configuration > in section 7 does not include this setting. This leads to confusion on > whether it's sufficient to enable this setting in hdfs-site.xml, or whether > it needs to be added to both configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14221) Reduce the number of time row comparison is done in a Scan
[
https://issues.apache.org/jira/browse/HBASE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952638#comment-14952638
]
stack commented on HBASE-14221:
---
Comment from left field: I've been profiling this w/e... and cutting down on
the amount of comparing is the most effective means of cutting CPU consumed
returning results... just stating what is obvious to most but I just tripped
over it again, anew..
> Reduce the number of time row comparison is done in a Scan
> --
>
> Key: HBASE-14221
> URL: https://issues.apache.org/jira/browse/HBASE-14221
> Project: HBase
> Issue Type: Sub-task
> Components: Scanners
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
> Fix For: 2.0.0
>
> Attachments: 14221-0.98-takeALook.txt, HBASE-14221.patch,
> HBASE-14221_1.patch, HBASE-14221_1.patch, HBASE-14221_6.patch,
> withmatchingRowspatch.png, withoutmatchingRowspatch.png
>
>
> When we tried to do some profiling with the PE tool found this.
> Currently we do row comparisons in 3 places in a simple Scan case.
> 1) ScanQueryMatcher
> {code}
>int ret = this.rowComparator.compareRows(curCell, cell);
> if (!this.isReversed) {
> if (ret <= -1) {
> return MatchCode.DONE;
> } else if (ret >= 1) {
> // could optimize this, if necessary?
> // Could also be called SEEK_TO_CURRENT_ROW, but this
> // should be rare/never happens.
> return MatchCode.SEEK_NEXT_ROW;
> }
> } else {
> if (ret <= -1) {
> return MatchCode.SEEK_NEXT_ROW;
> } else if (ret >= 1) {
> return MatchCode.DONE;
> }
> }
> {code}
> 2) In StoreScanner next() while starting to scan the row
> {code}
> if (!scannerContext.hasAnyLimit(LimitScope.BETWEEN_CELLS) ||
> matcher.curCell == null ||
> isNewRow || !CellUtil.matchingRow(peeked, matcher.curCell)) {
> this.countPerRow = 0;
> matcher.setToNewRow(peeked);
> }
> {code}
> Particularly to see if we are in a new row.
> 3) In HRegion
> {code}
> scannerContext.setKeepProgress(true);
> heap.next(results, scannerContext);
> scannerContext.setKeepProgress(tmpKeepProgress);
> nextKv = heap.peek();
> moreCellsInRow = moreCellsInRow(nextKv, currentRowCell);
> {code}
> Here again there are cases where we need to careful for a MultiCF case. Was
> trying to solve this for the MultiCF case but is having lot of cases to
> solve. But atleast for a single CF case I think these comparison can be
> reduced.
> So for a single CF case in the SQM we are able to find if we have crossed a
> row using the code pasted above in SQM. That comparison is definitely needed.
> Now in case of a single CF the HRegion is going to have only one element in
> the heap and so the 3rd comparison can surely be avoided if the
> StoreScanner.next() was over due to MatchCode.DONE caused by SQM.
> Coming to the 2nd compareRows that we do in StoreScanner. next() - even that
> can be avoided if we know that the previous next() call was over due to a new
> row. Doing all this I found that the compareRows in the profiler which was
> 19% got reduced to 13%. Initially we can solve for single CF case which can
> be extended to MultiCF cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14420) Zombie Stomping Session
[ https://issues.apache.org/jira/browse/HBASE-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14420: -- Attachment: none_fix.txt Try > Zombie Stomping Session > --- > > Key: HBASE-14420 > URL: https://issues.apache.org/jira/browse/HBASE-14420 > Project: HBase > Issue Type: Umbrella > Components: test >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: hangers.txt, none_fix (1).txt, none_fix.txt, > none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt, none_fix.txt > > > Patch build are now failing most of the time because we are dropping zombies. > I confirm we are doing this on non-apache build boxes too. > Left-over zombies consume resources on build boxes (OOME cannot create native > threads). Having to do multiple test runs in the hope that we can get a > non-zombie-making build or making (arbitrary) rulings that the zombies are > 'not related' is a productivity sink. And so on... > This is an umbrella issue for a zombie stomping session that started earlier > this week. Will hang sub-issues of this one. Am running builds back-to-back > on little cluster to turn out the monsters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14532) [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config and not described in section 7
[
https://issues.apache.org/jira/browse/HBASE-14532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952651#comment-14952651
]
Hadoop QA commented on HBASE-14532:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12766054/HBASE-14532.patch
against master branch at commit 587f5bc11f9d5d37557baf36c7df110af860a95c.
ATTACHMENT ID: 12766054
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15958//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15958//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15958//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15958//console
This message is automatically generated.
> [book] dfs.client.read.shortcircuit is referenced as hbase-site.xml config
> and not described in section 7
> -
>
> Key: HBASE-14532
> URL: https://issues.apache.org/jira/browse/HBASE-14532
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Rick Moritz
>Assignee: Misty Stanley-Jones
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-14532.patch
>
>
> After trying to figure out whether shortcircuit reads would work on my
> system, I studied the book and found conflicting information.
> It's suggested in section 92.2, that dfs.client.read.shortcircuit is an
> option in hbase-site.xml, but the supposedly complete default configuration
> in section 7 does not include this setting. This leads to confusion on
> whether it's sufficient to enable this setting in hdfs-site.xml, or whether
> it needs to be added to both configurations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Attachment: HBASE-14557_branch-1.2.patch
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14590) Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name
[ https://issues.apache.org/jira/browse/HBASE-14590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-14590: --- Attachment: HBASE-14590.patch > Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name > --- > > Key: HBASE-14590 > URL: https://issues.apache.org/jira/browse/HBASE-14590 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-14590.patch > > > As per the discussion under HBASE-14398, shorten these big method names to > getXXXPosition -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14590) Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name
[ https://issues.apache.org/jira/browse/HBASE-14590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-14590: --- Status: Patch Available (was: Open) > Shorten ByteBufferedCell#getXXXPositionInByteBuffer method name > --- > > Key: HBASE-14590 > URL: https://issues.apache.org/jira/browse/HBASE-14590 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-14590.patch > > > As per the discussion under HBASE-14398, shorten these big method names to > getXXXPosition -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14582) Regionserver status webpage bucketcache list can become huge
[ https://issues.apache.org/jira/browse/HBASE-14582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952668#comment-14952668 ] Anoop Sam John commented on HBASE-14582: +1 foe removal of the individual bucket level details. The overall usage and free space info should be fine enough +1 for patch. > Regionserver status webpage bucketcache list can become huge > > > Key: HBASE-14582 > URL: https://issues.apache.org/jira/browse/HBASE-14582 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: James Hartshorn >Assignee: stack >Priority: Trivial > Attachments: 14582.txt, Screen Shot 2015-10-09 at 8.00.37 PM.png, > Screen Shot 2015-10-09 at 8.00.37 PM.png > > > The regionserver status page, such as http://127.0.0.1:60030/rs-status always > downloads information about every bucket in the cache. In some cases this > can be hundreds of thousands of buckets, causing megabytes of info to need to > be downloaded and significant browser instability and memory usage: > wc -l HBase-Region-Server-hb22.html > 2010116 HBase-Region-Server-hb22.html > ls -lah HBase-Region-Server-hb22.html > 32M Oct 6 19:23 HBase-Region-Server-hb22.html > Firefox "about:memory": > 1,330.18 MB (48.22%) -- top(http://hb22:60030/rs-status#bc_l2, id=2010) > 1,329.61 MB (48.20%) -- active/window(http://hb22:60030/rs-status#bc_l2) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14326) HBase book: fix definition of max min size to compact
[
https://issues.apache.org/jira/browse/HBASE-14326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Misty Stanley-Jones reassigned HBASE-14326:
---
Assignee: Misty Stanley-Jones
> HBase book: fix definition of max min size to compact
> -
>
> Key: HBASE-14326
> URL: https://issues.apache.org/jira/browse/HBASE-14326
> Project: HBase
> Issue Type: Bug
> Components: documentation
>Reporter: Vladimir Rodionov
>Assignee: Misty Stanley-Jones
> Fix For: 2.0.0
>
>
> I think we need to change wording/definition of these config parameters in
> HBase book, they are misleading:
> {quote}
> hbase.hstore.compaction.min.size
> Description
> A StoreFile smaller than this size will always be eligible for minor
> compaction. HFiles this size or larger are evaluated by
> hbase.hstore.compaction.ratio to determine if they are eligible. Because this
> limit represents the "automatic include"limit for all StoreFiles smaller than
> this value, this value may need to be reduced in write-heavy environments
> where many StoreFiles in the 1-2 MB range are being flushed, because every
> StoreFile will be targeted for compaction and the resulting StoreFiles may
> still be under the minimum size and require further compaction. If this
> parameter is lowered, the ratio check is triggered more quickly. This
> addressed some issues seen in earlier versions of HBase but changing this
> parameter is no longer necessary in most situations. Default: 128 MB
> expressed in bytes.
> Default
> 134217728
> hbase.hstore.compaction.max.size
> Description
> A StoreFile larger than this size will be excluded from compaction. The
> effect of raising hbase.hstore.compaction.max.size is fewer, larger
> StoreFiles that do not get compacted often. If you feel that compaction is
> happening too often without much benefit, you can try raising this value.
> Default: the value of LONG.MAX_VALUE, expressed in bytes.
> hbase.hstore.compaction.ratio
> Description
> For minor compaction, this ratio is used to determine whether a given
> StoreFile which is larger than hbase.hstore.compaction.min.size is eligible
> for compaction. Its effect is to limit compaction of large StoreFiles. The
> value of hbase.hstore.compaction.ratio is expressed as a floating-point
> decimal. A large ratio, such as 10, will produce a single giant StoreFile.
> Conversely, a low value, such as .25, will produce behavior similar to the
> BigTable compaction algorithm, producing four StoreFiles. A moderate value of
> between 1.0 and 1.4 is recommended. When tuning this value, you are balancing
> write costs with read costs. Raising the value (to something like 1.4) will
> have more write costs, because you will compact larger StoreFiles. However,
> during reads, HBase will need to seek through fewer StoreFiles to accomplish
> the read. Consider this approach if you cannot take advantage of Bloom
> filters. Otherwise, you can lower this value to something like 1.0 to reduce
> the background cost of writes, and use Bloom filters to control the number of
> StoreFiles touched during reads. For most cases, the default value is
> appropriate.
> Default
> 1.2F
> {quote}
> For details, see HBASE-14263.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Attachment: HBASE-14557_branch-1.patch
In branch-1 and branch-1.2 KVUtil#ensureKV() is used by other parts of code
other than MR. So just adding a new method which is doing the extra check of
class name and adding that in MR area. Other parts of code still uses the old
ensureKV() method.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Attachment: HBASE-14557_V2.patch
V1 patch had one issue where I was not setting the seqId into the new KV
created. Fixing that in this patch
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14556) Make prefetchOnOpen configurable for index and/or data blocks
[
https://issues.apache.org/jira/browse/HBASE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952628#comment-14952628
]
Andrew Purtell commented on HBASE-14556:
We changed KEEP_DELETED_CELLS from a boolean to an end on HBASE-12363. Although
that was a schema option not a CacheConfig option, perhaps we could do
something similar here for prefetchOnOpen.
> Make prefetchOnOpen configurable for index and/or data blocks
> -
>
> Key: HBASE-14556
> URL: https://issues.apache.org/jira/browse/HBASE-14556
> Project: HBase
> Issue Type: Bug
> Components: BlockCache, regionserver
>Reporter: Lars George
>
> This came up in user discussions. It would be great to add an extra option to
> the {{CacheConfig}} that allows to specify what blocks are cached during
> region/file opening. This should allows to set {{BlockIndexOnly}},
> {{BloomFilterOnly}}, {{AllIndexesOnly}}, {{DataOnly}}, and {{AllBlocks}}.
> For large datasets it is not viable to load all blocks into memory, but to
> speed up access it still makes sense to prefetch the index blocks (being
> block index and Bloom filter blocks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14556) Make prefetchOnOpen configurable for index and/or data blocks
[
https://issues.apache.org/jira/browse/HBASE-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952629#comment-14952629
]
Andrew Purtell commented on HBASE-14556:
Above comment fell victim to spell mangling on the phone. Should read "from a
boolean to an enum..."
> Make prefetchOnOpen configurable for index and/or data blocks
> -
>
> Key: HBASE-14556
> URL: https://issues.apache.org/jira/browse/HBASE-14556
> Project: HBase
> Issue Type: Bug
> Components: BlockCache, regionserver
>Reporter: Lars George
>
> This came up in user discussions. It would be great to add an extra option to
> the {{CacheConfig}} that allows to specify what blocks are cached during
> region/file opening. This should allows to set {{BlockIndexOnly}},
> {{BloomFilterOnly}}, {{AllIndexesOnly}}, {{DataOnly}}, and {{AllBlocks}}.
> For large datasets it is not viable to load all blocks into memory, but to
> speed up access it still makes sense to prefetch the index blocks (being
> block index and Bloom filter blocks).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14398) Create the fake keys required in the scan path to avoid copy to byte[]
[ https://issues.apache.org/jira/browse/HBASE-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952631#comment-14952631 ] Anoop Sam John commented on HBASE-14398: bq.You two have made this argument with me a few times. We should integrate the why into the Interface doc? I can write it up if that'd help Pls.. that will be great. bq.Ok on the getXXXPosition. Can raise a sub task and get it committed. Thanks boss. > Create the fake keys required in the scan path to avoid copy to byte[] > -- > > Key: HBASE-14398 > URL: https://issues.apache.org/jira/browse/HBASE-14398 > Project: HBase > Issue Type: Sub-task >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: HBASE-14398.patch, HBASE-14398_1.patch > > > Already we have created some fake keys for the ByteBufferedCells so that we > can avoid the copy requried to create fake keys. This JIRA aims to fill up > all such places so that the Offheap BBs are not copied to onheap byte[]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14558) Document ChaosMonkey enhancements from HBASE-14261
[ https://issues.apache.org/jira/browse/HBASE-14558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14558: Attachment: HBASE-14558-v1.patch Addressed the comments so far. Ready for review again. > Document ChaosMonkey enhancements from HBASE-14261 > -- > > Key: HBASE-14558 > URL: https://issues.apache.org/jira/browse/HBASE-14558 > Project: HBase > Issue Type: Sub-task > Components: documentation >Affects Versions: 2.0.0 >Reporter: Misty Stanley-Jones >Assignee: Misty Stanley-Jones > Fix For: 2.0.0 > > Attachments: HBASE-14558-v1.patch, HBASE-14558.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-13976) release manager list in ref guide is missing 0.94 line
[ https://issues.apache.org/jira/browse/HBASE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-13976: --- Assignee: Misty Stanley-Jones > release manager list in ref guide is missing 0.94 line > -- > > Key: HBASE-13976 > URL: https://issues.apache.org/jira/browse/HBASE-13976 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Sean Busbey >Assignee: Misty Stanley-Jones > > 0.94 has slowed substantially, but we haven't EOLed it yet so we should make > sure folks looking to get a fix included know where to look. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Fix Version/s: 1.3.0
1.2.0
2.0.0
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-14557:
---
Affects Version/s: (was: 2.0.0)
1.2.0
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13976) release manager list in ref guide is missing 0.94 line
[ https://issues.apache.org/jira/browse/HBASE-13976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952607#comment-14952607 ] Misty Stanley-Jones commented on HBASE-13976: - I originally had this in there and was asked to take it out, I think. I don't mind putting it back. Who should be listed? > release manager list in ref guide is missing 0.94 line > -- > > Key: HBASE-13976 > URL: https://issues.apache.org/jira/browse/HBASE-13976 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Sean Busbey > > 0.94 has slowed substantially, but we haven't EOLed it yet so we should make > sure folks looking to get a fix included know where to look. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952617#comment-14952617
]
ramkrishna.s.vasudevan commented on HBASE-14557:
bq.We need this fix in all used places.. You can see now this ensureKeyValue is
used only with MR code( mapper, reducers etc) All these places KeyValue is the
value type as specified as in WALPlayer case
Yes, this is what I too suspect. In MR code we may have to add a new API in
KVUtil like ensureKeyValue to solve this which just always creates a new KV
from the incoming bytes?
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952627#comment-14952627
]
Anoop Sam John commented on HBASE-14557:
For master branch, the new API is not added as we have avoided the usage of
ensureKV() from all other parts other than MR area. In MR area, we need this
change.. SO that is fine.. For master at least we will avoid the new API
which is like deprecated and we wont encourage its usage outside.
For Master how we can get rid of KV hard coding from the MR area also?
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14026) Clarify "Web API" in version and compatibility docs
[ https://issues.apache.org/jira/browse/HBASE-14026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14026: Attachment: HBASE-14026.patch Ready for review. > Clarify "Web API" in version and compatibility docs > --- > > Key: HBASE-14026 > URL: https://issues.apache.org/jira/browse/HBASE-14026 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14026.patch > > > per discussion on HBASE-13861, update our version and compatibility section > to clarify under operational compatibility that by "Web page API" we mean the > /jmx endpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14026) Clarify "Web API" in version and compatibility docs
[ https://issues.apache.org/jira/browse/HBASE-14026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-14026: Status: Patch Available (was: Open) > Clarify "Web API" in version and compatibility docs > --- > > Key: HBASE-14026 > URL: https://issues.apache.org/jira/browse/HBASE-14026 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Sean Busbey >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14026.patch > > > per discussion on HBASE-13861, update our version and compatibility section > to clarify under operational compatibility that by "Web page API" we mean the > /jmx endpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13390) Document the implications of installing security coprocessors with hbase.security.authorization as false
[ https://issues.apache.org/jira/browse/HBASE-13390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952590#comment-14952590 ] Misty Stanley-Jones commented on HBASE-13390: - If you can enumerate what the implications are, then someone less of a guru (like me) might be able to tackle this. > Document the implications of installing security coprocessors with > hbase.security.authorization as false > > > Key: HBASE-13390 > URL: https://issues.apache.org/jira/browse/HBASE-13390 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0 > > > Add a new section to the security section of the online manual documenting > the implications of installing security coprocessors with > hbase.security.authorization set to false in site configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14026) Clarify "Web API" in version and compatibility docs
[ https://issues.apache.org/jira/browse/HBASE-14026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones reassigned HBASE-14026: --- Assignee: Misty Stanley-Jones (was: Sean Busbey) > Clarify "Web API" in version and compatibility docs > --- > > Key: HBASE-14026 > URL: https://issues.apache.org/jira/browse/HBASE-14026 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Sean Busbey >Assignee: Misty Stanley-Jones >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-14026.patch > > > per discussion on HBASE-13861, update our version and compatibility section > to clarify under operational compatibility that by "Web page API" we mean the > /jmx endpoint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14557) MapReduce WALPlayer issue with NoTagsKeyValue
[
https://issues.apache.org/jira/browse/HBASE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14952618#comment-14952618
]
ramkrishna.s.vasudevan commented on HBASE-14557:
I think similar patch added. Just seeing it after comment.
> MapReduce WALPlayer issue with NoTagsKeyValue
> -
>
> Key: HBASE-14557
> URL: https://issues.apache.org/jira/browse/HBASE-14557
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Anoop Sam John
>Priority: Blocker
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-14557.patch, HBASE-14557.patch,
> HBASE-14557_V2.patch, HBASE-14557_branch-1.2.patch, HBASE-14557_branch-1.patch
>
>
> Running MapReduce WALPlayer to convert WAL into HFiles:
> {noformat}
> 15/10/05 20:28:08 INFO mapred.JobClient: Task Id :
> attempt_201508031611_0029_m_00_0, Status : FAILED
> java.io.IOException: Type mismatch in value from map: expected
> org.apache.hadoop.hbase.KeyValue, recieved
> org.apache.hadoop.hbase.NoTagsKeyValue
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:997)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:689)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
> at
> org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:111)
> at
> org.apache.hadoop.hbase.mapreduce.WALPlayer$WALKeyValueMapper.map(WALPlayer.java:96)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:751)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:368)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at
> java.security.AccessController.doPrivileged(AccessController.java:369)
> at javax.security.auth.Subject.doAs(Subject.java:572)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1502)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
