[jira] [Commented] (HBASE-2256) Delete row, followed quickly to put of the same row will sometimes fail.
[
https://issues.apache.org/jira/browse/HBASE-2256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825164#comment-13825164
]
He Liangliang commented on HBASE-2256:
--
Also encountered similar problem. What about this solution?
{code}
public class IncrementingWallTimeEnvironmentEdge implements EnvironmentEdge {
private long clock = -1;
public IncrementingWallTimeEnvironmentEdge() {
}
@Override
public long currentTimeMillis() {
long wallTime = System.currentTimeMillis() 10; // ~us, or any arbitrary
scaling factor
synchronized (this) {
if (clock wallTime) {
clock = wallTime;
}
return clock++;
}
}
}
{code}
This would solve this problem and guarantee the timestamp aligned with the wall
time clock in milliseconds as long as we set the scaling factor to a larger
enough number (i.e. make sure the speed of logical clock is slower than
System.currentTimeMillis()). Shift factor from 10 ~20 (1M-1G qps) is proper
value for current server configuration which also will not introduce wrapping
around concern (584M ~ 0.57M year).
Delete row, followed quickly to put of the same row will sometimes fail.
Key: HBASE-2256
URL: https://issues.apache.org/jira/browse/HBASE-2256
Project: HBase
Issue Type: Bug
Affects Versions: 0.20.3
Reporter: Clint Morgan
Attachments: hbase-2256.patch
Doing a Delete of a whole row, followed immediately by a put to that row will
sometimes miss a cell. Attached is a test to provoke the issue.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9869) Optimize HConnectionManager#getCachedLocation
[ https://issues.apache.org/jira/browse/HBASE-9869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825178#comment-13825178 ] Nicolas Liochon commented on HBASE-9869: bq. we're short of memory so we clear the cache in order to do more work if we need the cached data again That's exactly that was happening, and that's why we went from 114s to 72s with the patch. I guess that protobuf added a lot of pressure on the GC, hence this behavior (but I've done the test with the latest trunk, it includes a lot of cleanup already). I think that the right pattern is to have the structure as small as possible, to make it fit into the processor cache. With all these references all over the place it's difficult to know exactly how it's going to behave. bq. I thought HRI only had a table name, not a TableName? We did a load of work to make it so HRI didn't have a HTD; would be a shame to go back there There is an attribute TableName in HRI, and createRegionName copy the table name in another attribute regionName: so we store it twice... I will see if I can remove it. Optimize HConnectionManager#getCachedLocation - Key: HBASE-9869 URL: https://issues.apache.org/jira/browse/HBASE-9869 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 6869.v4.patch, 9869.v1.patch, 9869.v1.patch, 9869.v2.patch It javadoc says: TODO: This method during writing consumes 15% of CPU doing lookup. This is still true, says Yourkit. With 0.96, we also spend more time in these methods. We retry more, and the AsyncProcess calls it in parallel. I don't have the patch for this yet, but I will spend some time on it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825179#comment-13825179 ] Nicolas Liochon commented on HBASE-9983: Note that I've closed the issue as I committed the patch, but more work is needed... Lower the memory footprint of HRegionLocation - Key: HBASE-9983 URL: https://issues.apache.org/jira/browse/HBASE-9983 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: 9983.v2.patch, unbloated.patch Its more or less a desperate cause... If we have a lot of regions, the size of this class becomes a problem. It's under optimized for this case... This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[ https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825182#comment-13825182 ] Nicolas Liochon commented on HBASE-9988: bq. Is the test failure related? Yes, I need to fix it :-) DOn't use HRI#getEncodedName in the client -- Key: HBASE-9988 URL: https://issues.apache.org/jira/browse/HBASE-9988 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9988.v1.patch This functions does a lazy initialisation. It cost memory and it creates a synchronisation point. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7663) [Per-KV security] Visibility labels
[
https://issues.apache.org/jira/browse/HBASE-7663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825183#comment-13825183
]
Hadoop QA commented on HBASE-7663:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614342/HBASE-7663_V9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7910//console
This message is automatically generated.
[Per-KV security] Visibility labels
---
Key: HBASE-7663
URL: https://issues.apache.org/jira/browse/HBASE-7663
Project: HBase
Issue Type: Sub-task
Components: Coprocessors, security
Affects Versions: 0.98.0
Reporter: Andrew Purtell
Assignee: Anoop Sam John
Fix For: 0.98.0
Attachments: HBASE-7663.patch, HBASE-7663_V2.patch,
HBASE-7663_V3.patch, HBASE-7663_V4.patch, HBASE-7663_V5.patch,
HBASE-7663_V6.patch, HBASE-7663_V7.patch, HBASE-7663_V8.patch,
HBASE-7663_V9.patch
Implement Accumulo-style visibility labels. Consider the following design
principles:
- Coprocessor based implementation
- Minimal to no changes to core code
- Use KeyValue tags (HBASE-7448) to carry labels
- Use OperationWithAttributes# {get,set}Attribute for handling visibility
labels in the API
- Implement a new filter for evaluating visibility labels as KVs are streamed
through.
This approach would be consistent in deployment and API details with other
per-KV security work, supporting environments where they might be both be
employed, even stacked on some tables.
See the parent issue for more discussion.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9985) Add HBase shell command to start/stop tracing
[ https://issues.apache.org/jira/browse/HBASE-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825199#comment-13825199 ] Hudson commented on HBASE-9985: --- SUCCESS: Integrated in hbase-0.96 #194 (See [https://builds.apache.org/job/hbase-0.96/194/]) HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542895) * /hbase/branches/0.96/hbase-shell/src/main/ruby/shell.rb * /hbase/branches/0.96/hbase-shell/src/main/ruby/shell/commands/trace.rb Add HBase shell command to start/stop tracing - Key: HBASE-9985 URL: https://issues.apache.org/jira/browse/HBASE-9985 Project: HBase Issue Type: New Feature Components: shell Reporter: Masatake Iwasaki Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9985-0.patch, HBASE-9985-0.patch Adding utility command to HBase shell for starting/stopping tracing span of HTrace. This command also initializes SpanReceiverHost on client side. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[ https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9988: --- Attachment: 9988.v2.patch DOn't use HRI#getEncodedName in the client -- Key: HBASE-9988 URL: https://issues.apache.org/jira/browse/HBASE-9988 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9988.v1.patch, 9988.v2.patch This functions does a lazy initialisation. It cost memory and it creates a synchronisation point. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[ https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9988: --- Status: Open (was: Patch Available) DOn't use HRI#getEncodedName in the client -- Key: HBASE-9988 URL: https://issues.apache.org/jira/browse/HBASE-9988 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9988.v1.patch This functions does a lazy initialisation. It cost memory and it creates a synchronisation point. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[ https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9988: --- Status: Patch Available (was: Open) DOn't use HRI#getEncodedName in the client -- Key: HBASE-9988 URL: https://issues.apache.org/jira/browse/HBASE-9988 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9988.v1.patch, 9988.v2.patch This functions does a lazy initialisation. It cost memory and it creates a synchronisation point. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9982) TestClientNoCluster should use random numbers
[ https://issues.apache.org/jira/browse/HBASE-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825220#comment-13825220 ] Nicolas Liochon commented on HBASE-9982: [~stack], are you ok with the change? I think the new defaults are better as well. TestClientNoCluster should use random numbers - Key: HBASE-9982 URL: https://issues.apache.org/jira/browse/HBASE-9982 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.1 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9982.v1.patch, 9982.v2.patch Using random number increases the number of calls to the meta scanner. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9974) Rest sometimes returns incomplete xml/json data
[ https://issues.apache.org/jira/browse/HBASE-9974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825229#comment-13825229 ] Liu Shaohui commented on HBASE-9974: Thanks [~apurtell]. The problem is that there is an nginx between rest server and http client. But the permission of proxy_temp of nginx is misconfigured, which causes the writes to proxy_temp fails when the response buffer is full. So the http client gets incomplete xml/json data. After the correction of nginx's config, the problem disappeared. Can someone help to close this issue? Rest sometimes returns incomplete xml/json data --- Key: HBASE-9974 URL: https://issues.apache.org/jira/browse/HBASE-9974 Project: HBase Issue Type: Bug Components: REST Reporter: Liu Shaohui Rest sometimes return incomplete xml/json data. We found these exceptions in rest server. 13/11/15 11:40:51 ERROR mortbay.log:/log/1A:23:11:0C:06:22* javax.ws.rs.WebApplicationException: javax.xml.bind.MarshalException - with linked exception: [org.mortbay.jetty.EofException] at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:159) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:73) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:322) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: javax.xml.bind.MarshalException - with linked exception: [org.mortbay.jetty.EofException] at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:325) at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249) at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:75) at com.sun.jersey.json.impl.JSONMarshallerImpl.marshal(JSONMarshallerImpl.java:74) at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179) at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157) ... 24 more Caused by: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:307) at
[jira] [Resolved] (HBASE-9974) Rest sometimes returns incomplete xml/json data
[ https://issues.apache.org/jira/browse/HBASE-9974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui resolved HBASE-9974. Resolution: Not A Problem Assignee: Liu Shaohui Rest sometimes returns incomplete xml/json data --- Key: HBASE-9974 URL: https://issues.apache.org/jira/browse/HBASE-9974 Project: HBase Issue Type: Bug Components: REST Reporter: Liu Shaohui Assignee: Liu Shaohui Rest sometimes return incomplete xml/json data. We found these exceptions in rest server. 13/11/15 11:40:51 ERROR mortbay.log:/log/1A:23:11:0C:06:22* javax.ws.rs.WebApplicationException: javax.xml.bind.MarshalException - with linked exception: [org.mortbay.jetty.EofException] at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:159) at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1437) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349) at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339) at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537) at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699) at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.apache.hadoop.hbase.rest.filter.GzipFilter.doFilter(GzipFilter.java:73) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:322) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: javax.xml.bind.MarshalException - with linked exception: [org.mortbay.jetty.EofException] at com.sun.xml.bind.v2.runtime.MarshallerImpl.write(MarshallerImpl.java:325) at com.sun.xml.bind.v2.runtime.MarshallerImpl.marshal(MarshallerImpl.java:249) at javax.xml.bind.helpers.AbstractMarshallerImpl.marshal(AbstractMarshallerImpl.java:75) at com.sun.jersey.json.impl.JSONMarshallerImpl.marshal(JSONMarshallerImpl.java:74) at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:179) at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:157) ... 24 more Caused by: org.mortbay.jetty.EofException at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:791) at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:551) at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:572) at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1012) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:651) at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:580) at com.sun.jersey.spi.container.servlet.WebComponent$Writer.write(WebComponent.java:307) at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.write(ContainerResponse.java:134) at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.flushBuffer(UTF8XmlOutput.java:416) at com.sun.xml.bind.v2.runtime.output.UTF8XmlOutput.text(UTF8XmlOutput.java:369) at com.sun.xml.bind.v2.runtime.unmarshaller.Base64Data.writeTo(Base64Data.java:303) at
[jira] [Updated] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9987: --- Attachment: 9987.v2.patch Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9987: --- Status: Open (was: Patch Available) Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825238#comment-13825238 ] Nicolas Liochon commented on HBASE-9987: In v2 I've put back the cachedServer. We could do better than this, but short term it's safer to keep it. That's the only difference, so I will commit if the unit tests pass. Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Attachment: 9959.v5.patch Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Open (was: Patch Available) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Patch Available (was: Open) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825246#comment-13825246 ] Nicolas Liochon commented on HBASE-9959: TestNamespaceCommands works locally. If the new attempt works, I will commit. Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9975) Not starting ReplicationSink when using custom implementation for the ReplicationSink.
[
https://issues.apache.org/jira/browse/HBASE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825261#comment-13825261
]
Anoop Sam John commented on HBASE-9975:
---
Seems we have the same issue in the stop code part also. Can I commit as an
addendum for this issue?
Not starting ReplicationSink when using custom implementation for the
ReplicationSink.
--
Key: HBASE-9975
URL: https://issues.apache.org/jira/browse/HBASE-9975
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.13
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.96.1, 0.94.14
Attachments: HBASE-9975_94.patch, HBASE-9975_Trunk.patch
Not starting ReplicationSink when using custom implementation for the
ReplicationSink.
{code}
if (this.replicationSourceHandler == this.replicationSinkHandler
this.replicationSourceHandler != null) {
this.replicationSourceHandler.startReplicationService();
} else if (this.replicationSourceHandler != null) {
this.replicationSourceHandler.startReplicationService();
} else if (this.replicationSinkHandler != null) {
this.replicationSinkHandler.startReplicationService();
}
{code}
ReplicationSource and Sink are different as there is custom impl for
ReplicationSink. We can not have else ifs
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[
https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825266#comment-13825266
]
Hadoop QA commented on HBASE-9988:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614360/9988.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7911//console
This message is automatically generated.
DOn't use HRI#getEncodedName in the client
--
Key: HBASE-9988
URL: https://issues.apache.org/jira/browse/HBASE-9988
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9988.v1.patch, 9988.v2.patch
This functions does a lazy initialisation. It cost memory and it creates a
synchronisation point.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[
https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825284#comment-13825284
]
Hadoop QA commented on HBASE-9959:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614366/9959.v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7912//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7912//console
This message is automatically generated.
Remove some array copy - server side
Key: HBASE-9959
URL: https://issues.apache.org/jira/browse/HBASE-9959
Project: HBase
Issue Type: Bug
Components: Protobufs, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch,
9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch,
9959.v4.patch, 9959.v5.patch, 9959.v5.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Open (was: Patch Available) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Patch Available (was: Open) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Attachment: 9959.v5.patch Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9987: --- Status: Patch Available (was: Open) Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Open (was: Patch Available) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Patch Available (was: Open) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Attachment: 9976.v4.patch Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825325#comment-13825325 ] Hudson commented on HBASE-8163: --- SUCCESS: Integrated in HBase-TRUNK #4685 (See [https://builds.apache.org/job/HBase-TRUNK/4685/]) Add note on https://issues.apache.org/jira/browse/HBASE-8163 -- config for write-heavy situation (stack: rev 1542903) * /hbase/trunk/src/main/docbkx/performance.xml MemStoreChunkPool: An improvement for JAVA GC when using MSLAB -- Key: HBASE-8163 URL: https://issues.apache.org/jira/browse/HBASE-8163 Project: HBase Issue Type: New Feature Components: Performance, regionserver Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.95.0 Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hbase-8163v6.patch *Usage:* Disable pool(default): configure hbase.hregion.memstore.chunkpool.maxsize as 0 Enable pool: configure hbase.hregion.memstore.chunkpool.maxsize as a percentage of global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and max sizes of memstore, e.g. 0.5) *Background*： When we use mslab,we will copy the keyvalue together in a structure called *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. *Problem*： When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed by JVM if no reference to it. Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost of YGC When a Chunk object has no reference? It should meet two following condition: 1.Memstore which this chunk belongs to is flushed 2.No scanner is opening on the memstore which this chunk belongs to *Solution:* 1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by JVM 2.When a Chunk has no reference, put it back to the pool 3.The pool has a max capacity, it will skip the chunks when achieve the max size 4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create new one by pool, so we could be able to reuse the old chunks *Test results:* Environment: hbase-version:0.94 -Xms4G -Xmx4G -Xmn2G Row size=50 bytes, Value size=1024 bytes 50 concurrent theads per client, insert 10,000,000 rows Before: Avg write request per second:12953 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 747 36.503 48 2.492 38.995 After: Avg write request per second:14025 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 711 20.344 4 0.284 20.628 *Improvement: YGC 40+%; WPS 5%+* review board : https://reviews.apache.org/r/10056/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9985) Add HBase shell command to start/stop tracing
[ https://issues.apache.org/jira/browse/HBASE-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825324#comment-13825324 ] Hudson commented on HBASE-9985: --- SUCCESS: Integrated in HBase-TRUNK #4685 (See [https://builds.apache.org/job/HBase-TRUNK/4685/]) HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542894) * /hbase/trunk/hbase-shell/src/main/ruby/shell/commands/trace.rb HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542893) * /hbase/trunk/hbase-shell/src/main/ruby/shell.rb Add HBase shell command to start/stop tracing - Key: HBASE-9985 URL: https://issues.apache.org/jira/browse/HBASE-9985 Project: HBase Issue Type: New Feature Components: shell Reporter: Masatake Iwasaki Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9985-0.patch, HBASE-9985-0.patch Adding utility command to HBase shell for starting/stopping tracing span of HTrace. This command also initializes SpanReceiverHost on client side. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825330#comment-13825330 ] Nicolas Liochon commented on HBASE-9976: v4 seems ok locally. I've tested it with all the other jiras in progress: yourkit does not show any blocked thread anymore, with two or twenty clients. The difference seems to come from all the valueOf that check the cache content before doing any ByteBuffer to byte array creation. Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[
https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825349#comment-13825349
]
Hadoop QA commented on HBASE-9959:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614380/9959.v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7914//console
This message is automatically generated.
Remove some array copy - server side
Key: HBASE-9959
URL: https://issues.apache.org/jira/browse/HBASE-9959
Project: HBase
Issue Type: Bug
Components: Protobufs, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch,
9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch,
9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[
https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825354#comment-13825354
]
Hadoop QA commented on HBASE-9987:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614364/9987.v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7913//console
This message is automatically generated.
Remove some synchronisation points in HConnectionManager
Key: HBASE-9987
URL: https://issues.apache.org/jira/browse/HBASE-9987
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9987.v1.patch, 9987.v2.patch
Change a Map to a concurrentMap
Removed the cachedServer (introduced in HBASE-4785). I suspect that this
function is not needed anymore as we also have a list of dead servers, and
accessing the list is not blocking. I will dig into this more however.
The patch gives a 10% improvement with the NoClusterClient.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[
https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825368#comment-13825368
]
Hadoop QA commented on HBASE-9976:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614389/9976.v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 hadoop1.0{color}. The patch failed to compile against the
hadoop 1.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7915//console
This message is automatically generated.
Don't create duplicated TableName objects
-
Key: HBASE-9976
URL: https://issues.apache.org/jira/browse/HBASE-9976
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9976.v1.patch, 9976.v4.patch
A profiling show that the table name is reponsible for 25% of the memory
needed to keep the region locations. As well, comparisons will be faster if
two identical table names are a single java object.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Shi updated HBASE-9969: Attachment: hbase-9969-v3.patch upload v3 - add an assertion as Ted mentioned - prepend a 16 bytes prefix to row keys in the benchmark (will update the benchmark result tomorrow when I have access to my devbox) Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0, 0.96.1, 0.94.15 Attachments: 9969-0.94.txt, hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9982) TestClientNoCluster should use random numbers
[ https://issues.apache.org/jira/browse/HBASE-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825380#comment-13825380 ] stack commented on HBASE-9982: -- [~nkeywal] +1 TestClientNoCluster should use random numbers - Key: HBASE-9982 URL: https://issues.apache.org/jira/browse/HBASE-9982 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.1 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9982.v1.patch, 9982.v2.patch Using random number increases the number of calls to the meta scanner. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-5945) Reduce buffer copies in IPC server response path
[ https://issues.apache.org/jira/browse/HBASE-5945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825383#comment-13825383 ] stack commented on HBASE-5945: -- I will commit this later today unless objection. Reduce buffer copies in IPC server response path Key: HBASE-5945 URL: https://issues.apache.org/jira/browse/HBASE-5945 Project: HBase Issue Type: Improvement Components: IPC/RPC Affects Versions: 0.95.2 Reporter: Todd Lipcon Assignee: stack Fix For: 0.96.1 Attachments: 5945-in-progress.2.1.patch, 5945-in-progress.2.patch, 5945-in-progress.patch, 5945v2.txt, 5945v4.txt, 5945v4.txt, 5945v5.txt, buffer-copies.txt, even-fewer-copies.txt, hbase-5495.txt, with_patch.png, without_patch.png The new PB code is sloppy with buffers and makes several needless copies. This increases GC time a lot. A few simple changes can cut this back down. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9982) TestClientNoCluster should use random numbers
[ https://issues.apache.org/jira/browse/HBASE-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9982: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed, thanks, Nick Stack! TestClientNoCluster should use random numbers - Key: HBASE-9982 URL: https://issues.apache.org/jira/browse/HBASE-9982 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.1 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9982.v1.patch, 9982.v2.patch Using random number increases the number of calls to the meta scanner. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825387#comment-13825387 ] Nicolas Liochon commented on HBASE-9959: this starts to be boring :-). May be I'm going to split this in smaller pieces. Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9987: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks for the review, lads. Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825391#comment-13825391 ] Chao Shi commented on HBASE-9969: - bq. I wonder if you should make ~10 cols/row, and NUM_KEYS_PER_SCANNER/10 rows. This should exacerbate the problem i mentioned above. I tried this on my laptop but seems your case above is even faster than before. Maybe there is something wrong with my environment. I will try it on my devbox tomorrow. bq. This literally sits at the core of HBase, and we'd better be a 100% sure it's OK. I'm totally agree with you! I think we can delay the optimization to future and make it correct first. As it is a so critical piece and the original implementation is so tricky (the bloomfilter part), I'd like to invite all folks to review this patch, particularity who know KeyValueHeap better than me (e.g. [~liyin]). Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0, 0.96.1, 0.94.15 Attachments: 9969-0.94.txt, hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Patch Available (was: Open) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Attachment: 9976.v6.patch Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825394#comment-13825394 ] Nicolas Liochon commented on HBASE-9976: v6 same patch, rebased. v4 was working locally, so it's strange. Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Open (was: Patch Available) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Open (was: Patch Available) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[
https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825397#comment-13825397
]
Hadoop QA commented on HBASE-9976:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614399/9976.v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 hadoop1.0{color}. The patch failed to compile against the
hadoop 1.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7917//console
This message is automatically generated.
Don't create duplicated TableName objects
-
Key: HBASE-9976
URL: https://issues.apache.org/jira/browse/HBASE-9976
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch
A profiling show that the table name is reponsible for 25% of the memory
needed to keep the region locations. As well, comparisons will be faster if
two identical table names are a single java object.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Status: Patch Available (was: Open) Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9959: --- Attachment: 9959.v5.patch Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9988) DOn't use HRI#getEncodedName in the client
[ https://issues.apache.org/jira/browse/HBASE-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825399#comment-13825399 ] Nicolas Liochon commented on HBASE-9988: The patch changed, so I don't mind another review if you have time, Nick :-). I've removed something used only in trace mode, and fixed the previous issue. DOn't use HRI#getEncodedName in the client -- Key: HBASE-9988 URL: https://issues.apache.org/jira/browse/HBASE-9988 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9988.v1.patch, 9988.v2.patch This functions does a lazy initialisation. It cost memory and it creates a synchronisation point. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HBASE-9989) Add a test on get in TestClientNoCluster
Nicolas Liochon created HBASE-9989: -- Summary: Add a test on get in TestClientNoCluster Key: HBASE-9989 URL: https://issues.apache.org/jira/browse/HBASE-9989 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9989) Add a test on get in TestClientNoCluster
[ https://issues.apache.org/jira/browse/HBASE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825446#comment-13825446 ] Nicolas Liochon commented on HBASE-9989: I don't submit it to save an execution round on Apache builds, but it's ready for integration. Add a test on get in TestClientNoCluster Key: HBASE-9989 URL: https://issues.apache.org/jira/browse/HBASE-9989 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9989.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9989) Add a test on get in TestClientNoCluster
[ https://issues.apache.org/jira/browse/HBASE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9989: --- Attachment: 9989.v1.patch Add a test on get in TestClientNoCluster Key: HBASE-9989 URL: https://issues.apache.org/jira/browse/HBASE-9989 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9989.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[
https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825455#comment-13825455
]
Hadoop QA commented on HBASE-9969:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614398/hbase-9969-v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 1.3.9) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7916//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7916//console
This message is automatically generated.
Improve KeyValueHeap using loser tree
-
Key: HBASE-9969
URL: https://issues.apache.org/jira/browse/HBASE-9969
Project: HBase
Issue Type: Improvement
Components: Performance, regionserver
Reporter: Chao Shi
Assignee: Chao Shi
Fix For: 0.98.0, 0.96.1, 0.94.15
Attachments: 9969-0.94.txt, hbase-9969-v2.patch, hbase-9969-v3.patch,
hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt
LoserTree is the better data structure than binary heap. It saves half of the
comparisons on each next(), though the time complexity is on O(logN).
Currently A scan or get will go through two KeyValueHeaps, one is merging KVs
read from multiple HFiles in a single store, the other is merging results
from multiple stores. This patch should improve the both cases whenever CPU
is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811).
All of the optimization work is done in KeyValueHeap and does not change its
public interfaces. The new code looks more cleaner and simpler to understand.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9989) Add a test on get in TestClientNoCluster
[ https://issues.apache.org/jira/browse/HBASE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825470#comment-13825470 ] Nicolas Liochon commented on HBASE-9989: fwiw, here are the builders we create for each get request (not taking into account that we need to parse the result). ClientProtos.GetRequest.Builder requestBuilder = ClientProtos.GetRequest.newBuilder(); HBaseProtos.RegionSpecifier.Builder regionBuilder = HBaseProtos.RegionSpecifier.newBuilder(); ClientProtos.Get.Builder getBuilder = ClientProtos.Get.newBuilder(); FilterProtos.Filter.Builder filterBuilder = FilterProtos.Filter.newBuilder(); HBaseProtos.TimeRange.Builder timeRangeBuilder = HBaseProtos.TimeRange.newBuilder(); HBaseProtos.NameBytesPair.Builder attributeBuilder = HBaseProtos.NameBytesPair.newBuilder(); ClientProtos.Column.Builder columnBuilder = ClientProtos.Column.newBuilder(); Add a test on get in TestClientNoCluster Key: HBASE-9989 URL: https://issues.apache.org/jira/browse/HBASE-9989 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9989.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825471#comment-13825471 ] Jonathan Hsieh commented on HBASE-9969: --- bq. I have not found a scenario, *yet*, where this is slower. Seems generally safe to pull into all branches. Emphasis mine. I'm a bit concerned with such a fundamental change going in without a way of going back in the case we find a slower case, especially in the stable branch. Wouldn't it be prudent if this were to make it into 0.94 make it an option? Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0, 0.96.1, 0.94.15 Attachments: 9969-0.94.txt, hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[
https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825480#comment-13825480
]
Hadoop QA commented on HBASE-9959:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614402/9959.v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7918//console
This message is automatically generated.
Remove some array copy - server side
Key: HBASE-9959
URL: https://issues.apache.org/jira/browse/HBASE-9959
Project: HBase
Issue Type: Bug
Components: Protobufs, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch,
9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch,
9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
Nicolas Liochon created HBASE-9990: -- Summary: RpcRetryingCallerFactory uses the conf for each newCaller Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 You can construct a RpcRetryingCallerFactory, but actually the conf is read for each call. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
[
https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825517#comment-13825517
]
Nicolas Liochon commented on HBASE-9990:
The issue lies in the subclass, not in RpcRetryingCallerFactory itself actually.
If I understand well the code, the factory is supposed to return a class
extending this object.
@InterfaceAudience.Private
public class RpcRetryingCallerT {
As the interface is private, I could pretend it should be used and change it.
I actually don't have a better solution...
RpcRetryingCallerFactory uses the conf for each newCaller
---
Key: HBASE-9990
URL: https://issues.apache.org/jira/browse/HBASE-9990
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
You can construct a RpcRetryingCallerFactory, but actually the conf is read
for each call. Reading the conf is obviously expensive, and a profiling
session shows it. If we want to sent hundreds of thousands of queries per
second, we should not do that.
RpcRetryingCallerFactory.newCaller is called for each get, for example.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
[
https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825517#comment-13825517
]
Nicolas Liochon edited comment on HBASE-9990 at 11/18/13 5:43 PM:
--
The issue lies in the subclass, not in RpcRetryingCallerFactory itself actually.
If I understand well the code, the factory is supposed to return a class
extending this:
@InterfaceAudience.Private
public class RpcRetryingCallerT {
As the interface is private, I could pretend it should not be used and change
it.
I actually don't have a better solution...
was (Author: nkeywal):
The issue lies in the subclass, not in RpcRetryingCallerFactory itself actually.
If I understand well the code, the factory is supposed to return a class
extending this object.
@InterfaceAudience.Private
public class RpcRetryingCallerT {
As the interface is private, I could pretend it should not be used and change
it.
I actually don't have a better solution...
RpcRetryingCallerFactory uses the conf for each newCaller
---
Key: HBASE-9990
URL: https://issues.apache.org/jira/browse/HBASE-9990
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
You can construct a RpcRetryingCallerFactory, but actually the conf is read
for each call. Reading the conf is obviously expensive, and a profiling
session shows it. If we want to sent hundreds of thousands of queries per
second, we should not do that.
RpcRetryingCallerFactory.newCaller is called for each get, for example.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Comment Edited] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
[
https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825517#comment-13825517
]
Nicolas Liochon edited comment on HBASE-9990 at 11/18/13 5:43 PM:
--
The issue lies in the subclass, not in RpcRetryingCallerFactory itself actually.
If I understand well the code, the factory is supposed to return a class
extending this object.
@InterfaceAudience.Private
public class RpcRetryingCallerT {
As the interface is private, I could pretend it should not be used and change
it.
I actually don't have a better solution...
was (Author: nkeywal):
The issue lies in the subclass, not in RpcRetryingCallerFactory itself actually.
If I understand well the code, the factory is supposed to return a class
extending this object.
@InterfaceAudience.Private
public class RpcRetryingCallerT {
As the interface is private, I could pretend it should be used and change it.
I actually don't have a better solution...
RpcRetryingCallerFactory uses the conf for each newCaller
---
Key: HBASE-9990
URL: https://issues.apache.org/jira/browse/HBASE-9990
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
You can construct a RpcRetryingCallerFactory, but actually the conf is read
for each call. Reading the conf is obviously expensive, and a profiling
session shows it. If we want to sent hundreds of thousands of queries per
second, we should not do that.
RpcRetryingCallerFactory.newCaller is called for each get, for example.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9969) Improve KeyValueHeap using loser tree
[ https://issues.apache.org/jira/browse/HBASE-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825519#comment-13825519 ] Liyin Tang commented on HBASE-9969: --- That's a very promising idea ! Will take a closer look. Nice work [~stepinto] ! Improve KeyValueHeap using loser tree - Key: HBASE-9969 URL: https://issues.apache.org/jira/browse/HBASE-9969 Project: HBase Issue Type: Improvement Components: Performance, regionserver Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.98.0, 0.96.1, 0.94.15 Attachments: 9969-0.94.txt, hbase-9969-v2.patch, hbase-9969-v3.patch, hbase-9969.patch, hbase-9969.patch, kvheap-benchmark.png, kvheap-benchmark.txt LoserTree is the better data structure than binary heap. It saves half of the comparisons on each next(), though the time complexity is on O(logN). Currently A scan or get will go through two KeyValueHeaps, one is merging KVs read from multiple HFiles in a single store, the other is merging results from multiple stores. This patch should improve the both cases whenever CPU is the bottleneck (e.g. scan with filter over cached blocks, HBASE-9811). All of the optimization work is done in KeyValueHeap and does not change its public interfaces. The new code looks more cleaner and simpler to understand. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[
https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825529#comment-13825529
]
Ted Yu commented on HBASE-9976:
---
I got the following when compiling patch v6:
{code}
[ERROR]
/Users/tyu/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java:[443,20]
cannot find symbol
[ERROR] symbol : method compare(int,int)
[ERROR] location: class java.lang.Integer
{code}
Don't create duplicated TableName objects
-
Key: HBASE-9976
URL: https://issues.apache.org/jira/browse/HBASE-9976
Project: HBase
Issue Type: Bug
Components: Client, regionserver
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch
A profiling show that the table name is reponsible for 25% of the memory
needed to keep the region locations. As well, comparisons will be faster if
two identical table names are a single java object.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (HBASE-9991) Enhance test-patch.sh with post back of snippet of compilation error if any is detected
[
https://issues.apache.org/jira/browse/HBASE-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-9991:
--
Summary: Enhance test-patch.sh with post back of snippet of compilation
error if any is detected (was: Enhance test-patch.sh with snippet of
compilation error if any is detected)
Enhance test-patch.sh with post back of snippet of compilation error if any
is detected
---
Key: HBASE-9991
URL: https://issues.apache.org/jira/browse/HBASE-9991
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Currently test-patch.sh would post back something like this:
{code}
-1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
{code}
Snippet of the compilation error should be included so that people can see
what caused the error.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HBASE-9991) Enhance test-patch.sh with snippet of compilation error if any is detected
Ted Yu created HBASE-9991:
-
Summary: Enhance test-patch.sh with snippet of compilation error
if any is detected
Key: HBASE-9991
URL: https://issues.apache.org/jira/browse/HBASE-9991
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Currently test-patch.sh would post back something like this:
{code}
-1 hadoop1.0. The patch failed to compile against the hadoop 1.0 profile.
{code}
Snippet of the compilation error should be included so that people can see what
caused the error.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9982) TestClientNoCluster should use random numbers
[ https://issues.apache.org/jira/browse/HBASE-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825547#comment-13825547 ] Hudson commented on HBASE-9982: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #123 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/123/]) HBASE-9982 TestClientNoCluster should use random numbers (nkeywal: rev 1543052) * /hbase/branches/0.96/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java TestClientNoCluster should use random numbers - Key: HBASE-9982 URL: https://issues.apache.org/jira/browse/HBASE-9982 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.1 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9982.v1.patch, 9982.v2.patch Using random number increases the number of calls to the meta scanner. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825546#comment-13825546 ] Hudson commented on HBASE-9987: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #123 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/123/]) HBASE-9987 Remove some synchronisation points in HConnectionManager (nkeywal: rev 1543050) * /hbase/branches/0.96/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9985) Add HBase shell command to start/stop tracing
[ https://issues.apache.org/jira/browse/HBASE-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825545#comment-13825545 ] Hudson commented on HBASE-9985: --- SUCCESS: Integrated in hbase-0.96-hadoop2 #123 (See [https://builds.apache.org/job/hbase-0.96-hadoop2/123/]) HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542895) * /hbase/branches/0.96/hbase-shell/src/main/ruby/shell.rb * /hbase/branches/0.96/hbase-shell/src/main/ruby/shell/commands/trace.rb Add HBase shell command to start/stop tracing - Key: HBASE-9985 URL: https://issues.apache.org/jira/browse/HBASE-9985 Project: HBase Issue Type: New Feature Components: shell Reporter: Masatake Iwasaki Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9985-0.patch, HBASE-9985-0.patch Adding utility command to HBase shell for starting/stopping tracing span of HTrace. This command also initializes SpanReceiverHost on client side. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825552#comment-13825552 ] Nicolas Liochon commented on HBASE-9976: Thanks Ted, it's likely because I'm using the jdk 1.7. I'm going to fix this. Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-9973:
---
Attachment: 9973-v2.patch
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
Attachments: 9973-v2.patch, 9973.patch
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9975) Not starting ReplicationSink when using custom implementation for the ReplicationSink.
[
https://issues.apache.org/jira/browse/HBASE-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1382#comment-1382
]
Lars Hofhansl commented on HBASE-9975:
--
It won't be in 0.94.14, though. So a separate issue would be better.
+1 on doing the same for stop.
Not starting ReplicationSink when using custom implementation for the
ReplicationSink.
--
Key: HBASE-9975
URL: https://issues.apache.org/jira/browse/HBASE-9975
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.13
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.98.0, 0.96.1, 0.94.14
Attachments: HBASE-9975_94.patch, HBASE-9975_Trunk.patch
Not starting ReplicationSink when using custom implementation for the
ReplicationSink.
{code}
if (this.replicationSourceHandler == this.replicationSinkHandler
this.replicationSourceHandler != null) {
this.replicationSourceHandler.startReplicationService();
} else if (this.replicationSourceHandler != null) {
this.replicationSourceHandler.startReplicationService();
} else if (this.replicationSinkHandler != null) {
this.replicationSinkHandler.startReplicationService();
}
{code}
ReplicationSource and Sink are different as there is custom impl for
ReplicationSink. We can not have else ifs
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825558#comment-13825558 ] Ted Yu commented on HBASE-9976: --- Ah, I used jdk 1.6 :-) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825559#comment-13825559
]
Himanshu Vashishtha commented on HBASE-9973:
Yes, that's why I removed the compaction call, thanks. Yes, the region is
flushed when we close it.
Good call, I skipped the ACL_GLOBAL_NAME variable. Added it and moved the
result check up.
mvn test -Dtest=Test*Namespace* passes on local. Also, ran the upgrade against
a 94 installation.
On a 94 set up:
{code}
hbase(main):006:0 scan '_acl_'
ROW COLUMN+CELL
TestTable column=l:root, timestamp=1384659937736,
value=RW
_acl_ column=l:root, timestamp=1384659909517,
value=CRA
_acl_ column=l:root2, timestamp=1384659916521,
value=CRA
2 row(s) in 0.0520 seconds
{code}
After upgrade to a 96 setup:
{code}
hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:root,
timestamp=1384659937736, value=RW
hbase:aclcolumn=l:root,
timestamp=1384797939800, value=CRA
hbase:aclcolumn=l:root2,
timestamp=1384797939800, value=CRA
2 row(s) in 0.0190 seconds
{code}
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
Attachments: 9973-v2.patch, 9973.patch
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825576#comment-13825576 ] Vladimir Rodionov commented on HBASE-9983: -- Its always good to measure the improvement. Any numbers, Nicolas, you can share? Lower the memory footprint of HRegionLocation - Key: HBASE-9983 URL: https://issues.apache.org/jira/browse/HBASE-9983 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: 9983.v2.patch, unbloated.patch Its more or less a desperate cause... If we have a lot of regions, the size of this class becomes a problem. It's under optimized for this case... This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Summary: HTable uses the conf for each newCaller (was: RpcRetryingCallerFactory uses the conf for each newCaller) HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each call. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Status: Patch Available (was: Open) RpcRetryingCallerFactory uses the conf for each newCaller --- Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each call. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Description: You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. was: You can construct a RpcRetryingCallerFactory, but actually the conf is read for each call. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825582#comment-13825582 ] Ted Yu commented on HBASE-9990: --- +1 HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825583#comment-13825583 ] Nicolas Liochon commented on HBASE-9990: v1: The 'clone' is far from perfect, but let's see how hadoop qa reacts. HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9990) HTable uses the conf for each newCaller
[
https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825585#comment-13825585
]
Hadoop QA commented on HBASE-9990:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12614442/9990.v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 hadoop1.0{color}. The patch failed to compile against the
hadoop 1.0 profile.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/7920//console
This message is automatically generated.
HTable uses the conf for each newCaller
-
Key: HBASE-9990
URL: https://issues.apache.org/jira/browse/HBASE-9990
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 0.98.0, 0.96.0
Reporter: Nicolas Liochon
Assignee: Nicolas Liochon
Fix For: 0.98.0, 0.96.1
Attachments: 9990.v1.patch
You can construct a RpcRetryingCallerFactory, but actually the conf is read
for each caller creation. Reading the conf is obviously expensive, and a
profiling session shows it. If we want to sent hundreds of thousands of
queries per second, we should not do that.
RpcRetryingCallerFactory.newCaller is called for each get, for example.
This is not a regression, we have something similar in 0.94.
On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after
a few thousand calls to get.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9983) Lower the memory footprint of HRegionLocation
[ https://issues.apache.org/jira/browse/HBASE-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825598#comment-13825598 ] Nicolas Liochon commented on HBASE-9983: I haven't done a measure before/after for this one (would have been interested, I agree). Right now, so with this jira included, I'm seeing (with Yourkit) that an entry the location skip list takes ~550 bytes. This will be lowered with HBASE-9976 and HBASE-9988. Then we have one ServerName instance by entry as well... Lower the memory footprint of HRegionLocation - Key: HBASE-9983 URL: https://issues.apache.org/jira/browse/HBASE-9983 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: 9983.v2.patch, unbloated.patch Its more or less a desperate cause... If we have a lot of regions, the size of this class becomes a problem. It's under optimized for this case... This patch is just about the trivial part -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-9973:
---
Attachment: 9973-v2.patch
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
Attachments: 9973-v2.patch, 9973-v2.patch, 9973.patch
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9985) Add HBase shell command to start/stop tracing
[ https://issues.apache.org/jira/browse/HBASE-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825599#comment-13825599 ] Hudson commented on HBASE-9985: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #842 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/842/]) HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542894) * /hbase/trunk/hbase-shell/src/main/ruby/shell/commands/trace.rb HBASE-9985 Add HBase shell command to start/stop tracing (stack: rev 1542893) * /hbase/trunk/hbase-shell/src/main/ruby/shell.rb Add HBase shell command to start/stop tracing - Key: HBASE-9985 URL: https://issues.apache.org/jira/browse/HBASE-9985 Project: HBase Issue Type: New Feature Components: shell Reporter: Masatake Iwasaki Priority: Minor Fix For: 0.98.0, 0.96.1 Attachments: HBASE-9985-0.patch, HBASE-9985-0.patch Adding utility command to HBase shell for starting/stopping tracing span of HTrace. This command also initializes SpanReceiverHost on client side. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-8163) MemStoreChunkPool: An improvement for JAVA GC when using MSLAB
[ https://issues.apache.org/jira/browse/HBASE-8163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825601#comment-13825601 ] Hudson commented on HBASE-8163: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #842 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/842/]) Add note on https://issues.apache.org/jira/browse/HBASE-8163 -- config for write-heavy situation (stack: rev 1542903) * /hbase/trunk/src/main/docbkx/performance.xml MemStoreChunkPool: An improvement for JAVA GC when using MSLAB -- Key: HBASE-8163 URL: https://issues.apache.org/jira/browse/HBASE-8163 Project: HBase Issue Type: New Feature Components: Performance, regionserver Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.98.0, 0.95.0 Attachments: hbase-0.95-8163v6.patch, hbase-8163v1.patch, hbase-8163v2.patch, hbase-8163v3.patch, hbase-8163v4.patch, hbase-8163v5.patch, hbase-8163v6.patch *Usage:* Disable pool(default): configure hbase.hregion.memstore.chunkpool.maxsize as 0 Enable pool: configure hbase.hregion.memstore.chunkpool.maxsize as a percentage of global memstore size (between 0.0 and 1.0, recommend to set it as the gap between min and max sizes of memstore, e.g. 0.5) *Background*： When we use mslab,we will copy the keyvalue together in a structure called *MemStoreLAB$Chunk*, therefore we could decrease the heap fragment. *Problem*： When one chunk is full, we will create a new chunk, and then the old chunk will be reclaimed by JVM if no reference to it. Mostly the chunk object would be promoted when doing Young GC, cause increasing the cost of YGC When a Chunk object has no reference? It should meet two following condition: 1.Memstore which this chunk belongs to is flushed 2.No scanner is opening on the memstore which this chunk belongs to *Solution:* 1.Create a chunk pool to manage the no-reference chunks, instead of being reclaimed by JVM 2.When a Chunk has no reference, put it back to the pool 3.The pool has a max capacity, it will skip the chunks when achieve the max size 4.When we need new Chunk to store KeyValue, get it from the pool if exists, else create new one by pool, so we could be able to reuse the old chunks *Test results:* Environment: hbase-version:0.94 -Xms4G -Xmx4G -Xmn2G Row size=50 bytes, Value size=1024 bytes 50 concurrent theads per client, insert 10,000,000 rows Before: Avg write request per second:12953 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 747 36.503 48 2.492 38.995 After: Avg write request per second:14025 After testing, final result of jstat -gcutil : YGC YGCT FGC FGCT GCT 711 20.344 4 0.284 20.628 *Improvement: YGC 40+%; WPS 5%+* review board : https://reviews.apache.org/r/10056/ -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9987) Remove some synchronisation points in HConnectionManager
[ https://issues.apache.org/jira/browse/HBASE-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825600#comment-13825600 ] Hudson commented on HBASE-9987: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #842 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/842/]) HBASE-9987 Remove some synchronisation points in HConnectionManager (nkeywal: rev 1543051) * /hbase/trunk/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java Remove some synchronisation points in HConnectionManager Key: HBASE-9987 URL: https://issues.apache.org/jira/browse/HBASE-9987 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9987.v1.patch, 9987.v2.patch Change a Map to a concurrentMap Removed the cachedServer (introduced in HBASE-4785). I suspect that this function is not needed anymore as we also have a list of dead servers, and accessing the list is not blocking. I will dig into this more however. The patch gives a 10% improvement with the NoClusterClient. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9982) TestClientNoCluster should use random numbers
[ https://issues.apache.org/jira/browse/HBASE-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825602#comment-13825602 ] Hudson commented on HBASE-9982: --- SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #842 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/842/]) HBASE-9982 TestClientNoCluster should use random numbers (nkeywal: rev 1543053) * /hbase/trunk/hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestClientNoCluster.java TestClientNoCluster should use random numbers - Key: HBASE-9982 URL: https://issues.apache.org/jira/browse/HBASE-9982 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.1 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9982.v1.patch, 9982.v2.patch Using random number increases the number of calls to the meta scanner. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825604#comment-13825604
]
Enis Soztutar commented on HBASE-9973:
--
+1. Nice find.
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
Attachments: 9973-v2.patch, 9973-v2.patch, 9973.patch
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HBASE-9959) Remove some array copy - server side
[ https://issues.apache.org/jira/browse/HBASE-9959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825606#comment-13825606 ] Nicolas Liochon commented on HBASE-9959: The third attempt worked. Any issue if I commit this? Remove some array copy - server side Key: HBASE-9959 URL: https://issues.apache.org/jira/browse/HBASE-9959 Project: HBase Issue Type: Bug Components: Protobufs, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9959-trunk.v1.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959-trunk.v2.patch, 9959.v1.patch, 9959.v3.patch, 9959.v4.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch, 9959.v5.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) RpcRetryingCallerFactory uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Attachment: 9990.v1.patch RpcRetryingCallerFactory uses the conf for each newCaller --- Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each call. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Status: Open (was: Patch Available) HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Status: Patch Available (was: Open) HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Attachment: 9990.v1.patch HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9990) HTable uses the conf for each newCaller
[ https://issues.apache.org/jira/browse/HBASE-9990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9990: --- Attachment: (was: 9990.v1.patch) HTable uses the conf for each newCaller - Key: HBASE-9990 URL: https://issues.apache.org/jira/browse/HBASE-9990 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9990.v1.patch You can construct a RpcRetryingCallerFactory, but actually the conf is read for each caller creation. Reading the conf is obviously expensive, and a profiling session shows it. If we want to sent hundreds of thousands of queries per second, we should not do that. RpcRetryingCallerFactory.newCaller is called for each get, for example. This is not a regression, we have something similar in 0.94. On the 0.96, we see the creation of: java.util.regex.Matcher: 15739712b after a few thousand calls to get. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Open (was: Patch Available) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch, 9976.v7.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Attachment: 9976.v7.patch Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch, 9976.v7.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Liochon updated HBASE-9976: --- Status: Patch Available (was: Open) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.96.0, 0.98.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch, 9976.v7.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9976) Don't create duplicated TableName objects
[ https://issues.apache.org/jira/browse/HBASE-9976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825628#comment-13825628 ] Nicolas Liochon commented on HBASE-9976: v7, without the new fancy 1.7 features :-) Don't create duplicated TableName objects - Key: HBASE-9976 URL: https://issues.apache.org/jira/browse/HBASE-9976 Project: HBase Issue Type: Bug Components: Client, regionserver Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Fix For: 0.98.0, 0.96.1 Attachments: 9976.v1.patch, 9976.v4.patch, 9976.v6.patch, 9976.v7.patch A profiling show that the table name is reponsible for 25% of the memory needed to keep the region locations. As well, comparisons will be faster if two identical table names are a single java object. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-9989) Add a test on get in TestClientNoCluster
[ https://issues.apache.org/jira/browse/HBASE-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825633#comment-13825633 ] Nicolas Liochon commented on HBASE-9989: I suppose that the solution to this is to create the builders with the HTable objects, then to pass them all along... Add a test on get in TestClientNoCluster Key: HBASE-9989 URL: https://issues.apache.org/jira/browse/HBASE-9989 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.98.0, 0.96.0 Reporter: Nicolas Liochon Assignee: Nicolas Liochon Priority: Trivial Fix For: 0.98.0, 0.96.1 Attachments: 9989.v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (HBASE-9973) [ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade to 0.96.x from 0.94.x or 0.92.x
[
https://issues.apache.org/jira/browse/HBASE-9973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Himanshu Vashishtha updated HBASE-9973:
---
Component/s: migration
[ACL]: Users with 'Admin' ACL permission will lose permissions after upgrade
to 0.96.x from 0.94.x or 0.92.x
Key: HBASE-9973
URL: https://issues.apache.org/jira/browse/HBASE-9973
Project: HBase
Issue Type: Bug
Components: migration, security
Affects Versions: 0.96.0, 0.96.1
Reporter: Aleksandr Shulman
Assignee: Himanshu Vashishtha
Labels: acl
Fix For: 0.96.1
Attachments: 9973-v2.patch, 9973-v2.patch, 9973.patch
In our testing, we have uncovered that the ACL permissions for users with the
'A' credential do not hold after the upgrade to 0.96.x.
This is because in the ACL table, the entry for the admin user is a
permission on the '_acl_' table with permission 'A'. However, because of the
namespace transition, there is no longer an '_acl_' table. Therefore, that
entry in the hbase:acl table is no longer valid.
Example:
{code}hbase(main):002:0 scan 'hbase:acl'
ROW COLUMN+CELL
TestTablecolumn=l:hdfs, timestamp=1384454830701, value=RW
TestTablecolumn=l:root, timestamp=1384455875586, value=RWCA
_acl_column=l:root, timestamp=1384454767568, value=C
_acl_column=l:tableAdmin, timestamp=1384454788035, value=A
hbase:aclcolumn=l:root, timestamp=1384455875786, value=C
{code}
In this case, the following entry becomes meaningless:
{code} _acl_column=l:tableAdmin, timestamp=1384454788035,
value=A {code}
As a result,
Proposed fix:
I see the fix being relatively straightforward. As part of the migration,
change any entries in the '_acl_' table with key '_acl_' into a new row with
key 'hbase:acl', all else being the same. And the old entry would be deleted.
This can go into the standard migration script that we expect users to run.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
