[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001005#comment-14001005 ] Lars Hofhansl commented on HBASE-11096: --- Looked at the 0.94 patch. +1 for 0.94. stop method of Master and RegionServer coprocessor is not invoked -- Key: HBASE-11096 URL: https://issues.apache.org/jira/browse/HBASE-11096 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.94.19 Reporter: Qiang Tian Assignee: Qiang Tian Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.patch the stop method of coprocessor specified by hbase.coprocessor.master.classes and hbase.coprocessor.regionserver.classes is not invoked. If coprocessor allocates OS resources, it could cause master/regionserver resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[
https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001100#comment-14001100
]
Hadoop QA commented on HBASE-11104:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645444/HBASE-11104_trunk.patch
against trunk revision .
ATTACHMENT ID: 12645444
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9537//console
This message is automatically generated.
IntegrationTestImportTsv#testRunFromOutputCommitter misses credential
initialization
Key: HBASE-11104
URL: https://issues.apache.org/jira/browse/HBASE-11104
Project: HBase
Issue Type: Test
Reporter: Ted Yu
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: 11104-v1.txt, HBASE-11104_98_v3.patch,
HBASE-11104_trunk.patch, HBASE-11104_trunk.patch
IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships
the HBase dependencies.
However, call to TableMapReduceUtil.initCredentials(job) is missing - making
this test fail on a secure cluster.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001097#comment-14001097 ] Andrew Purtell commented on HBASE-11165: I think we may want to be autotuning Memstores based on usage metrics. Lazy allocation of mslabs and such if recent history is read only or write rarely. Just pointing out that autotuning is a related independent interest that could benefit here. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11169) nit: fix incorrect javadoc in OrderedBytes about BlobVar and BlobCopy
[ https://issues.apache.org/jira/browse/HBASE-11169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001106#comment-14001106 ] Hudson commented on HBASE-11169: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-11169 nit: fix incorrect javadoc in OrderedBytes about BlobVar and BlobCopy (jmhsieh: rev 1595390) * /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/util/OrderedBytes.java nit: fix incorrect javadoc in OrderedBytes about BlobVar and BlobCopy - Key: HBASE-11169 URL: https://issues.apache.org/jira/browse/HBASE-11169 Project: HBase Issue Type: Bug Components: util Affects Versions: 0.95.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Priority: Trivial Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: HBASE-11169.patch Trivial error in javadoc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001105#comment-14001105 ] Hudson commented on HBASE-6990: --- SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-6990 ADDENDUM (jmhsieh: rev 1595408) * /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/util/PrettyPrinter.java HBASE-6990 pretty print TTL (Esteban Gutierrez) (jmhsieh: rev 1595397) * /hbase/branches/0.98/hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/branches/0.98/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java Pretty print TTL Key: HBASE-6990 URL: https://issues.apache.org/jira/browse/HBASE-6990 Project: HBase Issue Type: Improvement Components: Usability Reporter: Jean-Daniel Cryans Assignee: Esteban Gutierrez Priority: Minor Attachments: HBASE-6990.v0.patch, HBASE-6990.v1.patch, HBASE-6990.v2.patch, HBASE-6990.v3.patch, HBASE-6990.v4.patch I've seen a lot of users getting confused by the TTL configuration and I think that if we just pretty printed it it would solve most of the issues. For example, let's say a user wanted to set a TTL of 90 days. That would be 7776000. But let's say that it was typo'd to 7776 instead, it gives you 900 days! So when we print the TTL we could do something like x days, x hours, x minutes, x seconds (real_ttl_value). This would also help people when they use ms instead of seconds as they would see really big values in there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11168) [docs] Remove references to RowLocks in post 0.96 docs.
[ https://issues.apache.org/jira/browse/HBASE-11168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001114#comment-14001114 ] Hudson commented on HBASE-11168: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-11168 [docs] Remove references to Rowlocks in post-0.96 docs. (jmhsieh: rev 1595393) * /hbase/branches/0.98/src/main/docbkx/book.xml * /hbase/branches/0.98/src/main/docbkx/troubleshooting.xml [docs] Remove references to RowLocks in post 0.96 docs. --- Key: HBASE-11168 URL: https://issues.apache.org/jira/browse/HBASE-11168 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.95.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.99.0, 0.98.3 Attachments: hbase-11168.patch Row locks were removed in 0.95 by HBASE-7315 / HBASE-2332. There are a few vestiges of them in the docs. Remove. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10212) New rpc metric: number of active handler
[ https://issues.apache.org/jira/browse/HBASE-10212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001107#comment-14001107 ] Hudson commented on HBASE-10212: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) Amend HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594133) * /hbase/branches/0.98/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java Amend HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594130) * /hbase/branches/0.98/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/branches/0.98/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcMetrics.java HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594118) * /hbase/branches/0.98/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapper.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java New rpc metric: number of active handler Key: HBASE-10212 URL: https://issues.apache.org/jira/browse/HBASE-10212 Project: HBase Issue Type: Improvement Components: IPC/RPC Reporter: Chao Shi Assignee: Chao Shi Fix For: 0.94.17 Attachments: hbase-10212.patch The attached patch adds a new metric: number of active handler threads. We found this is a good metric to measure how busy of a server. If this number is too high (compared to the total number of handlers), the server has risks in getting call queue full. We used to monitor # reads or # writes. However we found this often produce false alerts, because a read touching HDFS will produce much high workload than a block-cached read. The attached patch is based on our internal 0.94 branch, but I think it pretty easy to port to rebase to other branches if you think it is useful. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10003) OnlineMerge should be extended to allow bulk merging
[ https://issues.apache.org/jira/browse/HBASE-10003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001117#comment-14001117 ] takeshi.miao commented on HBASE-10003: -- Hi [~esteban] Tks for your follow-up, actually I have not much progress about this ticket, and plan to finish it in following two month with my available time. But I afraid this may impact community release plan or something. If it is, I think maybe some one can take care of this ticket...Sorry for this bad news, due to I have my schedule totally full recently for dealing with company's requirements... Tks for your understanding. OnlineMerge should be extended to allow bulk merging Key: HBASE-10003 URL: https://issues.apache.org/jira/browse/HBASE-10003 Project: HBase Issue Type: Improvement Components: Admin, Usability Affects Versions: 0.98.0, 0.94.6 Reporter: Clint Heath Assignee: takeshi.miao Priority: Critical Labels: noob Now that we have Online Merge capabilities, the function of that tool should be extended to make it much easier for HBase operations folks to use. Currently it is a very manual process (one fraught with confusion) to hand pick two regions that are contiguous to each other in the META table such that the admin can manually request those two regions to be merged. In the real world, when admins find themselves wanting to merge regions, it's usually because they've greatly increased their hbase.hregion.max.filesize property and they have way too many regions on a table and want to reduce the region count for that entire table quickly and easily. Why can't the OnlineMerge command just take a -max argument along with a table name which tells it to go ahead and merge all regions of said table until the resulting regions are all of max size? This takes the voodoo out of the process and quickly gets the admin what they're looking for. As part of this improvement, I also suggest a -regioncount argument for OnlineMerge, which will attempt to reduce the table's region count down to the specified #. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10251) Restore API Compat for PerformanceEvaluation.generateValue()
[ https://issues.apache.org/jira/browse/HBASE-10251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001115#comment-14001115 ] Hudson commented on HBASE-10251: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-10251 Restore API Compat for PerformanceEvaluation.generateValue() (Dima Spivak via JD) (jdcryans: rev 1594638) * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java Restore API Compat for PerformanceEvaluation.generateValue() Key: HBASE-10251 URL: https://issues.apache.org/jira/browse/HBASE-10251 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.0, 0.98.1, 0.99.0, 0.98.2 Reporter: Aleksandr Shulman Assignee: Dima Spivak Labels: api_compatibility Fix For: 0.99.0, 0.98.3 Attachments: HBASE-10251-v2.patch, HBASE_10251.patch, HBASE_10251_v3.patch Observed: A couple of my client tests fail to compile against trunk because the method PerformanceEvaluation.generateValue was removed as part of HBASE-8496. This is an issue because it was used in a number of places, including unit tests. Since we did not explicitly label this API as private, it's ambiguous as to whether this could/should have been used by people writing apps against 0.96. If they used it, then they would be broken upon upgrade to 0.98 and trunk. Potential Solution: The method was renamed to generateData, but the logic is still the same. We can reintroduce it as deprecated in 0.98, as compat shim over generateData. The patch should be a few lines. We may also consider doing so in trunk, but I'd be just as fine with leaving it out. More generally, this raises the question about what other code is in this grey-area, where it is public, is used outside of the package, but is not explicitly labeled with an AudienceInterface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11143) Improve replication metrics
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001109#comment-14001109 ] Hudson commented on HBASE-11143: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-11143 Improve replication metrics. (larsh: rev 1594469) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/MetricsSource.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java Improve replication metrics --- Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.99.0, 0.94.20, 0.98.3 Attachments: 11143-0.94-v2.txt, 11143-0.94-v3.txt, 11143-0.94.txt, 11143-trunk.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. Edit: Also adds a new shippedKBs metric to track the amount of data that is shipped via replication. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11189) Subprocedure should be marked as complete upon failure
[
https://issues.apache.org/jira/browse/HBASE-11189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001112#comment-14001112
]
Hudson commented on HBASE-11189:
SUCCESS: Integrated in HBase-0.98 #307 (See
[https://builds.apache.org/job/HBase-0.98/307/])
HBASE-11189 Subprocedure should be marked as complete upon failure (tedyu: rev
1595356)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureMember.java
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/Subprocedure.java
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/procedure/TestZKProcedure.java
Subprocedure should be marked as complete upon failure
--
Key: HBASE-11189
URL: https://issues.apache.org/jira/browse/HBASE-11189
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
Assignee: Ted Yu
Fix For: 0.99.0, 0.98.3
Attachments: 11189-v1.txt, 11189-v2.txt
ProcedureMember#submitSubprocedure() uses the following check:
{code}
if (!rsub.isComplete()) {
LOG.error(Subproc ' + procName + ' is already running. Bailing
out);
return false;
}
{code}
If a subprocedure of that name previously ran but failed, its complete field
would stay false, leading to early bailout.
A failed subprocedure should mark itself complete.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11137) Add mapred.TableSnapshotInputFormat
[ https://issues.apache.org/jira/browse/HBASE-11137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001108#comment-14001108 ] Hudson commented on HBASE-11137: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-11137 Add mapred.TableSnapshotInputFormat (ndimiduk: rev 1594984) * /hbase/branches/0.98/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableSnapshotInputFormat.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSnapshotInputFormat.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableSnapshotInputFormat.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java Add mapred.TableSnapshotInputFormat --- Key: HBASE-11137 URL: https://issues.apache.org/jira/browse/HBASE-11137 Project: HBase Issue Type: Improvement Components: mapreduce, Performance Affects Versions: 0.98.0, 0.96.2 Reporter: Nick Dimiduk Assignee: Nick Dimiduk Fix For: 0.99.0, 0.98.3 Attachments: HBASE-11137.00-0.98.patch, HBASE-11137.00.patch, HBASE-11137.01.patch, HBASE-11137.01_rerun.patch, HBASE-11137.02-0.98.patch, HBASE-11137.02.patch We should have feature parity between mapreduce and mapred implementations. This is important for Hive. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11166) Categorize tests in hbase-prefix-tree module
[ https://issues.apache.org/jira/browse/HBASE-11166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001110#comment-14001110 ] Hudson commented on HBASE-11166: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) HBASE-11166 Categorize tests in hbase-prefix-tree module (Rekha Joshi) (tedyu: rev 1595539) * /hbase/branches/0.98/hbase-prefix-tree/pom.xml * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/keyvalue/TestKeyValueTool.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/blockmeta/TestBlockMeta.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/builder/TestTokenizer.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/builder/TestTreeDepth.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/column/TestColumnBuilder.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/row/TestPrefixTreeSearcher.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/row/TestRowEncoder.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/codec/prefixtree/timestamp/TestTimestampEncoder.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/util/bytes/TestByteRange.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/util/vint/TestFIntTool.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/util/vint/TestVIntTool.java * /hbase/branches/0.98/hbase-prefix-tree/src/test/java/org/apache/hadoop/hbase/util/vint/TestVLongTool.java Categorize tests in hbase-prefix-tree module Key: HBASE-11166 URL: https://issues.apache.org/jira/browse/HBASE-11166 Project: HBase Issue Type: Test Affects Versions: 0.98.2 Reporter: Ted Yu Assignee: Rekha Joshi Priority: Minor Fix For: 0.99.0, 0.98.3 Attachments: 11166-v2.txt, HBASE-11166.1.patch, HBASE-11166.3.patch Jeff Bowles discovered that tests in hbase-prefix-tree module, e.g. TestTimestampEncoder, don't have test category -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6990: -- Resolution: Fixed Fix Version/s: 0.98.3 0.96.3 0.99.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pretty print TTL Key: HBASE-6990 URL: https://issues.apache.org/jira/browse/HBASE-6990 Project: HBase Issue Type: Improvement Components: Usability Reporter: Jean-Daniel Cryans Assignee: Esteban Gutierrez Priority: Minor Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: HBASE-6990.v0.patch, HBASE-6990.v1.patch, HBASE-6990.v2.patch, HBASE-6990.v3.patch, HBASE-6990.v4.patch I've seen a lot of users getting confused by the TTL configuration and I think that if we just pretty printed it it would solve most of the issues. For example, let's say a user wanted to set a TTL of 90 days. That would be 7776000. But let's say that it was typo'd to 7776 instead, it gives you 900 days! So when we print the TTL we could do something like x days, x hours, x minutes, x seconds (real_ttl_value). This would also help people when they use ms instead of seconds as they would see really big values in there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10561) Forward port: HBASE-10212 New rpc metric: number of active handler
[ https://issues.apache.org/jira/browse/HBASE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1400#comment-1400 ] Hudson commented on HBASE-10561: SUCCESS: Integrated in HBase-0.98 #307 (See [https://builds.apache.org/job/HBase-0.98/307/]) Amend HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594133) * /hbase/branches/0.98/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java Amend HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594130) * /hbase/branches/0.98/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * /hbase/branches/0.98/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestRpcMetrics.java HBASE-10561 Forward port: HBASE-10212 New rpc metric: number of active handler (liangxie: rev 1594118) * /hbase/branches/0.98/hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapper.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/FifoRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperImpl.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcScheduler.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerWrapperStub.java Forward port: HBASE-10212 New rpc metric: number of active handler -- Key: HBASE-10561 URL: https://issues.apache.org/jira/browse/HBASE-10561 Project: HBase Issue Type: Sub-task Components: IPC/RPC Reporter: Lars Hofhansl Assignee: Liang Xie Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: HBASE-10561.txt The metrics implementation has changed a lot in 0.96. Forward port HBASE-10212 to 0.96 and later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11134) Add a -list-snapshots option to SnapshotInfo
[
https://issues.apache.org/jira/browse/HBASE-11134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001113#comment-14001113
]
Hudson commented on HBASE-11134:
SUCCESS: Integrated in HBase-0.98 #307 (See
[https://builds.apache.org/job/HBase-0.98/307/])
HBASE-11134 Add a -list-snapshots option to SnapshotInfo (mbertozzi: rev
1594855)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
Add a -list-snapshots option to SnapshotInfo
Key: HBASE-11134
URL: https://issues.apache.org/jira/browse/HBASE-11134
Project: HBase
Issue Type: Improvement
Components: snapshots
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
Attachments: HBASE-11134-v0.patch, HBASE-11134-v1.patch
Add a -list-snapshots option to SnapshotInfo to show all the snapshots
available. Also add a -remote-dir option to simplify the usage of
SnapshotInfo in case the snapshot dir is not the one of the current hbase
cluster
{code}
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -list-snapshots
SNAPSHOT | CREATION TIME| TABLE NAME
foo | 2014-05-07T22:40:13 | testtb
bar | 2014-05-07T22:40:16 | testtb
$ hbase org.apache.hadoop.hbase.snapshot.SnapshotInfo -remote-dir
file:///backup/ -snapshot my_local_snapshot
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10877) HBase non-retriable exception list should be expanded
[
https://issues.apache.org/jira/browse/HBASE-10877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001146#comment-14001146
]
Nan Zhu commented on HBASE-10877:
-
met the same problem with 0.98.1 + HDFS 2.3, the full stack looks like:
[code]
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0.0:0
failed 4 times, most recent failure: Exception failure in TID 6 on host
ip-172-31-34-177.us-west-2.compute.internal: java.lang.IllegalAccessError:
com/google/protobuf/HBaseZeroCopyByteString
org.apache.hadoop.hbase.protobuf.RequestConverter.buildRegionSpecifier(RequestConverter.java:930)
org.apache.hadoop.hbase.protobuf.RequestConverter.buildGetRowOrBeforeRequest(RequestConverter.java:133)
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRowOrBefore(ProtobufUtil.java:1466)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1236)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1110)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:1067)
org.apache.hadoop.hbase.client.AsyncProcess.findDestLocation(AsyncProcess.java:356)
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:301)
org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:955)
org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1239)
org.apache.hadoop.hbase.client.HTable.close(HTable.java:1276)
org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordWriter.close(TableOutputFormat.java:112)
org.apache.spark.rdd.PairRDDFunctions.org$apache$spark$rdd$PairRDDFunctions$$writeShard$1(PairRDDFunctions.scala:720)
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:730)
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:730)
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
org.apache.spark.scheduler.Task.run(Task.scala:51)
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:744)
[code]
can anyone tell me how to fix it?
HBase non-retriable exception list should be expanded
-
Key: HBASE-10877
URL: https://issues.apache.org/jira/browse/HBASE-10877
Project: HBase
Issue Type: Improvement
Reporter: Sergey Shelukhin
Priority: Minor
Example where retries do not make sense:
{noformat}
2014-03-31 20:54:27,765 WARN [InputInitializer [Map 1] #0]
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation:
Encountered problems when prefetch hbase:meta table:
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=35, exceptions:
Mon Mar 31 20:45:17 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: class
com.google.protobuf.HBaseZeroCopyByteString cannot access its superclass
com.google.protobuf.LiteralByteString
Mon Mar 31 20:45:17 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:17 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:18 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:20 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:24 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:34 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:45 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:45:55 UTC 2014,
org.apache.hadoop.hbase.client.RpcRetryingCaller@343d511e,
java.lang.IllegalAccessError: com/google/protobuf/HBaseZeroCopyByteString
Mon Mar 31 20:46:05 UTC 2014,
[jira] [Commented] (HBASE-11007) BLOCKCACHE in schema descriptor seems not aptly named
[
https://issues.apache.org/jira/browse/HBASE-11007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001164#comment-14001164
]
Hadoop QA commented on HBASE-11007:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645436/11007v2.txt
against trunk revision .
ATTACHMENT ID: 12645436
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9538//console
This message is automatically generated.
BLOCKCACHE in schema descriptor seems not aptly named
-
Key: HBASE-11007
URL: https://issues.apache.org/jira/browse/HBASE-11007
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.19
Reporter: Varun Sharma
Assignee: Varun Sharma
Priority: Minor
Fix For: 0.99.0
Attachments: 11007.txt, 11007v2.txt
Hi,
It seems that setting BLOCKCACHE key to false will disable the Data blocks
from being cached but will continue to cache bloom and index blocks. This
same property seems to be called cacheDataOnRead inside CacheConfig.
Should this be called CACHE_DATA_ON_READ instead of BLOCKCACHE similar to the
other CACHE_DATA_ON_WRITE/CACHE_INDEX_ON_WRITE. We got quite confused and
ended up adding our own property CACHE_DATA_ON_READ - we also added some unit
tests for the same.
What do folks think about this ?
Thanks
Varun
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6506) Setting CACHE_BLOCKS to false in an hbase shell scan doesn't work
[
https://issues.apache.org/jira/browse/HBASE-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001166#comment-14001166
]
Hadoop QA commented on HBASE-6506:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645445/6506.txt
against trunk revision .
ATTACHMENT ID: 12645445
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScan(TestMultiTableInputFormat.java:244)
at
org.apache.hadoop.hbase.mapreduce.TestMultiTableInputFormat.testScanYZYToEmpty(TestMultiTableInputFormat.java:195)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9540//console
This message is automatically generated.
Setting CACHE_BLOCKS to false in an hbase shell scan doesn't work
-
Key: HBASE-6506
URL: https://issues.apache.org/jira/browse/HBASE-6506
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.94.0
Reporter: Josh Wymer
Priority: Minor
Labels: cache, ruby, scan, shell
Fix For: 0.99.0
Attachments: 6506.txt
Original Estimate: 1m
Remaining Estimate: 1m
I was attempting to prevent blocks from being cached by setting CACHE_BLOCKS
= false in the hbase shell when doing a scan but I kept seeing tons of
evictions when I ran it. After inspecting table.rb I found this line:
cache = args[CACHE_BLOCKS] || true
The problem then is that if CACHE_BLOCKS is false then this expression will
always return true. Therefore, it's impossible to turn off block caching.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-6990) Pretty print TTL
[ https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001240#comment-14001240 ] Esteban Gutierrez commented on HBASE-6990: -- [~lhofhansl] I think is just a nice to have, but not a critical thing. Pretty print TTL Key: HBASE-6990 URL: https://issues.apache.org/jira/browse/HBASE-6990 Project: HBase Issue Type: Improvement Components: Usability Reporter: Jean-Daniel Cryans Assignee: Esteban Gutierrez Priority: Minor Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: HBASE-6990.v0.patch, HBASE-6990.v1.patch, HBASE-6990.v2.patch, HBASE-6990.v3.patch, HBASE-6990.v4.patch I've seen a lot of users getting confused by the TTL configuration and I think that if we just pretty printed it it would solve most of the issues. For example, let's say a user wanted to set a TTL of 90 days. That would be 7776000. But let's say that it was typo'd to 7776 instead, it gives you 900 days! So when we print the TTL we could do something like x days, x hours, x minutes, x seconds (real_ttl_value). This would also help people when they use ms instead of seconds as they would see really big values in there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11155) Fix Validation Errors in Ref Guide
[ https://issues.apache.org/jira/browse/HBASE-11155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001221#comment-14001221 ] Misty Stanley-Jones commented on HBASE-11155: - [~stack] yes, happy to have you commit it. Let's revisit the right way to do the fallback content elsewhere, as it is mostly a build issue. I like the idea of using a real hbase-default.xml (a last-known-good one) with a big fat label saying it's stale. Fix Validation Errors in Ref Guide -- Key: HBASE-11155 URL: https://issues.apache.org/jira/browse/HBASE-11155 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11155.patch Before I do serious documentation work, I have to fix all of the validation errors that are somehow not causing the Ref Guide to break the builds. I will attach one patch per file -- that's the easiest way I know how to do it. I will try not to make any content changes, only validation changes. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11090) Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption
[ https://issues.apache.org/jira/browse/HBASE-11090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11090: --- Fix Version/s: (was: 0.98.3) Backport HBASE-11083 ExportSnapshot should provide capability to limit bandwidth consumption Key: HBASE-11090 URL: https://issues.apache.org/jira/browse/HBASE-11090 Project: HBase Issue Type: Task Reporter: Ted Yu Assignee: Ted Yu Fix For: 0.99.0 Attachments: 11090-0.98-v1.txt, 11090-trunk-v2.txt, 11090-trunk.txt HBASE-11083 allows ExportSnapshot to limit bandwidth usage. Here is *one* approach for backporting: Create the following classes (class name is tentative): hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/util/ThrottledInputStream.java hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/util/ThrottledInputStream.java each of which extends the corresponding ThrottledInputStream in hadoop-1 / hadoop-2 ExportSnapshot would reference util.ThrottledInputStream, depending on which compatibility module gets bundled. ThrottledInputStream.java in hadoop-1 branch was backported through MAPREDUCE-5081 which went into 1.2.0 release. We need to decide how hadoop releases earlier than 1.2.0 should be supported. *Second* approach for backporting is to make a copy of ThrottledInputStream and include it in hbase codebase. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001210#comment-14001210 ] Lars Hofhansl commented on HBASE-11165: --- Yep. My point was that we need to tackle from a lot of different angles: * autotuning memstore sizes, and lazy allocation (as Andy says), or sharing memstores * make large regions more workable, splits, compations, etc * allow more RAM to be used by region server (off heap memstores) * fast assignments (colocating HMaster and the RS hosting META, 50m should then not be a problem if everything is local to a single machine) * allow smaller units of computation in M/R * split META? And then colocate with multiple HMasters? Right now it is hard to even utilize the disks on a reasonably sized commodity machine... Say you can host 100 regions on a server (30gb heap). With 20gb regions the max diskspace you can utilize per box is this 100*20gb*3 (HDFS replication factor) = 6tb... Of course the replicas are distributed across data nodes, but the averages still hold, you need 1 box for each 6tb of diskspace. Larger regions and smaller memstores can fix that, but currently these lead to other issues. (See also http://hadoop-hbase.blogspot.com/2013/01/hbase-region-server-memory-sizing.html, where I blogged about this a while ago) Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[ https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001252#comment-14001252 ] Vandana Ayyalasomayajula commented on HBASE-11104: -- I looked at https://builds.apache.org/job/PreCommit-HBASE-Build/9537/artifact/trunk/patchprocess/patchJavadocWarnings.txt Nothing seems related to patch. IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization Key: HBASE-11104 URL: https://issues.apache.org/jira/browse/HBASE-11104 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: 11104-v1.txt, HBASE-11104_98_v3.patch, HBASE-11104_trunk.patch, HBASE-11104_trunk.patch IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships the HBase dependencies. However, call to TableMapReduceUtil.initCredentials(job) is missing - making this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11199) One-time effort to pretty-print the Docbook XML, to make further patch review easier
Misty Stanley-Jones created HBASE-11199: --- Summary: One-time effort to pretty-print the Docbook XML, to make further patch review easier Key: HBASE-11199 URL: https://issues.apache.org/jira/browse/HBASE-11199 Project: HBase Issue Type: Task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Careful not to pretty-print literal layouts and document (at least in this JIRA) the pretty-printing process for next time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11190) Fix easy typos in documentation
[ https://issues.apache.org/jira/browse/HBASE-11190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001217#comment-14001217 ] Misty Stanley-Jones commented on HBASE-11190: - [~stack] It wasn't intended to be pretty-printing, but correction of a very specific list of typos found by a colleague. If it was pretty-printing, that's a side effect. Regardless you have asked for this a few times now so I will create a JIRA and do it. :) I think it won't be as easy as you might think, to avoid accidentally indenting programlistings and screen output. But I will undertake it, to make later patch reviews easier. Fix easy typos in documentation --- Key: HBASE-11190 URL: https://issues.apache.org/jira/browse/HBASE-11190 Project: HBase Issue Type: Bug Components: documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Priority: Trivial Fix For: 0.99.0 Attachments: HBASE-11190.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11120: Attachment: HBASE-11120.patch This is my attempt to understand and explain the new compaction algorithms. I think I have probably missed some bits and misconstrued some bits. Please provide feedback and corrections as necessary. I have left the old algorithm explanation in place and just re-labeled it as prior to 0.96.x. Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11120) Update documentation about major compaction algorithm
[ https://issues.apache.org/jira/browse/HBASE-11120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11120: Status: Patch Available (was: Open) Update documentation about major compaction algorithm - Key: HBASE-11120 URL: https://issues.apache.org/jira/browse/HBASE-11120 Project: HBase Issue Type: Bug Components: Compaction, documentation Affects Versions: 0.98.2 Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11120.patch [14:20:38] jdcryans seems that there's http://hbase.apache.org/book.html#compaction and http://hbase.apache.org/book.html#managed.compactions [14:20:56] jdcryans the latter doesn't say much, except that you should manage them [14:21:44] jdcryans the former gives a good description of the _old_ selection algo [14:45:25] jdcryans this is the new selection algo since C5 / 0.96.0: https://issues.apache.org/jira/browse/HBASE-7842 -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11199) One-time effort to pretty-print the Docbook XML, to make further patch review easier
[ https://issues.apache.org/jira/browse/HBASE-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11199: Attachment: HBASE-11199.patch So, Oxygen has a reasonable pretty-print system for Docbook XML. You just go to the text view (non-wysiwyg), cmd+A to select-all, cmd+shift+P to pretty-print. It ignores programlisting, screen, literallayout, and others that seem reasonable. It wraps long lines when possible, as well. at around 100 characters, occasionally overflowing a character or two because of word breaks. Let me know if you want any of these settings tweaked and I will do it and re-submit the patch. I can also look around for a command-line reformatter which we could hook in with the automatic builds or somehow with the check-in process, if you like that approach. I know that they exist because I've used one before, and specifically had it set up with reasonable defaults for Docbook. I just can't remember what it was called! One-time effort to pretty-print the Docbook XML, to make further patch review easier Key: HBASE-11199 URL: https://issues.apache.org/jira/browse/HBASE-11199 Project: HBase Issue Type: Task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11199.patch Careful not to pretty-print literal layouts and document (at least in this JIRA) the pretty-printing process for next time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11199) One-time effort to pretty-print the Docbook XML, to make further patch review easier
[ https://issues.apache.org/jira/browse/HBASE-11199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Misty Stanley-Jones updated HBASE-11199: Status: Patch Available (was: Open) One-time effort to pretty-print the Docbook XML, to make further patch review easier Key: HBASE-11199 URL: https://issues.apache.org/jira/browse/HBASE-11199 Project: HBase Issue Type: Task Components: documentation Reporter: Misty Stanley-Jones Assignee: Misty Stanley-Jones Attachments: HBASE-11199.patch Careful not to pretty-print literal layouts and document (at least in this JIRA) the pretty-printing process for next time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11147) Avoid creating List of KeyValue in FilterBase#filterRowCells(ListCell)
[
https://issues.apache.org/jira/browse/HBASE-11147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly reassigned HBASE-11147:
---
Assignee: Gustavo Anatoly
Avoid creating List of KeyValue in FilterBase#filterRowCells(ListCell)
Key: HBASE-11147
URL: https://issues.apache.org/jira/browse/HBASE-11147
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Trivial
Currently a List of KeyValue's is always created:
{code}
ListKeyValue kvs = new ArrayListKeyValue(ignored.size());
{code}
When passed ignored List is of KeyValue (which is the only implementation of
Cell at the moment), the above step should be avoided.
This would reduce creation of short-lived objects.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HBASE-11146) HMaster instantiates both MasterCoprocessorHost and RegionServerCoprocessorHost
[ https://issues.apache.org/jira/browse/HBASE-11146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Tian resolved HBASE-11146. Resolution: Won't Fix work as design HMaster instantiates both MasterCoprocessorHost and RegionServerCoprocessorHost --- Key: HBASE-11146 URL: https://issues.apache.org/jira/browse/HBASE-11146 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.99.0 Reporter: Qiang Tian Assignee: Qiang Tian See HBASE-11096. in 0.99, HRegionServer is the base class of HMaster. master and regionserver share the same run method, the master instantiates both MasterCoprocessorHost and RegionServerCoprocessorHost below is example logs, a coprocessor is start/stop twice--one is for real regionserver, the other is for the RegionServerCoprocessorHost in master. 2014-05-08 00:33:51,632 INFO [M:0;bdvm135:36021] coprocessor.TestCoprocessorStop$FooCoprocessor(66): st art coprocessor on regionserver 2014-05-08 00:33:51,633 INFO [RS:0;bdvm135:47513] coprocessor.TestCoprocessorStop$FooCoprocessor(66): s tart coprocessor on regionserver ... 2014-05-08 00:34:03,166 INFO [main] regionserver.HRegionServer(1624): call stack of stop java.io.IOException at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1624) at org.apache.hadoop.hbase.master.ServerManager.shutdownCluster(ServerManager.java:975) at org.apache.hadoop.hbase.master.HMaster.shutdown(HMaster.java:1623) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:256) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:437) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:519) at org.apache.hadoop.hbase.coprocessor.TestCoprocessorStop.testStopped(TestCoprocessorStop.java: 114) ... 2014-05-08 00:34:03,215 INFO [main] regionserver.HRegionServer(1629): rsHost code path called 2014-05-08 00:34:03,228 DEBUG [main] coprocessor.CoprocessorHost(258): Stop coprocessor org.apache.hadoo p.hbase.coprocessor.TestCoprocessorStop$FooCoprocessor 2014-05-08 00:34:03,462 INFO [main] coprocessor.TestCoprocessorStop$FooCoprocessor(88): create file hdf s://localhost:8155/user/tianq/test-data/f0c9423c-e505-4feb-907e-c7bd6e16545b/regionserver1399534399680 r eturn rc true ... 2014-05-08 00:34:03,482 INFO [main] regionserver.HRegionServer(1624): call stack of stop java.io.IOException at org.apache.hadoop.hbase.regionserver.HRegionServer.stop(HRegionServer.java:1624) at org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:264) at org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:437) at org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:519) at org.apache.hadoop.hbase.coprocessor.TestCoprocessorStop.testStopped(TestCoprocessorStop.java: 114) 2014-05-08 00:34:03,485 INFO [main] regionserver.HRegionServer(1629): rsHost code path called 2014-05-08 00:34:03,485 DEBUG [main] coprocessor.CoprocessorHost(258): Stop coprocessor org.apache.hadoo p.hbase.coprocessor.TestCoprocessorStop$FooCoprocessor 2014-05-08 00:34:03,493 INFO [main] coprocessor.TestCoprocessorStop$FooCoprocessor(88): create file hdf s://localhost:8155/user/tianq/test-data/f0c9423c-e505-4feb-907e-c7bd6e16545b/regionserver1399534399680 r eturn rc false -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked
[ https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14001345#comment-14001345 ] Qiang Tian commented on HBASE-11096: Thanks [~lhofhansl] [~apurtell], is it OK for trunk? thanks! stop method of Master and RegionServer coprocessor is not invoked -- Key: HBASE-11096 URL: https://issues.apache.org/jira/browse/HBASE-11096 Project: HBase Issue Type: Bug Affects Versions: 0.96.2, 0.98.1, 0.94.19 Reporter: Qiang Tian Assignee: Qiang Tian Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.patch the stop method of coprocessor specified by hbase.coprocessor.master.classes and hbase.coprocessor.regionserver.classes is not invoked. If coprocessor allocates OS resources, it could cause master/regionserver resource leak or hang during exit. -- This message was sent by Atlassian JIRA (v6.2#6252)
