[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422953#comment-13422953 ] stack commented on HBASE-6455: -- +1 on patch org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path -- Key: HBASE-6455 URL: https://issues.apache.org/jira/browse/HBASE-6455 Project: HBase Issue Type: Bug Components: performance Affects Versions: 0.94.0, 0.96.0 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Labels: benchmark Fix For: 0.96.0 Attachments: HBASE-6455_trunk.patch I was the running PerformanceEvaluation test with a modified job configuration where the job output path is created before the splits and that unmasked the issue because the the PeInputFormat.getSplits() function expects to find only files under the input path. The attached patch addresses both the issues. # Creates separate input and output path rooted under a single folder e.g. user_home/performance_evaluation/MMddHHmmss/inputs and user_home/performance_evaluation/MMddHHmmss/outputs, and # The PeInputFormat.getSplits(), now skips any folder found under the input path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6460) hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
[
https://issues.apache.org/jira/browse/HBASE-6460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jie Huang updated HBASE-6460:
-
Attachment: hbase-6460.patch
A quick and simple fix in the patch file
hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
-
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
Attachments: hbase-6460.patch
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage
information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6453) Hbase Replication point in time feature
[ https://issues.apache.org/jira/browse/HBASE-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422872#comment-13422872 ] terry zhang commented on HBASE-6453: Hiļ¼Stack. I think we can use point in time feature with Snapshots feature (HBASE-6055) in below case. 1. Master cluster in data center China do a Snapshot at Time A 2. copy the Snapshot to Slave cluster in data center US and Set Replication Time stamp to A 3. Restore snapshots (HBASE-6230 ) for Slave cluster and start replication. Them Slave cluster data will be as same as Master Cluster . And data is more safe and US user can visit Slave cluster for getting or scanning data to decrease the stress for China data center. Enable replication can not control the accurate time so may be it will lose some data or replicate some useless data. Mysql also has point in time/position feature in it replication Framework. It is very convenience for data center administrate to use. We can give some better name for this operation cause I am not good at naming ... Hbase Replication point in time feature --- Key: HBASE-6453 URL: https://issues.apache.org/jira/browse/HBASE-6453 Project: HBase Issue Type: New Feature Components: replication Affects Versions: 0.94.0 Reporter: terry zhang Assignee: terry zhang Attachments: hbase-6453-v1.patch Now we can not control when hbase replication start to work. this patch support we set a time stamp filter . All the row which is below this time stamp will not be replicated. We also can delete and show this time stamp in hbase shell if we want to change it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Summary: Pluggable compaction policies via coprocessors (was: Pluggable policy for smallestReadPoint in HRegion) Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengsheng Huang updated HBASE-6456: Attachment: HBASE-6456.patch Attached a patch for it Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Priority: Minor Attachments: HBASE-6456.patch The command I ran is as below: bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1 And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6458) reuse BinaryComparator
binlijin created HBASE-6458: --- Summary: reuse BinaryComparator Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6456) Export Utility throws ClassNotFoundException
Shengsheng Huang created HBASE-6456: --- Summary: Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Priority: Minor The command I ran is as below: bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1 And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6460) hbck -repareHoles shortcut doesn't enable -fixHdfsOrphans
Jie Huang created HBASE-6460:
Summary: hbck -repareHoles shortcut doesn't enable
-fixHdfsOrphans
Key: HBASE-6460
URL: https://issues.apache.org/jira/browse/HBASE-6460
Project: HBase
Issue Type: Bug
Components: hbck
Affects Versions: 0.94.0, 0.96.0
Reporter: Jie Huang
Priority: Minor
According to the hbck's help info, shortcut - -repairHoles will enable
-fixHdfsOrphans as below.
{noformat}
-repairHoles Shortcut for -fixAssignments -fixMeta -fixHdfsHoles
-fixHdfsOrphans
{noformat}
However, in the implementation, the function fsck.setFixHdfsOrphans(false);
is called in -repairHoles. This is not consistent with the usage information.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable policy for smallestReadPoint in HRegion
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Attachment: 6427-notReady.txt Here's an initial idea for a patch. For this to work coprocessors need to have access to KeyValueScanner, StoreScanner, etc. So that is violating/leaking some of the previous abstractions. This patch would allow a coprocessor to actually produce a scanner for a flush or a compaction, while not requiring it to reimplement all the logic. A coprocessor can now in fact just override the TTL and/or # versions for a flush/compaction. Please let me know what you think and how to improve this. And for the love of god, please think of better names for the two new hooks. Pluggable policy for smallestReadPoint in HRegion - Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422971#comment-13422971 ] stack commented on HBASE-5705: -- All of above seems reasonable. Here is some response. bq. ...Could you please point me to the block of code in the patch where this is. I do not see it now. bq. ...and in the case of ProtobufRpcEngine, they are done using RpcRequestWritable (a thin wrapper over PB objects). The Writable wrapper goes away when we move to pb engine? bq. ...In that sense, we need the message size. Its header length only right? Thats ok I'd say. But for the value, when its a 50M cell, we won't have to read it into a buffer to find its size before sending will we when we are pb? (pb does this internally probably but we do we have to do it again outside of pb to put it in rpc header?) Introduce Protocol Buffer RPC engine Key: HBASE-5705 URL: https://issues.apache.org/jira/browse/HBASE-5705 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB aware can be made to go through this RPC engine. The approach, in my current thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422956#comment-13422956 ] stack commented on HBASE-6431: -- @Alex What do you think of Ted patch? Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422873#comment-13422873 ] Hadoop QA commented on HBASE-6431: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537948/6431-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2439//console This message is automatically generated. Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-6458: Attachment: HBASE-6458-0.94.patch reuse BinaryComparator -- Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422946#comment-13422946 ] wanbin commented on HBASE-6457: --- By modifying the file (add partitionid as suffix) name may be able to repair the problem, when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss --- Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.92.1 Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wanbin updated HBASE-6457: -- Affects Version/s: 0.92.1 when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss --- Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.92.1 Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6459) improve speed of create table
[
https://issues.apache.org/jira/browse/HBASE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423001#comment-13423001
]
zhou wenjian commented on HBASE-6459:
-
I'll give a patch later
improve speed of create table
-
Key: HBASE-6459
URL: https://issues.apache.org/jira/browse/HBASE-6459
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Fix For: 0.94.1
in CreateTableHandler
for (int regionIdx = 0; regionIdx this.newRegions.length; regionIdx++) {
HRegionInfo newRegion = this.newRegions[regionIdx];
// 1. Create HRegion
HRegion region = HRegion.createHRegion(newRegion,
this.fileSystemManager.getRootDir(), this.conf,
this.hTableDescriptor, null, false, true);
regionInfos.add(region.getRegionInfo());
if (regionIdx % batchSize == 0) {
// 2. Insert into META
MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
regionInfos.clear();
}
// 3. Close the new region to flush to disk. Close log file too.
region.close();
}
All the region will be create serially.
If we have thousands of regions, that will be a huge cost.
We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shengsheng Huang updated HBASE-6372: Attachment: HBASE-6372.patch I think this is useful. Attached a patch for it. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
wanbin created HBASE-6457: - Summary: when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks
[ https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422909#comment-13422909 ] Alex Feinberg commented on HBASE-5991: -- Mostly done in terms of implementing the locks itself based on the recipe (with recoverable zooKeeper). Should have this integrated into HMaster (in place of my DistributedLock code) and have a diff ready soon. Introduce sequential ZNode based read/write locks -- Key: HBASE-5991 URL: https://issues.apache.org/jira/browse/HBASE-5991 Project: HBase Issue Type: Improvement Reporter: Alex Feinberg Assignee: Alex Feinberg This is a continuation of HBASE-5494: Currently table-level write locks have been implemented using non-sequential ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to track converting the table-level locks to sequential ZNodes and supporting read-write locks, as to solve the issue of preventing schema changes during region splits or merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-6458: Attachment: HBASE-6458-0.94-2.patch reuse BinaryComparator -- Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6459) improve speed of create table
zhou wenjian created HBASE-6459:
---
Summary: improve speed of create table
Key: HBASE-6459
URL: https://issues.apache.org/jira/browse/HBASE-6459
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Fix For: 0.94.1
in CreateTableHandler
for (int regionIdx = 0; regionIdx this.newRegions.length; regionIdx++) {
HRegionInfo newRegion = this.newRegions[regionIdx];
// 1. Create HRegion
HRegion region = HRegion.createHRegion(newRegion,
this.fileSystemManager.getRootDir(), this.conf,
this.hTableDescriptor, null, false, true);
regionInfos.add(region.getRegionInfo());
if (regionIdx % batchSize == 0) {
// 2. Insert into META
MetaEditor.addRegionsToMeta(this.catalogTracker, regionInfos);
regionInfos.clear();
}
// 3. Close the new region to flush to disk. Close log file too.
region.close();
}
All the region will be create serially.
If we have thousands of regions, that will be a huge cost.
We can improve it by create the region in parallel
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423137#comment-13423137 ] Zhihong Ted Yu commented on HBASE-6455: --- Integrated to trunk. Thanks for the patch, Aditya. Thanks for the review, Stack. org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path -- Key: HBASE-6455 URL: https://issues.apache.org/jira/browse/HBASE-6455 Project: HBase Issue Type: Bug Components: performance Affects Versions: 0.94.0, 0.96.0 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Labels: benchmark Fix For: 0.96.0 Attachments: HBASE-6455_trunk.patch I was the running PerformanceEvaluation test with a modified job configuration where the job output path is created before the splits and that unmasked the issue because the the PeInputFormat.getSplits() function expects to find only files under the input path. The attached patch addresses both the issues. # Creates separate input and output path rooted under a single folder e.g. user_home/performance_evaluation/MMddHHmmss/inputs and user_home/performance_evaluation/MMddHHmmss/outputs, and # The PeInputFormat.getSplits(), now skips any folder found under the input path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423161#comment-13423161 ] Lars Hofhansl commented on HBASE-6458: -- Yeah, that's stupid. +1 reuse BinaryComparator -- Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423200#comment-13423200 ] Jimmy Xiang commented on HBASE-6272: Patch 6 was uploaded RB to address Ram's comments. Please review. In-memory region state is inconsistent -- Key: HBASE-6272 URL: https://issues.apache.org/jira/browse/HBASE-6272 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang AssignmentManger stores region state related information in several places: regionsInTransition, regions (region info to server name map), and servers (server name to region info set map). However the access to these places is not coordinated properly. It leads to inconsistent in-memory region state information. Sometimes, some region could even be offline, and not in transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423201#comment-13423201 ] Lars Hofhansl commented on HBASE-5954: -- Thinking more about HTable#hsync, I think that would hard to make useful to an application. The application would need to do know which RegionServers to hsync the WAL on (unless we want to flush for a Table, which means all RegionServer hosting regions for this table need to hsync their WAL, and that does not seem to be useful). A similar argument goes for the RPCs (which will be split to multiple RegionServers). So #6 and #7 are out I think. I think Todd was right after all (just took me a long time to come around to it), a flag per Put/Delete/Increment/Append/etc, would be best option. Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[
https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423241#comment-13423241
]
Lars Hofhansl commented on HBASE-5954:
--
Another question: HBase 0.96 now has the hadoop{1|2}-compat projects.
On the other hand I do not want to have completely diverging implementations
for this for 0.94 and 0.96 (which would mean to use reflection in both
branches). Any thoughts on that?
Allow proper fsync support for HBase
Key: HBASE-5954
URL: https://issues.apache.org/jira/browse/HBASE-5954
Project: HBase
Issue Type: Improvement
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Fix For: 0.96.0, 0.94.2
Attachments: 5954-trunk-hdfs-trunk-v2.txt,
5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt,
5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt,
5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423256#comment-13423256 ] Alex Newman commented on HBASE-6431: Just looked at it. It's fine! Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423272#comment-13423272 ] Andrew Purtell commented on HBASE-5954: --- bq. a flag per Put/Delete/Increment/Append/etc, would be best option Makes sense, since both you and Todd got here after giving it consideration. bq. On the other hand I do not want to have completely diverging implementations for this for 0.94 and 0.96 I think we have to take this pain, a reflection based strategy in 0.94 and a module based strategy in 0.96. I this case I'd judge it worth it. But that's going to be a high bar for other things. Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6435: --- Attachment: 6435.v2.patch Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Attachments: 6435.unfinished.patch, 6435.v2.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423278#comment-13423278 ] stack commented on HBASE-6272: -- I'm up for committing this to trunk. What about you Ram? In-memory region state is inconsistent -- Key: HBASE-6272 URL: https://issues.apache.org/jira/browse/HBASE-6272 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang AssignmentManger stores region state related information in several places: regionsInTransition, regions (region info to server name map), and servers (server name to region info set map). However the access to these places is not coordinated properly. It leads to inconsistent in-memory region state information. Sometimes, some region could even be offline, and not in transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423279#comment-13423279 ] Zhihong Ted Yu commented on HBASE-6431: --- Will integrate in the afternoon if there is no objection. Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423280#comment-13423280 ] nkeywal commented on HBASE-6435: v2. May need some clean up on logs + a check to unactivate it for hadoop 2 for example. Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Attachments: 6435.unfinished.patch, 6435.v2.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423282#comment-13423282 ] nkeywal commented on HBASE-6435: + I need to test it on a real cluster (emulating locations on a mini cluster can be dangerous...) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Attachments: 6435.unfinished.patch, 6435.v2.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423275#comment-13423275 ] Andrew Purtell commented on HBASE-5954: --- More on my comment above. We have two options: we can start breaking out reflection into modules in 0.94 too, or save all of that for 0.96. I don't have a strong opinion but if I had to make a choice the refactoring should be in next major version / trunk / currently unstable. Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Attachments: 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, 5954-trunk-hdfs-trunk.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423291#comment-13423291 ] stack commented on HBASE-6458: -- +1 on version2. Want to make one for trunk that we can commit Binlijin? reuse BinaryComparator -- Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423296#comment-13423296 ] stack commented on HBASE-6431: -- Patch is good by me. Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6458) reuse BinaryComparator
[ https://issues.apache.org/jira/browse/HBASE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423299#comment-13423299 ] stack commented on HBASE-6458: -- Binlijin Please fix the title and add a description. As is, the subject is too general (what BinaryComparator?), etc. Thanks. reuse BinaryComparator -- Key: HBASE-6458 URL: https://issues.apache.org/jira/browse/HBASE-6458 Project: HBase Issue Type: Improvement Affects Versions: 0.94.1 Reporter: binlijin Fix For: 0.94.2 Attachments: HBASE-6458-0.94-2.patch, HBASE-6458-0.94.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423302#comment-13423302 ] Alex Newman commented on HBASE-6431: Oh I should mention I ran the patch through our jenkins and it looks good. Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423305#comment-13423305
]
Andrew Purtell commented on HBASE-6427:
---
Or use polymorphism?
{noformat}
+ public InternalScanner preFlush(final
ObserverContextRegionCoprocessorEnvironment c,
+ Store store, KeyValueScanner scanner) throws IOException;
+
+ @Deprecated
public void preFlush(ObserverContextRegionCoprocessorEnvironment e) throws
IOException;
{noformat}
{noformat}
+ public InternalScanner preCompact(final
ObserverContextRegionCoprocessorEnvironment c,
+ Store store, List? extends KeyValueScanner scanners, ScanType
scanType, long earliestPutTs)
+ throws IOException;
+
+ @Deprecated
public void preCompact(ObserverContextRegionCoprocessorEnvironment e
...
{noformat}
Pluggable compaction policies via coprocessors
--
Key: HBASE-6427
URL: https://issues.apache.org/jira/browse/HBASE-6427
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor
Attachments: 6427-notReady.txt
When implementing higher level stores on top of HBase it is necessary to
allow dynamic control over how long KVs must be kept around.
Semi-static config options for ColumnFamilies (# of version or TTL) is not
sufficient.
The simplest way to achieve this is to have a pluggable class to determine
the smallestReadpoint for Region. That way outside code can control what KVs
to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423311#comment-13423311 ] Devaraj Das commented on HBASE-5705: bq. The Writable wrapper goes away when we move to pb engine? Yes (AFAICT). bq. Its header length only right? Thats ok I'd say. But for the value, when its a 50M cell, we won't have to read it into a buffer to find its size before sending will we when we are pb? (pb does this internally probably but we do we have to do it again outside of pb to put it in rpc header?) No we don't have to write to a buffer in the case of PB objects. There is a getSerializedSize method on PB objects (and assuming that the 50M cell is encapsulated in the PB object as a byte-array, PB would simply call byte-array.length...). Introduce Protocol Buffer RPC engine Key: HBASE-5705 URL: https://issues.apache.org/jira/browse/HBASE-5705 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB aware can be made to go through this RPC engine. The approach, in my current thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423310#comment-13423310 ] stack commented on HBASE-6457: -- Wanbin Do we have this issue in trunk? Can you put up sample name-clash? Thanks. when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss --- Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.92.1 Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6372: - Status: Patch Available (was: Open) Looks good (Has some small formatting issues... will fix on application). Running by hadoopqa to make sure its happy. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423320#comment-13423320 ] ramkrishna.s.vasudevan commented on HBASE-6272: --- I just added minor comments. They are just questions. If they are fine i am ok with commit. May be once we get in this, we can fix any bugs in future if any. Thanks Jimmy and Stack. In-memory region state is inconsistent -- Key: HBASE-6272 URL: https://issues.apache.org/jira/browse/HBASE-6272 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang AssignmentManger stores region state related information in several places: regionsInTransition, regions (region info to server name map), and servers (server name to region info set map). However the access to these places is not coordinated properly. It leads to inconsistent in-memory region state information. Sometimes, some region could even be offline, and not in transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423325#comment-13423325 ] Lars Hofhansl commented on HBASE-6427: -- Uh... I like that. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423331#comment-13423331 ] stack commented on HBASE-5705: -- I'm going to commit this in next day unless objection. Introduce Protocol Buffer RPC engine Key: HBASE-5705 URL: https://issues.apache.org/jira/browse/HBASE-5705 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 5705-1.patch, 5705-2.1.patch, 5705-2.2.patch Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB aware can be made to go through this RPC engine. The approach, in my current thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423332#comment-13423332 ] stack commented on HBASE-6272: -- Jimmy, if you've addressed Ram's minor comments, commit I'd say. In-memory region state is inconsistent -- Key: HBASE-6272 URL: https://issues.apache.org/jira/browse/HBASE-6272 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang AssignmentManger stores region state related information in several places: regionsInTransition, regions (region info to server name map), and servers (server name to region info set map). However the access to these places is not coordinated properly. It leads to inconsistent in-memory region state information. Sometimes, some region could even be offline, and not in transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423348#comment-13423348 ] Elliott Clark commented on HBASE-6411: -- bq.what about old MasterMetrics? Are we sure we want to replace it? I though we want to leave it for some time The class is still there so we're not breaking compat. All that we're doing is changing which metrics system exports to file/jmx/ganglia. From the conversations I've had it seems like people are fine with changing to the new metrics system as the backing as long as we don't remove classes that could be used externally. eg. we'll have to leave the current implementation of MetricsHistorgram as it's possible that someone is using it outside of hbase internal code. (That's all just my understanding and if someone disagree's that's fine too.) bq.Should we probably unify that? E.g. push managing sink.ageOfLastAppliedOp metrics and such to xxxMetricsSource? Stuffing dealing with the replication source uses dynamic metric names so it relies on the more general interface and since we wanted those metrics under the same node in jmx, I think the way we have it is fine for now; or at least good enough to leave it as is until after your patch gets applied. Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[
https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423351#comment-13423351
]
Jean-Daniel Cryans commented on HBASE-2214:
---
Following this discussion on user@
http://permalink.gmane.org/gmane.comp.java.hadoop.hbase.user/27936
If the max scanner result size is only set on the region server it's possible
miss rows. Even with the latest patch in this jira it's still possible since
the client doesn't tell the server it's going to use the default instead of
whatever the server is using.
Maybe the {{ClientScanner}} could modify the {{Scan}} that it passes the max
scanner result it uses to the region server
Do HBASE-1996 -- setting size to return in scan rather than count of rows --
properly
-
Key: HBASE-2214
URL: https://issues.apache.org/jira/browse/HBASE-2214
Project: HBase
Issue Type: New Feature
Reporter: stack
Fix For: 0.96.0, 0.94.2
Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt,
HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt,
HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt
The notion that you set size rather than row count specifying how many rows a
scanner should return in each cycle was raised over in hbase-1966. Its a
good one making hbase regular though the data under it may vary.
HBase-1966 was committed but the patch was constrained by the fact that it
needed to not change RPC interface. This issue is about doing hbase-1966 for
0.21 in a clean, unconstrained way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423365#comment-13423365 ] Lars Hofhansl commented on HBASE-6427: -- The part I still have think through is how to handle actual use scans. Be default a user scan will also filter TTL/Versions, so it's one thing to prevent the KVs from being compacted away and another to actually make them visible to user scans. A similar approach can be followed in preScannerOpen, as long as the coprocessor has enough access to internal region data structure to rebuild the default scanner. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423367#comment-13423367 ] Andrew Purtell commented on HBASE-6427: --- BaseRegionObserver should reimplement the default behavior in the new methods? Anybody who inherits would get it. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423368#comment-13423368 ] Andrew Purtell commented on HBASE-6427: --- Or, better yet, BaseRegionObserver calls out to a Store static method that does it, with some javadoc to make it clear what's going on? Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6272) In-memory region state is inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423370#comment-13423370 ] Jimmy Xiang commented on HBASE-6272: Thanks for review. I will commit it to trunk tomorrow if no objection. In-memory region state is inconsistent -- Key: HBASE-6272 URL: https://issues.apache.org/jira/browse/HBASE-6272 Project: HBase Issue Type: Bug Reporter: Jimmy Xiang Assignee: Jimmy Xiang AssignmentManger stores region state related information in several places: regionsInTransition, regions (region info to server name map), and servers (server name to region info set map). However the access to these places is not coordinated properly. It leads to inconsistent in-memory region state information. Sometimes, some region could even be offline, and not in transition. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423372#comment-13423372
]
Lars Hofhansl commented on HBASE-6427:
--
Hmm... The default behavior is followed when RegionObserver.pre{flush|compact}
return a null scanner, which is what BaseRegionObserver does by default.
BaseRegionObserver implementing the default behavior would not really buy
anything (unless I am missing something).
As for last comment above, I think we'd need a preStoreScannerOpen, which would
be called in Store.getScanner (right before the new StoreScanner is created) to
allow the coprocessor to return a custom scanner here too.
Pluggable compaction policies via coprocessors
--
Key: HBASE-6427
URL: https://issues.apache.org/jira/browse/HBASE-6427
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor
Attachments: 6427-notReady.txt, 6427-v1.txt
When implementing higher level stores on top of HBase it is necessary to
allow dynamic control over how long KVs must be kept around.
Semi-static config options for ColumnFamilies (# of version or TTL) is not
sufficient.
The simplest way to achieve this is to have a pluggable class to determine
the smallestReadpoint for Region. That way outside code can control what KVs
to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423373#comment-13423373 ] Alex Baranau commented on HBASE-6411: - Here's why I'm concerned about rewriting existing MasterMetrics class: as a part of revising metrics we are likely to change existing metrics (incl. names that we expose, esp. dependent on old stuff like MetricsTimeVaryingRate). Though we now just copied in the initial patch all metrics from old master metrics, they may change. That's why in my first version I just added empty placeholder class with single example metric (cluster_requests), so that after we revise existing we can fill in more (and they are likely to be different from the old Master/RegionServer metrics). I guess this was discussed a bit in HBASE-4050. May be that is more true for RegionServer metrics though where we have a lot of such old metrics types we don't want to support anymore. But I guess the approach should be the same w.r.t. overwrite or add new metrics class. Thanx for your answers! Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[
https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423374#comment-13423374
]
Andrew Purtell commented on HBASE-6427:
---
bq. The default behavior is followed when RegionObserver.pre{flush|compact}
return a null scanner, which is what BaseRegionObserver does by default.
Fine, I was misled by the unit test code.
bq. I think we'd need a preStoreScannerOpen, which would be called in
Store.getScanner (right before the new StoreScanner is created) to allow the
coprocessor to return a custom scanner here too.
Sounds good to me.
Pluggable compaction policies via coprocessors
--
Key: HBASE-6427
URL: https://issues.apache.org/jira/browse/HBASE-6427
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
Priority: Minor
Attachments: 6427-notReady.txt, 6427-v1.txt
When implementing higher level stores on top of HBase it is necessary to
allow dynamic control over how long KVs must be kept around.
Semi-static config options for ColumnFamilies (# of version or TTL) is not
sufficient.
The simplest way to achieve this is to have a pluggable class to determine
the smallestReadpoint for Region. That way outside code can control what KVs
to retain.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423384#comment-13423384 ] Lars Hofhansl commented on HBASE-6427: -- Of course this has to be compared to simply making the ScanInfo pluggable in Store.java. What I want to achieve here is to have an external process (backup tool, transaction engine, etc) to be able to override HBase's default TTL/#Versions with very high fidelity (i.e. not via a dynamic schema change, which is too heavyweight/slow). The coprocessor approach is nice, because it provides a lot of flexibility for other future use cases and it does not invent a new concept. At the same time it adds complexity. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423387#comment-13423387 ] Andrew Purtell commented on HBASE-6427: --- bq. The coprocessor approach is nice, because it provides a lot of flexibility for other future use cases and it does not invent a new concept. At the same time it adds complexity. On balance the API change here is nice because it extends something that was too limited to address your use case such that now it works for you, and it also admits the possibility of others. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423392#comment-13423392 ] Lars Hofhansl commented on HBASE-2214: -- @Ferdy, are you sure you do not want to work on this anymore? I'll change Scan.java to be able to deal with two different version, so we can get this in 0.94 (admittedly, that is a small change only). bq. Maybe the ClientScanner could modify the Scan that it passes the max scanner result it uses to the region server Of course the requires the server to actually trust the client. Although I think with a rogue client it is quite easy to kill/mess-with a RegionServer in many other areas. Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Fix For: 0.96.0, 0.94.2 Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6372: -- Affects Version/s: 0.94.2 0.96.0 Targeting for trunk and 0.94 if you don't mind [~lhofhansl] and [~stack] Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423399#comment-13423399 ] Elliott Clark commented on HBASE-6411: -- You're 100% correct we are for sure going to add, remove, and rename metrics; these changes are potentially a big deal. However, 0.96 is the singularity release so now's the time. Lars wanted a big metrics clean up so we might as well get started on that while moving over to metrics 2. Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423406#comment-13423406 ] Elliott Clark commented on HBASE-6411: -- Thanks so much for all your work on the patch. Functionally it looks great. I'm +1 Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
Elliott Clark created HBASE-6461:
Summary: Killing the HRegionServer and DataNode hosting ROOT can
result in a malformed root table.
Key: HBASE-6461
URL: https://issues.apache.org/jira/browse/HBASE-6461
Project: HBase
Issue Type: Bug
Environment: hadoop-0.20.2-cdh3u3
HBase 0.94.1 RC1
Reporter: Elliott Clark
Spun up a new dfs on hadoop-0.20.2-cdh3u3
Started hbase
started running loadtest tool.
killed rs and dn holding root with killall -9 java on server sv4r27s44 at about
2012-07-25 22:40:00
After things stabilize Root is in a bad state. Ran hbck and got:
Exception in thread main
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in -ROOT- for region .META.,,1.1028785192 containing row
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
hbase(main):001:0 scan '-ROOT-'
ROW COLUMN+CELL
12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration
already set up for Hadoop, not re-installing.
.META.,,1column=info:regioninfo,
timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY =
'', ENCODED = 1028785192,}
.META.,,1column=info:v,
timestamp=1343255838525, value=\x00\x00
1 row(s) in 0.5930 seconds
Here's the master log: https://gist.github.com/3179194
I tried the same thing with 0.92.1 and I was able to get into a similar
situation, so I don't think this is anything new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423426#comment-13423426 ] Jonathan Leavitt commented on HBASE-6449: - @zhihong Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423431#comment-13423431 ] Zhihong Ted Yu commented on HBASE-6449: --- A completely independent project is good to have but requires more logistics. We can refine the htrace module using this JIRA so that HBase is able to use it. Then we can propose the promotion of this feature as an independent project. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423444#comment-13423444 ] Hudson commented on HBASE-6455: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #111 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/111/]) HBASE-6455 org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path (Aditya) (Revision 1366034) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path -- Key: HBASE-6455 URL: https://issues.apache.org/jira/browse/HBASE-6455 Project: HBase Issue Type: Bug Components: performance Affects Versions: 0.94.0, 0.96.0 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Labels: benchmark Fix For: 0.96.0 Attachments: HBASE-6455_trunk.patch I was the running PerformanceEvaluation test with a modified job configuration where the job output path is created before the splits and that unmasked the issue because the the PeInputFormat.getSplits() function expects to find only files under the input path. The attached patch addresses both the issues. # Creates separate input and output path rooted under a single folder e.g. user_home/performance_evaluation/MMddHHmmss/inputs and user_home/performance_evaluation/MMddHHmmss/outputs, and # The PeInputFormat.getSplits(), now skips any folder found under the input path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Attachment: 6427-v2.txt Add preStoreScannerOpen(...) to RegionObserver and related classes. Runs all of TestFromClientSide and TestCompaction with such a coprocessor. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6427: - Status: Patch Available (was: Open) Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reassigned HBASE-6427: Assignee: Lars Hofhansl Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423446#comment-13423446 ] Lars Hofhansl commented on HBASE-6372: -- 0.94 sounds good to me. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423447#comment-13423447 ] Jonathan Leavitt commented on HBASE-6449: - Sounds good. I will submit a new patch today that deals with most of your and Todd's changes as well as making htrace a module of hbase. Let's just be careful going forward to ensure that the tracing library can easily be broken out when necessary. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6334) TestImprovement for TestHRegion.testWritesWhileGetting
[
https://issues.apache.org/jira/browse/HBASE-6334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gregory Chanan updated HBASE-6334:
--
Fix Version/s: 0.96.0
Adding 0.96.0 as this was committed to trunk.
I also don't see this in the commit logs of 0.92.
TestImprovement for TestHRegion.testWritesWhileGetting
--
Key: HBASE-6334
URL: https://issues.apache.org/jira/browse/HBASE-6334
Project: HBase
Issue Type: Improvement
Affects Versions: 0.92.1, 0.94.0, 0.96.0
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
Fix For: 0.92.2, 0.96.0, 0.94.1
Attachments: HBASE-6334-92-v2.patch, HBASE-6334-94-v2.patch,
HBASE-6334-trunk-v2.patch, HBASE-6334-trunk.patch
Two improvements:
1. I was running testWritesWhileGetting on 0.90.6 (which lacks HBASE-4195)
and I couldn't get the 20% failures mentioned in the JIRA. It was more like
1%. So I increased the flush frequency and it failed most of the time.
2. The check on failure is wrong:
{code}
assertEquals(previousKV.getValue(), thisValue);
{code}
Since this is a byte array, we should use Bytes.compareTo.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6462) TestAcidGuarantees failed on trunk
Enis Soztutar created HBASE-6462:
Summary: TestAcidGuarantees failed on trunk
Key: HBASE-6462
URL: https://issues.apache.org/jira/browse/HBASE-6462
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Enis Soztutar
I've seen TestAcidGurantees fail with:
{code}
testGetAtomicity(org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey)
Time elapsed: 42.611 sec ERROR!
java.lang.RuntimeException: Deferred
at
org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76)
at
org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:103)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:298)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:248)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey.testGetAtomicity(IntegrationTestAcidGuaranteesWithChaosMonkey.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:18)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
Caused by: java.lang.RuntimeException: Failed after
147200!Expected=\x1BT\xC0i\x0CW\x9B\x108\xA0Got:
test_row_0/A:col0/1343328428704/Put/vlen=10/ts=0 val=
\x1BT\xC0i\x0CW\x9B\x108\xA0
test_row_0/A:col1/1343328428704/Put/vlen=10/ts=0 val=
\x1BT\xC0i\x0CW\x9B\x108\xA0
test_row_0/A:col10/1343328428704/Put/vlen=10/ts=0 val=
\x1BT\xC0i\x0CW\x9B\x108\xA0
...
test_row_0/B:col0/1343328425510/Put/vlen=10/ts=0 val=
4G\xE1T\x1B\xFDa\x98\xAC\xB6
test_row_0/B:col1/1343328425510/Put/vlen=10/ts=0 val=
4G\xE1T\x1B\xFDa\x98\xAC\xB6
test_row_0/B:col10/1343328425510/Put/vlen=10/ts=0 val=
...
test_row_0/C:col0/1343328425510/Put/vlen=10/ts=0 val=
4G\xE1T\x1B\xFDa\x98\xAC\xB6
test_row_0/C:col1/1343328425510/Put/vlen=10/ts=0 val=
4G\xE1T\x1B\xFDa\x98\xAC\xB6
test_row_0/C:col10/1343328425510/Put/vlen=10/ts=0 val=
{code}
Might be related to HBASE-2856, but haven't had the time to check the root
cause. The flusher thread was on.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423454#comment-13423454 ] Jean-Daniel Cryans commented on HBASE-2214: --- bq. Of course the requires the server to actually trust the client. Isn't that what this jira does already? Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Fix For: 0.96.0, 0.94.2 Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423458#comment-13423458 ] stack commented on HBASE-6449: -- bq. Do you think it is best to add it as a module as opposed to keeping it completely independent, and just having hbase depend on it? I think it best to have it as an independent project, at least at first (you don't need our blessing to do such a thing). Make a patch for hbase core that gives you the hooks you need and we'll commit that but better if all the rest can live outside especially as its being developed (no lag waiting on a committer to review and commit; just do it yourself out in github or wherever). What do you mean by having hbase dependent on htrace? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6431: -- Fix Version/s: 0.96.0 Integrated to trunk. Thanks for the patch, Alex. Thanks for the review, Stack. Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Fix For: 0.96.0 Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423463#comment-13423463 ] Andrew Purtell commented on HBASE-6456: --- +1 seems helpful at best and harmless at worst. Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Priority: Minor Attachments: HBASE-6456.patch The command I ran is as below: bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1 And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6456: -- Hadoop Flags: Reviewed Status: Patch Available (was: Open) Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Priority: Minor Attachments: HBASE-6456.patch The command I ran is as below: bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1 And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6456) Export Utility throws ClassNotFoundException
[ https://issues.apache.org/jira/browse/HBASE-6456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu reassigned HBASE-6456: - Assignee: Shengsheng Huang Export Utility throws ClassNotFoundException Key: HBASE-6456 URL: https://issues.apache.org/jira/browse/HBASE-6456 Project: HBase Issue Type: Bug Reporter: Shengsheng Huang Assignee: Shengsheng Huang Priority: Minor Attachments: HBASE-6456.patch The command I ran is as below: bin/hbase org.apache.hadoop.hbase.mapreduce.Driver export t1 ./t1 And I got the following exceptions: attempt_201207261322_0002_m_00_0, Status : FAILED Error: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.util.Bytes at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:306) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:247) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:150) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:100) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:371) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.au ... This exception can be resolved by adding hbase common jar into HADOOP_CLASSPATH. But this is an extra step for users and not so convenient. We could add Bytes.class into dependency Jars of the MapReduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423469#comment-13423469 ] Andrew Purtell commented on HBASE-6427: --- lgtm, good tests Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Hofhansl updated HBASE-6461:
-
Priority: Critical (was: Major)
Fix Version/s: 0.94.2
Killing the HRegionServer and DataNode hosting ROOT can result in a malformed
root table.
-
Key: HBASE-6461
URL: https://issues.apache.org/jira/browse/HBASE-6461
Project: HBase
Issue Type: Bug
Environment: hadoop-0.20.2-cdh3u3
HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
Fix For: 0.94.2
Spun up a new dfs on hadoop-0.20.2-cdh3u3
Started hbase
started running loadtest tool.
killed rs and dn holding root with killall -9 java on server sv4r27s44 at
about 2012-07-25 22:40:00
After things stabilize Root is in a bad state. Ran hbck and got:
Exception in thread main
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in -ROOT- for region .META.,,1.1028785192 containing row
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
hbase(main):001:0 scan '-ROOT-'
ROW COLUMN+CELL
12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration
already set up for Hadoop, not re-installing.
.META.,,1column=info:regioninfo,
timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY
= '', ENCODED = 1028785192,}
.META.,,1column=info:v,
timestamp=1343255838525, value=\x00\x00
1 row(s) in 0.5930 seconds
Here's the master log: https://gist.github.com/3179194
I tried the same thing with 0.92.1 and I was able to get into a similar
situation, so I don't think this is anything new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423474#comment-13423474
]
Lars Hofhansl commented on HBASE-6461:
--
Marked Critical against 0.94.2. Could also see this as a blocker for 0.94.1
(Although 0.94.1 needs to go out at some point, because of HBASE-6311)
Killing the HRegionServer and DataNode hosting ROOT can result in a malformed
root table.
-
Key: HBASE-6461
URL: https://issues.apache.org/jira/browse/HBASE-6461
Project: HBase
Issue Type: Bug
Environment: hadoop-0.20.2-cdh3u3
HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
Fix For: 0.94.2
Spun up a new dfs on hadoop-0.20.2-cdh3u3
Started hbase
started running loadtest tool.
killed rs and dn holding root with killall -9 java on server sv4r27s44 at
about 2012-07-25 22:40:00
After things stabilize Root is in a bad state. Ran hbck and got:
Exception in thread main
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in -ROOT- for region .META.,,1.1028785192 containing row
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
hbase(main):001:0 scan '-ROOT-'
ROW COLUMN+CELL
12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration
already set up for Hadoop, not re-installing.
.META.,,1column=info:regioninfo,
timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY
= '', ENCODED = 1028785192,}
.META.,,1column=info:v,
timestamp=1343255838525, value=\x00\x00
1 row(s) in 0.5930 seconds
Here's the master log: https://gist.github.com/3179194
I tried the same thing with 0.92.1 and I was able to get into a similar
situation, so I don't think this is anything new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6457) when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss
[ https://issues.apache.org/jira/browse/HBASE-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423476#comment-13423476 ] Zhihong Ted Yu commented on HBASE-6457: --- That would be nice, Andy. when use bulkload, hundreds of reduce write hfile, maybe create files of the same name, as a result some data loss --- Key: HBASE-6457 URL: https://issues.apache.org/jira/browse/HBASE-6457 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.92.1 Reporter: wanbin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6463) Support multiple memstore snapshots in order to support small/large flushes of cache.
Brian Nixon created HBASE-6463: -- Summary: Support multiple memstore snapshots in order to support small/large flushes of cache. Key: HBASE-6463 URL: https://issues.apache.org/jira/browse/HBASE-6463 Project: HBase Issue Type: Improvement Components: regionserver, util Affects Versions: 0.89-fb Reporter: Brian Nixon If cache is underutilized due to log size triggered flushes, should be able to buffer multiple snapshots in memory and flush all together into one file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423482#comment-13423482 ] Jonathan Leavitt commented on HBASE-6449: - bq. What do you mean by having hbase dependent on htrace? I meant having htrace as an independent maven project in the hbase source tree and add a dependency for htrace in hbase's pom. The original plan was to get the tracing library committed to hbase first so the hbase committers could review and would be more likely to commit the necessary hooks into hbase. It would also save the initial hassle of getting the library into maven. But you would recommend putting it on github, and only submitting patches with the instrumentation for hbase? If htrace were on github and the hooks were in hbase, would those downloading hbase source just have to grab htrace off github and have maven install it locally? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6372) Add scanner batching to Export job
[ https://issues.apache.org/jira/browse/HBASE-6372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423487#comment-13423487 ] Hadoop QA commented on HBASE-6372: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12537954/HBASE-6372.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2441//console This message is automatically generated. Add scanner batching to Export job -- Key: HBASE-6372 URL: https://issues.apache.org/jira/browse/HBASE-6372 Project: HBase Issue Type: Improvement Components: mapreduce Affects Versions: 0.96.0, 0.94.2 Reporter: Lars George Priority: Minor Labels: newbie Attachments: HBASE-6372.patch When a single row is too large for the RS heap then an OOME can take out the entire RS. Setting scanner batching in custom scans helps avoiding this scenario, but for the supplied Export job this is not set. Similar to HBASE-3421 we can set the batching to a low number - or if needed make it a command line option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423492#comment-13423492 ] Lars Hofhansl commented on HBASE-2214: -- Yep... Was just making a general observation. Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Fix For: 0.96.0, 0.94.2 Attachments: HBASE-2214-0.94-v2.txt, HBASE-2214-0.94-v3.txt, HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, HBASE-2214-v7.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6462) TestAcidGuarantees failed on trunk
[
https://issues.apache.org/jira/browse/HBASE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423489#comment-13423489
]
Enis Soztutar commented on HBASE-6462:
--
Another stack trace I got now has the 3rd CF with different values:
{code}
Caused by: java.lang.RuntimeException: Failed after
98200!Expected=\xFF[\x0B_\xAF\xCAQJ\xBDKGot:
test_row_2/A:col0/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK
test_row_2/A:col1/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK
..
test_row_2/B:col8/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK
test_row_2/B:col9/1343337923715/Put/vlen=10/ts=0 val= \xFF[\x0B_\xAF\xCAQJ\xBDK
..
test_row_2/C:col0/1343337921472/Put/vlen=10/ts=0 val=
\xEA\xD0\x15\xFB\xC0\xE7\xE3\xA0\xDB^
test_row_2/C:col1/1343337921472/Put/vlen=10/ts=0 val=
\xEA\xD0\x15\xFB\xC0\xE7\xE3\xA0\xDB^
..
{code}
TestAcidGuarantees failed on trunk
--
Key: HBASE-6462
URL: https://issues.apache.org/jira/browse/HBASE-6462
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0
Reporter: Enis Soztutar
I've seen TestAcidGurantees fail with:
{code}
testGetAtomicity(org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey)
Time elapsed: 42.611 sec ERROR!
java.lang.RuntimeException: Deferred
at
org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.checkException(MultithreadedTestUtil.java:76)
at
org.apache.hadoop.hbase.MultithreadedTestUtil$TestContext.stop(MultithreadedTestUtil.java:103)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:298)
at
org.apache.hadoop.hbase.TestAcidGuarantees.runTestAtomicity(TestAcidGuarantees.java:248)
at
org.apache.hadoop.hbase.IntegrationTestAcidGuaranteesWithChaosMonkey.testGetAtomicity(IntegrationTestAcidGuaranteesWithChaosMonkey.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:18)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:234)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:133)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:188)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:166)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:86)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:101)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
Caused by: java.lang.RuntimeException: Failed after
147200!Expected=\x1BT\xC0i\x0CW\x9B\x108\xA0Got:
test_row_0/A:col0/1343328428704/Put/vlen=10/ts=0 val=
\x1BT\xC0i\x0CW\x9B\x108\xA0
test_row_0/A:col1/1343328428704/Put/vlen=10/ts=0 val=
\x1BT\xC0i\x0CW\x9B\x108\xA0
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423501#comment-13423501 ] Hadoop QA commented on HBASE-6427: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538075/6427-v2.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 14 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient org.apache.hadoop.hbase.client.TestFromClientSide org.apache.hadoop.hbase.client.TestAdmin org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2440//console This message is automatically generated. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423506#comment-13423506 ] Jonathan Leavitt commented on HBASE-6449: - I updated the diff on [review board |https://reviews.apache.org/r/6127/] with many of Todd and Ted's suggestions. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-2945) stop using string for block name in LRU block cache/hfile
[ https://issues.apache.org/jira/browse/HBASE-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-2945. -- Resolution: Duplicate This is a duplication of HBASE-5001 stop using string for block name in LRU block cache/hfile - Key: HBASE-2945 URL: https://issues.apache.org/jira/browse/HBASE-2945 Project: HBase Issue Type: Bug Components: performance Affects Versions: 0.89.20100621 Reporter: ryan rawson all my profiling runs indicate there is a large number of string/char[] objects and string manipulation is taking a long time in profiling runs. These come from the LRU block cache, block id. Also we should support the eviction of blocks belonging to particular hfiles, and the current code doesn't help with that right now. Let's do something better. Whatever that might be. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423508#comment-13423508 ] stack commented on HBASE-6449: -- bq. I meant having htrace as an independent maven project in the hbase source tree and add a dependency for htrace in hbase's pom. But sounds like your intent is a generic tracing library going by statements above? Is that so? If that is the case, it seems odd dev'ing a generic lib inside in HBase core. You should do it as its own project I'd say; you can move faster and it'll be easier keep your generic focus. HBase project, IMO, will gladly be your guinea pig/proofing ground for your tracing lib work since we need tracing so bad. We can review/exercise your lib, we can add it as a dependency and make the necessary changes to pull it in and make it easy starting/stopping a trace, etc. but unless you think otherwise, lets keep the coupling loose. Correct me if I'm off Jonathan. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6431) Some FilterList Constructors break addFilter
[ https://issues.apache.org/jira/browse/HBASE-6431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423509#comment-13423509 ] Hudson commented on HBASE-6431: --- Integrated in HBase-TRUNK #3173 (See [https://builds.apache.org/job/HBase-TRUNK/3173/]) HBASE-6431 Some FilterList Constructors break addFilter (Alex Newman) (Revision 1366198) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/filter/FilterList.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilterList.java Some FilterList Constructors break addFilter Key: HBASE-6431 URL: https://issues.apache.org/jira/browse/HBASE-6431 Project: HBase Issue Type: Bug Components: filters Affects Versions: 0.92.1, 0.94.0 Reporter: Alex Newman Assignee: Alex Newman Priority: Minor Fix For: 0.96.0 Attachments: 0001-HBASE-6431.-Some-FilterList-Constructors-break-addFi.patch, 6431-v2.txt Some of the constructors for FilterList set the internal list of filters to list types which don't support the add operation. As a result FilterList(final ListFilter rowFilters) FilterList(final Filter... rowFilters) FilterList(final Operator operator, final ListFilter rowFilters) FilterList(final Operator operator, final Filter... rowFilters) may init private ListFilter filters = new ArrayListFilter(); incorrectly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6455) org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path
[ https://issues.apache.org/jira/browse/HBASE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423510#comment-13423510 ] Hudson commented on HBASE-6455: --- Integrated in HBase-TRUNK #3173 (See [https://builds.apache.org/job/HBase-TRUNK/3173/]) HBASE-6455 org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path (Aditya) (Revision 1366034) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java org.apache.hadoop.hbase.PerformanceEvaluation sets the map reduce output path as a child of input path -- Key: HBASE-6455 URL: https://issues.apache.org/jira/browse/HBASE-6455 Project: HBase Issue Type: Bug Components: performance Affects Versions: 0.94.0, 0.96.0 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor Labels: benchmark Fix For: 0.96.0 Attachments: HBASE-6455_trunk.patch I was the running PerformanceEvaluation test with a modified job configuration where the job output path is created before the splits and that unmasked the issue because the the PeInputFormat.getSplits() function expects to find only files under the input path. The attached patch addresses both the issues. # Creates separate input and output path rooted under a single folder e.g. user_home/performance_evaluation/MMddHHmmss/inputs and user_home/performance_evaluation/MMddHHmmss/outputs, and # The PeInputFormat.getSplits(), now skips any folder found under the input path and process only files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423512#comment-13423512 ] Hadoop QA commented on HBASE-6411: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538040/HBASE-6411-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 45 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause mvn compile goal to fail. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2442//testReport/ Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2442//console This message is automatically generated. Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423520#comment-13423520 ] stack commented on HBASE-6449: -- One more thing, I'd say HBase wants the generic lib. An hbase-specific would be great but won't be as effective as one that does NOT stop dead in its tracks at the HDFS API. A trace that goes all the ways down would be the cats pyjamas. Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423522#comment-13423522
]
Lars Hofhansl commented on HBASE-6461:
--
Any chance to repeat this with distributed log splitting disabled?
Killing the HRegionServer and DataNode hosting ROOT can result in a malformed
root table.
-
Key: HBASE-6461
URL: https://issues.apache.org/jira/browse/HBASE-6461
Project: HBase
Issue Type: Bug
Environment: hadoop-0.20.2-cdh3u3
HBase 0.94.1 RC1
Reporter: Elliott Clark
Priority: Critical
Fix For: 0.94.2
Spun up a new dfs on hadoop-0.20.2-cdh3u3
Started hbase
started running loadtest tool.
killed rs and dn holding root with killall -9 java on server sv4r27s44 at
about 2012-07-25 22:40:00
After things stabilize Root is in a bad state. Ran hbck and got:
Exception in thread main
org.apache.hadoop.hbase.client.NoServerForRegionException: No server address
listed in -ROOT- for region .META.,,1.1028785192 containing row
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:1016)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:841)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:810)
at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:232)
at org.apache.hadoop.hbase.client.HTable.init(HTable.java:172)
at org.apache.hadoop.hbase.util.HBaseFsck.connect(HBaseFsck.java:241)
at org.apache.hadoop.hbase.util.HBaseFsck.main(HBaseFsck.java:3236)
hbase(main):001:0 scan '-ROOT-'
ROW COLUMN+CELL
12/07/25 22:43:18 INFO security.UserGroupInformation: JAAS Configuration
already set up for Hadoop, not re-installing.
.META.,,1column=info:regioninfo,
timestamp=1343255838525, value={NAME = '.META.,,1', STARTKEY = '', ENDKEY
= '', ENCODED = 1028785192,}
.META.,,1column=info:v,
timestamp=1343255838525, value=\x00\x00
1 row(s) in 0.5930 seconds
Here's the master log: https://gist.github.com/3179194
I tried the same thing with 0.92.1 and I was able to get into a similar
situation, so I don't think this is anything new.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6449) Dapper like tracing
[ https://issues.apache.org/jira/browse/HBASE-6449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423524#comment-13423524 ] Jonathan Leavitt commented on HBASE-6449: - @stack you nailed it. The original goal was just tracing for hbase, but it seemed silly to not put in the extra effort necessary to keep the tracing code generic enough for use elsewhere. So it seems the consensus is that I should get the tracing code up on github and develop it there? Then I would submit patches that just integrate the library into htrace? Dapper like tracing --- Key: HBASE-6449 URL: https://issues.apache.org/jira/browse/HBASE-6449 Project: HBase Issue Type: New Feature Components: client, ipc Affects Versions: 0.96.0 Reporter: Jonathan Leavitt Labels: tracing Attachments: htrace1.diff, trace.png Add [Dapper|http://research.google.com/pubs/pub36356.html] like tracing to HBase. [Accumulo|http://accumulo.apache.org] added something similar with their cloudtrace package. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6461) Killing the HRegionServer and DataNode hosting ROOT can result in a malformed root table.
[
https://issues.apache.org/jira/browse/HBASE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423535#comment-13423535
]
Elliott Clark commented on HBASE-6461:
--
bq.Any chance to repeat this with distributed log splitting disabled?
Sure gimme a little bit. Reading the logs on my latest example. Once JD and I
are done (assuming that we don't solve it) I'll try with distributed log
splitting off.
On the latest run JD noticed that this in the logs of the server that was doing
the splitting (There was only one hlog nothing had rolled yet). 10.4.3.44 is
the server that had all java processes killed.
{noformat}
453-2012-07-26 22:13:18,079 WARN
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: File
hdfs://sv4r11s38:9901/hbase/.logs/sv4r3s44,9913,1343340207863-splitting/sv4r3s44%2C9913%2C1343340207863.1343340210132
might be still open, length is 0
454-2012-07-26 22:13:18,081 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
Recovering file
hdfs://sv4r11s38:9901/hbase/.logs/sv4r3s44,9913,1343340207863-splitting/sv4r3s44%2C9913%2C1343340207863.1343340210132
455-2012-07-26 22:13:19,083 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
Finished lease recover attempt for
hdfs://sv4r11s38:9901/hbase/.logs/sv4r3s44,9913,1343340207863-splitting/sv4r3s44%2C9913%2C1343340207863.1343340210132
456-2012-07-26 22:13:20,091 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 0 time(s).
457-2012-07-26 22:13:21,092 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 1 time(s).
458-2012-07-26 22:13:22,093 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 2 time(s).
459-2012-07-26 22:13:23,093 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 3 time(s).
460-2012-07-26 22:13:24,094 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 4 time(s).
461-2012-07-26 22:13:25,095 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 5 time(s).
462-2012-07-26 22:13:26,096 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 6 time(s).
463-2012-07-26 22:13:27,096 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 7 time(s).
464-2012-07-26 22:13:27,936 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=50.28 MB,
free=5.94 GB, max=5.99 GB, blocks=0, accesses=0, hits=0,
hitRatio=0cachingAccesses=0, cachingHits=0, cachingHitsRatio=0evictions=0,
evicted=0, evictedPerRun=NaN
465-2012-07-26 22:13:28,097 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 8 time(s).
466-2012-07-26 22:13:29,098 INFO org.apache.hadoop.ipc.Client: Retrying connect
to server: /10.4.3.44:9910. Already tried 9 time(s).
467:2012-07-26 22:13:29,100 WARN
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Could not open
hdfs://sv4r11s38:9901/hbase/.logs/sv4r3s44,9913,1343340207863-splitting/sv4r3s44%2C9913%2C1343340207863.1343340210132
for reading. File is empty
468-java.io.EOFException
469-at java.io.DataInputStream.readFully(DataInputStream.java:180)
470-at java.io.DataInputStream.readFully(DataInputStream.java:152)
471-at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1521)
472-at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1493)
473-at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1480)
474-at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1475)
475-at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.init(SequenceFileLogReader.java:55)
476-at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:175)
477-at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:716)
478-at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:821)
479-at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:734)
480-at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
481-at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:348)
482-at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111)
483-at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
484-at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195)
485-at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:163)
486-at java.lang.Thread.run(Thread.java:662)
{noformat}
So the log splitter is trying the dead server over and over. Then failing
[jira] [Commented] (HBASE-6427) Pluggable compaction policies via coprocessors
[ https://issues.apache.org/jira/browse/HBASE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423548#comment-13423548 ] Lars Hofhansl commented on HBASE-6427: -- I ran the failing tests locally, and they all pass. Will sit on this a bit longer, write a test that tests the actual scenario I am interested in, etc. Pluggable compaction policies via coprocessors -- Key: HBASE-6427 URL: https://issues.apache.org/jira/browse/HBASE-6427 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Lars Hofhansl Priority: Minor Attachments: 6427-notReady.txt, 6427-v1.txt, 6427-v2.txt When implementing higher level stores on top of HBase it is necessary to allow dynamic control over how long KVs must be kept around. Semi-static config options for ColumnFamilies (# of version or TTL) is not sufficient. The simplest way to achieve this is to have a pluggable class to determine the smallestReadpoint for Region. That way outside code can control what KVs to retain. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6411) Move Master Metrics to metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423567#comment-13423567 ] Zhihong Ted Yu commented on HBASE-6411: --- Put up some comments on review board. Move Master Metrics to metrics 2 Key: HBASE-6411 URL: https://issues.apache.org/jira/browse/HBASE-6411 Project: HBase Issue Type: Sub-task Reporter: Elliott Clark Assignee: Elliott Clark Attachments: HBASE-6411-0.patch, HBASE-6411-1.patch, HBASE-6411-2.patch, HBASE-6411-3.patch, HBASE-6411_concept.patch Move Master Metrics to metrics 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
