[jira] [Updated] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Mohamed Zahoor updated HBASE-6564: Attachment: HBASE-6564-trunk.patch1 Incorporated review comments. Thanks Ted and Stack. ./Zahoor HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-trunk.patch1 When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Mohamed Zahoor updated HBASE-6564: Attachment: (was: HBASE-6564-trunk.patch1) HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Mohamed Zahoor updated HBASE-6564: Attachment: HBASE-6564-trunk-ver2.patch Changed the Patch file name to stick to convention. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-trunk-ver2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Archimedes Trajano updated HBASE-4709:
--
Affects Version/s: 0.94.1
Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
Key: HBASE-4709
URL: https://issues.apache.org/jira/browse/HBASE-4709
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0, 0.94.0, 0.94.1
Reporter: Gary Helmling
Priority: Minor
Since switching over HBase to build with Hadoop 0.20.205.0, we've been
getting a lot of metrics related errors in the log files for tests:
{noformat}
2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
SelectChannelConnector@localhost:55751
Starting DataNode 1 with dfs.data.dir:
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
system not started: Cannot locate configuration: tried
hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered
with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at
com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at
com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at
com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
{noformat}
This seems to be due to errors initializing the new hadoop metrics2 code by
default, when running in a mini cluster. The errors themselves seem to be
harmless -- they're not breaking any tests -- but we should figure out what
configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437465#comment-13437465 ] Hadoop QA commented on HBASE-6564: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541515/HBASE-6564-trunk-ver2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2618//console This message is automatically generated. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-trunk-ver2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5449) Support for wire-compatible security functionality
[ https://issues.apache.org/jira/browse/HBASE-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5449: --- Attachment: HBASE-5449-v1.patch Added v1 patch using Gary's protobuf, that removes the TablePermission (merged with Permission). Thanks Gary! Support for wire-compatible security functionality -- Key: HBASE-5449 URL: https://issues.apache.org/jira/browse/HBASE-5449 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Matteo Bertozzi Attachments: AccessControl_protos.patch, HBASE-5449-v0.patch, HBASE-5449-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6165) Replication can overrun .META. scans on cluster re-start
[ https://issues.apache.org/jira/browse/HBASE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Himanshu Vashishtha updated HBASE-6165: --- Attachment: HBase-6165-v4.patch You are right Sir! Done. Replication can overrun .META. scans on cluster re-start Key: HBASE-6165 URL: https://issues.apache.org/jira/browse/HBASE-6165 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.2 Attachments: HBase-6165-v1.patch, HBase-6165-v2.patch, HBase-6165-v3.patch, HBase-6165-v4.patch When restarting a large set of regions on a reasonably small cluster the replication from another cluster tied up every xceiver meaning nothing could be onlined. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437469#comment-13437469 ] Hadoop QA commented on HBASE-6564: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541485/HBASE-6564-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2617//console This message is automatically generated. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-trunk-ver2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6567) make memory locking configuration of regioservers more flexible
[
https://issues.apache.org/jira/browse/HBASE-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437470#comment-13437470
]
Matteo Bertozzi commented on HBASE-6567:
@stack the patch contains also the hbase-env.sh with the two new variables a
comment that try to explain what the mlock is used for, but doesn't seems to be
in trunk. Do you think is not useful to expose that? (I've no strong opinion on
expose it or not, since most of the user can live without knowing it)
{code}
+# Uncomment and adjust to keep all the Region Server pages mapped to be memory
resident
+#HBASE_REGIONSERVER_MLOCK=true
+#HBASE_REGIONSERVER_UID=hbase
{code}
make memory locking configuration of regioservers more flexible
---
Key: HBASE-6567
URL: https://issues.apache.org/jira/browse/HBASE-6567
Project: HBase
Issue Type: Improvement
Components: scripts
Affects Versions: 0.96.0
Reporter: Roman Shaposhnik
Assignee: Matteo Bertozzi
Fix For: 0.96.0
Attachments: HBASE-6567-v0.patch
The current implementation of the memory locking feature of regisoservers has
a downside of not being flexible to configure for permanent use. Sure there
is a --mlock flag but that needs to be explicitly passed on every invocation
and thus require extra steps to be configured for permanent use (IOW, there's
not a single env variable I can set to have a desired effect). The only other
alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a
downside of being pretty cryptic to the novice user and has a killer problem
of not explicitly telling higher level scripts (like init.d or upstart ones)
which user the initial hbase process should be executed as.
I propose a very simple solution (which is essentially making --mlock setting
into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that
can be set in hbase-env.sh and has the following semantics:
* [default] not set: mlocking feature is disabled
* set but empty: mlocking feature is enabled and the target user is hbase
* set and not empty: mlocking feature is enabled and the target user is
the value of the variable
Thoughts?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Mohamed Zahoor updated HBASE-6564: Attachment: (was: HBASE-6564-trunk-ver2.patch) HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] J Mohamed Zahoor updated HBASE-6564: Attachment: HBASE-6564-v2.patch HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437477#comment-13437477 ] Hadoop QA commented on HBASE-6564: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541520/HBASE-6564-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 11 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2620//console This message is automatically generated. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6165) Replication can overrun .META. scans on cluster re-start
[ https://issues.apache.org/jira/browse/HBASE-6165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437478#comment-13437478 ] Hadoop QA commented on HBASE-6165: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541517/HBase-6165-v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2619//console This message is automatically generated. Replication can overrun .META. scans on cluster re-start Key: HBASE-6165 URL: https://issues.apache.org/jira/browse/HBASE-6165 Project: HBase Issue Type: Bug Reporter: Elliott Clark Assignee: Himanshu Vashishtha Fix For: 0.96.0, 0.94.2 Attachments: HBase-6165-v1.patch, HBase-6165-v2.patch, HBase-6165-v3.patch, HBase-6165-v4.patch When restarting a large set of regions on a reasonably small cluster the replication from another cluster tied up every xceiver meaning nothing could be onlined. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437479#comment-13437479 ] J Mohamed Zahoor commented on HBASE-6564: - Strange.. this patch applies to trunk in my setup and the TestReplication test also passes. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal updated HBASE-6435: --- Release Note: This JIRA adds a hook in the HDFS client to reorder the replica locations for HLog files. The default ordering in HDFS is rack aware + random. When reading a HLog file, we prefer not to use the replica on the same server as the region server that wrote the HLog: this server is likely to be not available, and this will delay the HBase recovery by one minute. This occurs because the recovery starts sooner in HBase than in HDFS: 3 minutes by default in HBase vs. 10:30 minutes in HDFS. This will be changed in HDFS-3703. Moreover, when a HDFS file is already opened for writing, a read triggers another call to get the file size, leading to another timeout (see HDFS-3704), but as well a wrong file size value (see HDFS-3701 and HBASE-6401). Technically: - his hook won't be useful anymore when HDFS-3702 or HDFS-3705 or HDFS-3706 is available and used in HBase. - the hook intercepts the calls to the nanemode and reorder the locations it returned, extracting the region server name from the HLog file. This server is put at the end of the list, ensuring it will be tried only if all the others fail. - It has been tested with HDFS 1.0.3. of HDFS 2.0 apha. - It can be deactivated (at master region server start-up) by setting hbase.filesystem.reorder.blocks to false in the HBase configuration. Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 6435.unfinished.patch, 6435.v10.patch, 6435.v10.patch, 6435.v12.patch, 6435.v12.patch, 6435.v12.patch, 6435-v12.txt, 6435.v13.patch, 6435.v2.patch, 6435.v7.patch, 6435.v8.patch, 6435.v9.patch, 6435.v9.patch, 6535.v11.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437485#comment-13437485 ] nkeywal commented on HBASE-6435: release notes done. Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 6435.unfinished.patch, 6435.v10.patch, 6435.v10.patch, 6435.v12.patch, 6435.v12.patch, 6435.v12.patch, 6435-v12.txt, 6435.v13.patch, 6435.v2.patch, 6435.v7.patch, 6435.v8.patch, 6435.v9.patch, 6435.v9.patch, 6535.v11.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6608) Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's
[
https://issues.apache.org/jira/browse/HBASE-6608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437498#comment-13437498
]
Hudson commented on HBASE-6608:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #136 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/136/])
HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted
before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java
Fix for HBASE-6160, META entries from daughters can be deleted before parent
entries, shouldn't compare HRegionInfo's
-
Key: HBASE-6608
URL: https://issues.apache.org/jira/browse/HBASE-6608
Project: HBase
Issue Type: Bug
Components: client, regionserver
Affects Versions: 0.92.1, 0.96.0, 0.94.2
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 0.92.2, 0.96.0, 0.94.2
Attachments: 6608-v2.patch, hbase-6608_v1.patch
Our nightlies discovered that the patch for HBASE-6160 did not actually fix
the issue of META entries from daughters can be deleted before parent
entries. Instead of reopening the HBASE-6160, it is cleaner to track it
here.
The original issue is:
{quote}
HBASE-5986 fixed and issue, where the client sees the META entry for the
parent, but not the children. However, after the fix, we have seen the
following issue in tests:
Region A is split to - B, C
Region B is split to - D, E
After some time, META entry for B is deleted since it is not needed anymore,
but META entry for Region A stays in META (C still refers it). In this case,
the client throws RegionOfflineException for B.
{quote}
The problem with the fix seems to be that we keep and compare HRegionInfo's
in the HashSet at CatalogJanitor.java#scan(), but HRI that are compared are
not equal.
{code}
HashSetHRegionInfo parentNotCleaned = new HashSetHRegionInfo(); //regions
whose parents are still around
for (Map.EntryHRegionInfo, Result e : splitParents.entrySet()) {
if (!parentNotCleaned.contains(e.getKey()) cleanParent(e.getKey(),
e.getValue())) {
cleaned++;
} else {
...
{code}
In the above case, Meta row for region A will contain a serialized version of
B that is not offline. However Meta row for region B will contain a
serialized version of B that is offline (MetaEditor.offlineParentInMeta()
does that). So the deserialized version we put to HashSet and the
deserialized version we query contains() from HashSet are different in the
offline field, thus HRI.equals() fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6160) META entries from daughters can be deleted before parent entries
[ https://issues.apache.org/jira/browse/HBASE-6160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437499#comment-13437499 ] Hudson commented on HBASE-6160: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #136 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/136/]) HBASE-6608 Fix for HBASE-6160, META entries from daughters can be deleted before parent entries, shouldn't compare HRegionInfo's (Enis) (Revision 1374676) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java META entries from daughters can be deleted before parent entries Key: HBASE-6160 URL: https://issues.apache.org/jira/browse/HBASE-6160 Project: HBase Issue Type: Bug Components: client, regionserver Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.92.2, 0.94.1 Attachments: HBASE-6160_v1.patch, HBASE-6160v2092.txt, HBASE-6160_v2.patch, HBASE-6160_v2.patch HBASE-5986 fixed and issue, where the client sees the META entry for the parent, but not the children. However, after the fix, we have seen the following issue in tests: Region A is split to - B, C Region B is split to - D, E After some time, META entry for B is deleted since it is not needed anymore, but META entry for Region A stays in META (C still refers it). In this case, the client throws RegionOfflineException for B. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6609) Startcluster fails on windows (references ls)
Archimedes Trajano created HBASE-6609:
-
Summary: Startcluster fails on windows (references ls)
Key: HBASE-6609
URL: https://issues.apache.org/jira/browse/HBASE-6609
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.94.1
Environment: Windows 7 64-bit
Eclipse
Reporter: Archimedes Trajano
When starting up the cluster using
@BeforeClass
public static void setUp() throws Exception {
utility = new HBaseTestingUtility();
utility.startMiniCluster();
}
I get the following stack trace under Windows in Eclipse. Note it is looking
for ls which is not present under Windows.
java.lang.RuntimeException: Error while running command to get file permissions
: java.io.IOException: Cannot run program ls: CreateProcess error=2, The
system cannot find the file specified
at java.lang.ProcessBuilder.start(Unknown Source)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:200)
at org.apache.hadoop.util.Shell.run(Shell.java:182)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:461)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:444)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:710)
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:443)
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:418)
at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1496)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:430)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:598)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:554)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:523)
at
net.trajano.nosql.hbase.test.FrameworkTest.setUp(FrameworkTest.java:17)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:27)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
at
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find
the file specified
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.init(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 37 more
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:468)
at
org.apache.hadoop.fs.RawLocalFileSystem$RawLocalFileStatus.getPermission(RawLocalFileSystem.java:418)
at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
at
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[
https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437515#comment-13437515
]
Zhihong Ted Yu commented on HBASE-6564:
---
@J:
If you look at patch v2 closely, you can see that it is composed of two patch
files (line 310).
I suggest renaming TestTableFamilyHandlers.java -
TestTableDeleteFamilyHandler.java because only TableDeleteFamilyHandler is
tested.
There're a lot of two (successive) empty lines in this new file.
Please remove redundant empty line.
{code}
* Copyright 2009 The Apache Software Foundation
{code}
The above line should be removed from license header.
{code}
// 4 - Check if all the 3 column families exists in FS
{code}
'exists' - 'exist'
{code}
public org.apache.hadoop.hbase.ResourceCheckerJUnitRule cu = new
org.apache.hadoop.hbase.ResourceCheckerJUnitRule();
{code}
The above line is longer than 100 characters. Please wrap.
Please regenerate patch and attach to this JIRA.
Thanks
HDFS space is not reclaimed when a column family is deleted
---
Key: HBASE-6564
URL: https://issues.apache.org/jira/browse/HBASE-6564
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.94.1
Reporter: J Mohamed Zahoor
Priority: Minor
Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch
When a column family of a table is deleted, the HDFS space of the column
family does not seem to be reclaimed even after a major compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5022) Optimize HBaseConfiguration#create
[ https://issues.apache.org/jira/browse/HBASE-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437516#comment-13437516 ] nkeywal commented on HBASE-5022: @jd I'm ok, that why I proposed to revert the change and remove the wrong comment line. Optimize HBaseConfiguration#create -- Key: HBASE-5022 URL: https://issues.apache.org/jira/browse/HBASE-5022 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.0 Reporter: nkeywal Assignee: nkeywal Priority: Minor Attachments: 5022.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437517#comment-13437517 ] Zhihong Ted Yu commented on HBASE-6564: --- For MasterFileSystem.deleteFamily(), if fs.delete() returns false, I think we should raise IOException. HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6564) HDFS space is not reclaimed when a column family is deleted
[ https://issues.apache.org/jira/browse/HBASE-6564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu reassigned HBASE-6564: - Assignee: J Mohamed Zahoor HDFS space is not reclaimed when a column family is deleted --- Key: HBASE-6564 URL: https://issues.apache.org/jira/browse/HBASE-6564 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.1 Reporter: J Mohamed Zahoor Assignee: J Mohamed Zahoor Priority: Minor Attachments: HBASE-6564-trunk.patch, HBASE-6564-v2.patch When a column family of a table is deleted, the HDFS space of the column family does not seem to be reclaimed even after a major compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
Matteo Bertozzi created HBASE-6610: -- Summary: HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: New Feature Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 0.96.0 To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6610: --- Status: Patch Available (was: Open) HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: New Feature Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 Attachments: HBASE-6610-v0.patch To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6611) Forcing region state offline cause double assignment
Jimmy Xiang created HBASE-6611: -- Summary: Forcing region state offline cause double assignment Key: HBASE-6611 URL: https://issues.apache.org/jira/browse/HBASE-6611 Project: HBase Issue Type: Bug Components: master Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 In assigning a region, assignment manager forces the region state offline if it is not. This could cause double assignment, for example, if the region is already assigned and in the Open state, you should not just change it's state to Offline, and assign it again. I think this could be the root cause for all double assignments IF the region state is reliable. After this loophole is closed, TestHBaseFsck should come up a different way to create some assignment inconsistencies, for example, calling region server to open a region directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6610: --- Issue Type: Sub-task (was: New Feature) Parent: HBASE-6230 HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: Sub-task Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 Attachments: HBASE-6610-v0.patch To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6610: --- Attachment: HBASE-6610-v0.patch HFileLink draft available on review board https://reviews.apache.org/r/6694/ HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: New Feature Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 Attachments: HBASE-6610-v0.patch To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6610: --- Status: Open (was: Patch Available) HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: Sub-task Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 Attachments: HBASE-6610-v0.patch To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6610: --- Attachment: (was: HBASE-6610-v0.patch) HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: Sub-task Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6610) HFileLink: Hardlink alternative for snapshot restore
[ https://issues.apache.org/jira/browse/HBASE-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437535#comment-13437535 ] Hadoop QA commented on HBASE-6610: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541526/HBASE-6610-v0.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 8 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestSplitLogManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2621//console This message is automatically generated. HFileLink: Hardlink alternative for snapshot restore Key: HBASE-6610 URL: https://issues.apache.org/jira/browse/HBASE-6610 Project: HBase Issue Type: Sub-task Components: io Affects Versions: 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Labels: snapshot Fix For: 0.96.0 To avoid copying data during restore snapshot we need to introduce an HFile Link that allows to reference a file that can be in the original path (/hbase/table/region/cf/hfile) or, if the file is archived, in the archive directory (/hbase/.archive/table/region/cf/hfile). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6612) Hbase command line improvements
Ionut Ignatescu created HBASE-6612: -- Summary: Hbase command line improvements Key: HBASE-6612 URL: https://issues.apache.org/jira/browse/HBASE-6612 Project: HBase Issue Type: New Feature Components: scripts, shell Affects Versions: 0.94.1 Reporter: Ionut Ignatescu Priority: Minor Currently, if the row key or any column value is something different than a string, when a scan is performed via command line, the value extracted are not decoded to a human-readable format. It would be nice to have support to some standard data types(long,double,etc..) or to specify some custom decoders(this would be extremely useful for tables having composed keys). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6613) Automatically merge empty regions
Ionut Ignatescu created HBASE-6613: -- Summary: Automatically merge empty regions Key: HBASE-6613 URL: https://issues.apache.org/jira/browse/HBASE-6613 Project: HBase Issue Type: New Feature Components: master, regionserver, util Affects Versions: 0.94.1 Reporter: Ionut Ignatescu Consider an usecase where row keys has an increasing value(time-series data for example) and data retention is set to a concrete value(60 days for example). After a period of time, longer than retention, empty regions will appear. This will cause high memory use on region servers. In my opinion, regions merge could be part of major compaction or another tool should be provided. From my understading, it is possible to merge 2 empty regions without make table offline, but it's not possible to merge one empty region with a non-empty region without close/unassing this regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6414: --- Attachment: 6414-6.patch.txt Ted had uploaded the last patch from RB (thanks, Ted!) but hadoopqa didn't pick it up.. Re-uploading the same patch. Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437565#comment-13437565 ] Hadoop QA commented on HBASE-6414: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541532/6414-6.patch.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestAssignmentManager Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2622//console This message is automatically generated. Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6589) RegionServer can't load class for dynamically loaded coprocessors with self defined class
[
https://issues.apache.org/jira/browse/HBASE-6589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437567#comment-13437567
]
Himanshu Vashishtha commented on HBASE-6589:
Are you using HBASE-6308? If yes, then MultiColumnSchema should be loaded by
CoprocessorClassLoader. This error is when you try to invoke the CP, or the
loading itself?
RegionServer can't load class for dynamically loaded coprocessors with self
defined class
-
Key: HBASE-6589
URL: https://issues.apache.org/jira/browse/HBASE-6589
Project: HBase
Issue Type: Bug
Components: coprocessors, regionserver
Reporter: ShiXing
When using coprocessor with custom classes like LongColumnInterpreter(mine is
MultiColumnSchema), the coprocessor can not work for hot deploy, if the
custom classes do not deploy in the regionserver's classpath. Although the
self-defined class is deployed in the regions' classpath through hdfs jar.
The exception threw at the regionserver's log:
{code}
2012-08-15 16:24:24,403 ERROR org.apache.hadoop.hbase.io.HbaseObjectWritable:
Error in readFields
java.io.IOException: Can't find class
com.taobao.hbase.coprocessor.MultiColumnSchema
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:674)
at
org.apache.hadoop.hbase.client.coprocessor.Exec.readFields(Exec.java:114)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:682)
at
org.apache.hadoop.hbase.ipc.Invocation.readFields(Invocation.java:125)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.processData(HBaseServer.java:1292)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Connection.readAndProcess(HBaseServer.java:1207)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener.doRead(HBaseServer.java:735)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:524)
at
org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:499)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassNotFoundException:
com.taobao.hbase.coprocessor.MultiColumnSchema
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.getClassByName(HbaseObjectWritable.java:784)
at
org.apache.hadoop.hbase.io.HbaseObjectWritable.readObject(HbaseObjectWritable.java:671)
... 11 more
{code}
It is similar as HBASE-4946, but I do not know how to solve this bug.
If add these custom class to the RegionServer's classloader may fix it, but
it is conflicted with HBASE-6308 to prevent dependency conflicts.
Does anyone have some idea?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6567) make memory locking configuration of regioservers more flexible
[ https://issues.apache.org/jira/browse/HBASE-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437576#comment-13437576 ] stack commented on HBASE-6567: -- @Matteo I made a mistake in application. Should be fixed now. Thanks for noticing it. make memory locking configuration of regioservers more flexible --- Key: HBASE-6567 URL: https://issues.apache.org/jira/browse/HBASE-6567 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.96.0 Reporter: Roman Shaposhnik Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: HBASE-6567-v0.patch The current implementation of the memory locking feature of regisoservers has a downside of not being flexible to configure for permanent use. Sure there is a --mlock flag but that needs to be explicitly passed on every invocation and thus require extra steps to be configured for permanent use (IOW, there's not a single env variable I can set to have a desired effect). The only other alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a downside of being pretty cryptic to the novice user and has a killer problem of not explicitly telling higher level scripts (like init.d or upstart ones) which user the initial hbase process should be executed as. I propose a very simple solution (which is essentially making --mlock setting into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that can be set in hbase-env.sh and has the following semantics: * [default] not set: mlocking feature is disabled * set but empty: mlocking feature is enabled and the target user is hbase * set and not empty: mlocking feature is enabled and the target user is the value of the variable Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4391) Add ability to start RS as root and call mlockall
[ https://issues.apache.org/jira/browse/HBASE-4391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi resolved HBASE-4391. Resolution: Fixed Add ability to start RS as root and call mlockall - Key: HBASE-4391 URL: https://issues.apache.org/jira/browse/HBASE-4391 Project: HBase Issue Type: New Feature Components: regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: 4391-v4.patch, HBASE-4391-v0.patch, HBASE-4391-v1.patch, HBASE-4391-v2.patch, HBASE-4391-v3.patch A common issue we've seen in practice is that users oversubscribe their region servers with too many MR tasks, etc. As soon as the machine starts swapping, the RS grinds to a halt, loses ZK session, aborts, etc. This can be combatted by starting the RS as root, calling mlockall(), and then setuid down to the hbase user. We should not require this, but we should provide it as an option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5934) Add the ability for Performance Evaluation to set the table compression
[ https://issues.apache.org/jira/browse/HBASE-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5934: --- Fix Version/s: 0.96.0 Add the ability for Performance Evaluation to set the table compression --- Key: HBASE-5934 URL: https://issues.apache.org/jira/browse/HBASE-5934 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Matteo Bertozzi Priority: Minor Fix For: 0.96.0 Attachments: HBASE-5934-v0.patch, HBASE-5934-v1.patch When testing it's nice to get a more realistic set of numbers. As such allowing the tool to create pre-split regions was needed. Compression should be the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5934) Add the ability for Performance Evaluation to set the table compression
[ https://issues.apache.org/jira/browse/HBASE-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-5934: --- Resolution: Fixed Status: Resolved (was: Patch Available) Add the ability for Performance Evaluation to set the table compression --- Key: HBASE-5934 URL: https://issues.apache.org/jira/browse/HBASE-5934 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Matteo Bertozzi Priority: Minor Attachments: HBASE-5934-v0.patch, HBASE-5934-v1.patch When testing it's nice to get a more realistic set of numbers. As such allowing the tool to create pre-split regions was needed. Compression should be the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6414: - Attachment: 6414-v7.txt v6 w/ a DEBUG log removed (we were logging every call).. I changed it to trace level. Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt, 6414-v7.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6414: - Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Status: Resolved (was: Patch Available) Committed v7. Good stuff DD. Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt, 6414-v7.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6567) make memory locking configuration of regioservers more flexible
[ https://issues.apache.org/jira/browse/HBASE-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437588#comment-13437588 ] Hudson commented on HBASE-6567: --- Integrated in HBase-TRUNK #3240 (See [https://builds.apache.org/job/HBase-TRUNK/3240/]) HBASE-6567 make memory locking configuration of regioservers more flexible; INCOMPLETE APPLICATION (Revision 1374848) Result = FAILURE stack : Files : * /hbase/trunk/conf/hbase-env.sh make memory locking configuration of regioservers more flexible --- Key: HBASE-6567 URL: https://issues.apache.org/jira/browse/HBASE-6567 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.96.0 Reporter: Roman Shaposhnik Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: HBASE-6567-v0.patch The current implementation of the memory locking feature of regisoservers has a downside of not being flexible to configure for permanent use. Sure there is a --mlock flag but that needs to be explicitly passed on every invocation and thus require extra steps to be configured for permanent use (IOW, there's not a single env variable I can set to have a desired effect). The only other alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a downside of being pretty cryptic to the novice user and has a killer problem of not explicitly telling higher level scripts (like init.d or upstart ones) which user the initial hbase process should be executed as. I propose a very simple solution (which is essentially making --mlock setting into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that can be set in hbase-env.sh and has the following semantics: * [default] not set: mlocking feature is disabled * set but empty: mlocking feature is enabled and the target user is hbase * set and not empty: mlocking feature is enabled and the target user is the value of the variable Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6435) Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes
[ https://issues.apache.org/jira/browse/HBASE-6435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437590#comment-13437590 ] stack commented on HBASE-6435: -- @N Nice note. You should write a blog on it and your other findings. You going to commit? Reading WAL files after a recovery leads to time lost in HDFS timeouts when using dead datanodes Key: HBASE-6435 URL: https://issues.apache.org/jira/browse/HBASE-6435 Project: HBase Issue Type: Improvement Components: master, regionserver Affects Versions: 0.96.0 Reporter: nkeywal Assignee: nkeywal Fix For: 0.96.0 Attachments: 6435.unfinished.patch, 6435.v10.patch, 6435.v10.patch, 6435.v12.patch, 6435.v12.patch, 6435.v12.patch, 6435-v12.txt, 6435.v13.patch, 6435.v2.patch, 6435.v7.patch, 6435.v8.patch, 6435.v9.patch, 6435.v9.patch, 6535.v11.patch HBase writes a Write-Ahead-Log to revover from hardware failure. This log is written with 'append' on hdfs. Through ZooKeeper, HBase gets informed usually in 30s that it should start the recovery process. This means reading the Write-Ahead-Log to replay the edits on the other servers. In standards deployments, HBase process (regionserver) are deployed on the same box as the datanodes. It means that when the box stops, we've actually lost one of the edits, as we lost both the regionserver and the datanode. As HDFS marks a node as dead after ~10 minutes, it appears as available when we try to read the blocks to recover. As such, we are delaying the recovery process by 60 seconds as the read will usually fail with a socket timeout. If the file is still opened for writing, it adds an extra 20s + a risk of losing edits if we connect with ipc to the dead DN. Possible solutions are: - shorter dead datanodes detection by the NN. Requires a NN code change. - better dead datanodes management in DFSClient. Requires a DFS code change. - NN customisation to write the WAL files on another DN instead of the local one. - reordering the blocks returned by the NN on the client side to put the blocks on the same DN as the dead RS at the end of the priority queue. Requires a DFS code change or a kind of workaround. The solution retained is the last one. Compared to what was discussed on the mailing list, the proposed patch will not modify HDFS source code but adds a proxy. This for two reasons: - Some HDFS functions managing block orders are static (MD5MD5CRC32FileChecksum). Implementing the hook in the DFSClient would require to implement partially the fix, change the DFS interface to make this function non static, or put the hook static. None of these solution is very clean. - Adding a proxy allows to put all the code in HBase, simplifying dependency management. Nevertheless, it would be better to have this in HDFS. But this solution allows to target the last version only, and this could allow minimal interface changes such as non static methods. Moreover, writing the blocks to the non local DN would be an even better solution long term. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5449) Support for wire-compatible security functionality
[ https://issues.apache.org/jira/browse/HBASE-5449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437593#comment-13437593 ] stack commented on HBASE-5449: -- Is UserTablePermission from Gary? You think it worth adding a bit of commentary to the .proto file or you think it basically self-explanatory (I could buy that... Permission in a .proto file named AccessControl is pretty hard to misinterpret). I see a TablePermission under access. When you say above you remove TablePermission, you mean from pb file? I think this committable. Gary or Andrew want to take a looksee? I'll commit in a few days unless they intercede. Good on you Matteo. Support for wire-compatible security functionality -- Key: HBASE-5449 URL: https://issues.apache.org/jira/browse/HBASE-5449 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Matteo Bertozzi Attachments: AccessControl_protos.patch, HBASE-5449-v0.patch, HBASE-5449-v1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4709) Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
[
https://issues.apache.org/jira/browse/HBASE-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437596#comment-13437596
]
stack commented on HBASE-4709:
--
bq. In issue page someone tells it does not prevent James to come up.
Can you tell us more why the above WARN message is preventing James launching?
Maybe its not coming up for some other reason? You have more logs for us to
look at? Ask on dev list and add log there rather than here?
Hadoop metrics2 setup in test MiniDFSClusters spewing JMX errors
Key: HBASE-4709
URL: https://issues.apache.org/jira/browse/HBASE-4709
Project: HBase
Issue Type: Bug
Components: test
Affects Versions: 0.92.0, 0.94.0, 0.94.1
Reporter: Gary Helmling
Priority: Minor
Since switching over HBase to build with Hadoop 0.20.205.0, we've been
getting a lot of metrics related errors in the log files for tests:
{noformat}
2011-10-30 22:00:22,858 INFO [main] log.Slf4jLog(67): jetty-6.1.26
2011-10-30 22:00:22,871 INFO [main] log.Slf4jLog(67): Extract
jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/0.20.205.0/hadoop-core-0.20.205.0.jar!/webapps/datanode
to /tmp/Jetty_localhost_55751_datanode.kw16hy/webapp
2011-10-30 22:00:23,048 INFO [main] log.Slf4jLog(67): Started
SelectChannelConnector@localhost:55751
Starting DataNode 1 with dfs.data.dir:
/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/HBase-TRUNK/trunk/target/test-data/7ba65a16-03ad-4624-b769-57405945ef58/dfscluster_3775fc23-1b51-4966-8133-205564bae762/dfs/data/data4
2011-10-30 22:00:23,237 WARN [main] impl.MetricsSystemImpl(137): Metrics
system not started: Cannot locate configuration: tried
hadoop-metrics2-datanode.properties, hadoop-metrics2.properties
2011-10-30 22:00:23,237 WARN [main] util.MBeans(59):
Hadoop:service=DataNode,name=MetricsSystem,sub=Control
javax.management.InstanceAlreadyExistsException: MXBean already registered
with name Hadoop:service=NameNode,name=MetricsSystem,sub=Control
at
com.sun.jmx.mbeanserver.MXBeanLookup.addReference(MXBeanLookup.java:120)
at
com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:143)
at
com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:56)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.initSystemMBean(MetricsSystemImpl.java:500)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:140)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:40)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1483)
at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1459)
at
org.apache.hadoop.hdfs.MiniDFSCluster.startDataNodes(MiniDFSCluster.java:417)
at org.apache.hadoop.hdfs.MiniDFSCluster.init(MiniDFSCluster.java:280)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:349)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:518)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:474)
at
org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:461)
{noformat}
This seems to be due to errors initializing the new hadoop metrics2 code by
default, when running in a mini cluster. The errors themselves seem to be
harmless -- they're not breaking any tests -- but we should figure out what
configuration we need to eliminate them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437601#comment-13437601 ] Hudson commented on HBASE-6414: --- Integrated in HBase-TRUNK #3241 (See [https://builds.apache.org/job/HBase-TRUNK/3241/]) HBASE-6414 Remove the WritableRpcEngine associated Invocation classes (Revision 1374860) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ClientCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/protobuf/RPC.proto * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestDelayedRpcProtos.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityRpc.java * /hbase/trunk/hbase-server/src/test/protobuf/test_delayed_rpc.proto Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt, 6414-v7.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6567) make memory locking configuration of regioservers more flexible
[ https://issues.apache.org/jira/browse/HBASE-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437609#comment-13437609 ] Hudson commented on HBASE-6567: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #137 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/137/]) HBASE-6567 make memory locking configuration of regioservers more flexible; INCOMPLETE APPLICATION (Revision 1374848) Result = FAILURE stack : Files : * /hbase/trunk/conf/hbase-env.sh make memory locking configuration of regioservers more flexible --- Key: HBASE-6567 URL: https://issues.apache.org/jira/browse/HBASE-6567 Project: HBase Issue Type: Improvement Components: scripts Affects Versions: 0.96.0 Reporter: Roman Shaposhnik Assignee: Matteo Bertozzi Fix For: 0.96.0 Attachments: HBASE-6567-v0.patch The current implementation of the memory locking feature of regisoservers has a downside of not being flexible to configure for permanent use. Sure there is a --mlock flag but that needs to be explicitly passed on every invocation and thus require extra steps to be configured for permanent use (IOW, there's not a single env variable I can set to have a desired effect). The only other alternative -- the explicit setting of HBASE_REGIONSERVER_OPTS -- has a downside of being pretty cryptic to the novice user and has a killer problem of not explicitly telling higher level scripts (like init.d or upstart ones) which user the initial hbase process should be executed as. I propose a very simple solution (which is essentially making --mlock setting into an env. variable): add a variable called HBASE_REGIONSERVER_MLOCK that can be set in hbase-env.sh and has the following semantics: * [default] not set: mlocking feature is disabled * set but empty: mlocking feature is enabled and the target user is hbase * set and not empty: mlocking feature is enabled and the target user is the value of the variable Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6414) Remove the WritableRpcEngine associated Invocation classes
[ https://issues.apache.org/jira/browse/HBASE-6414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437608#comment-13437608 ] Hudson commented on HBASE-6414: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #137 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/137/]) HBASE-6414 Remove the WritableRpcEngine associated Invocation classes (Revision 1374860) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ClientCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/ProtobufRpcEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandler.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/monitoring/MonitoredRPCHandlerImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/protobuf/RPC.proto * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestDelayedRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestPBOnWritableRpc.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/protobuf/generated/TestDelayedRpcProtos.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestHMasterRPCException.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityRpc.java * /hbase/trunk/hbase-server/src/test/protobuf/test_delayed_rpc.proto Remove the WritableRpcEngine associated Invocation classes Key: HBASE-6414 URL: https://issues.apache.org/jira/browse/HBASE-6414 Project: HBase Issue Type: Improvement Affects Versions: 0.96.0 Reporter: Devaraj Das Assignee: Devaraj Das Fix For: 0.96.0 Attachments: 6414-1.patch.txt, 6414-3.patch.txt, 6414-4.patch.txt, 6414-4.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-5.patch.txt, 6414-6.patch.txt, 6414-6.patch.txt, 6414-6.txt, 6414-initial.patch.txt, 6414-initial.patch.txt, 6414-v7.txt Remove the WritableRpcEngine Invocation classes once HBASE-5705 gets committed and all the protocols are rebased to use PB. Raising this jira in advance.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3996) Support multiple tables and scanners as input to the mapper in map/reduce jobs
[ https://issues.apache.org/jira/browse/HBASE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437625#comment-13437625 ] Lars Hofhansl commented on HBASE-3996: -- @Eran and Bohdan: Are you still interested in finishing this? @Stack: So these are naming and package issues? Or would you restructure the code? Support multiple tables and scanners as input to the mapper in map/reduce jobs -- Key: HBASE-3996 URL: https://issues.apache.org/jira/browse/HBASE-3996 Project: HBase Issue Type: Improvement Components: mapreduce Reporter: Eran Kutner Assignee: Eran Kutner Fix For: 0.96.0, 0.94.2 Attachments: 3996-v2.txt, 3996-v3.txt, 3996-v4.txt, 3996-v5.txt, 3996-v6.txt, 3996-v7.txt, HBase-3996.patch It seems that in many cases feeding data from multiple tables or multiple scanners on a single table can save a lot of time when running map/reduce jobs. I propose a new MultiTableInputFormat class that would allow doing this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6588) enable table throws npe and leaves trash in zk in competition with delete table
[
https://issues.apache.org/jira/browse/HBASE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437626#comment-13437626
]
Zhou wenjian commented on HBASE-6588:
-
@stack
has corrected what Ram found.
the synchronous delete now uses async delete, that is right.
will update for v5 as suggested
enable table throws npe and leaves trash in zk in competition with delete
table
---
Key: HBASE-6588
URL: https://issues.apache.org/jira/browse/HBASE-6588
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Zhou wenjian
Fix For: 0.94.2
Attachments: HBASE-6588-trunk.patch, HBASE-6588-trunk-v2.patch,
HBASE-6588-trunk-v3.patch, HBASE-6588-trunk-v4.patch
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 2 catalog row(s) and gc'd 0 unreferenced parent region(s)
2012-08-15 19:25:12,260 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Deleting region
test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META and FS
2012-08-15 19:25:12,263 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted region test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META
2012-08-15 19:25:12,265 INFO
org.apache.hadoop.hbase.master.handler.EnableTableHandler: Attemping to
enable the table test
2012-08-15 19:25:12,265 WARN org.apache.hadoop.hbase.zookeeper.ZKTable:
Moving table test state to enabling but was not first in disabled state: null
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:25:12,270 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:25:12,270 ERROR org.apache.hadoop.hbase.executor.EventHandler:
Caught throwable while processing event C_M_ENABLE_TABLE
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:116)
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:97)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
table is disabled now, then we enable and delete the table at the same time.
Since the thread num of MASTER_TABLE_OPERATIONS is 1 by default. The two
operations are serial in master.Before deletetable deletes all the regions in
meta, CreateTableHandler ships the check of tableExists,then it will block
until deletetable finishs, then CreateTableHandler will set zk enabling, and
find no data in meta:
regionsInMeta = MetaReader.getTableRegions(this.ct, tableName, true);
int countOfRegionsInTable = regionsInMeta.size();
npe will be throwed here. And we could not create the same table anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6588) enable table throws npe and leaves trash in zk in competition with delete table
[
https://issues.apache.org/jira/browse/HBASE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhou wenjian updated HBASE-6588:
Attachment: HBASE-6588-trunk-v5.patch
enable table throws npe and leaves trash in zk in competition with delete
table
---
Key: HBASE-6588
URL: https://issues.apache.org/jira/browse/HBASE-6588
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Zhou wenjian
Fix For: 0.94.2
Attachments: HBASE-6588-trunk.patch, HBASE-6588-trunk-v2.patch,
HBASE-6588-trunk-v3.patch, HBASE-6588-trunk-v4.patch,
HBASE-6588-trunk-v5.patch
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 2 catalog row(s) and gc'd 0 unreferenced parent region(s)
2012-08-15 19:25:12,260 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Deleting region
test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META and FS
2012-08-15 19:25:12,263 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted region test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META
2012-08-15 19:25:12,265 INFO
org.apache.hadoop.hbase.master.handler.EnableTableHandler: Attemping to
enable the table test
2012-08-15 19:25:12,265 WARN org.apache.hadoop.hbase.zookeeper.ZKTable:
Moving table test state to enabling but was not first in disabled state: null
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:25:12,270 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:25:12,270 ERROR org.apache.hadoop.hbase.executor.EventHandler:
Caught throwable while processing event C_M_ENABLE_TABLE
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:116)
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:97)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
table is disabled now, then we enable and delete the table at the same time.
Since the thread num of MASTER_TABLE_OPERATIONS is 1 by default. The two
operations are serial in master.Before deletetable deletes all the regions in
meta, CreateTableHandler ships the check of tableExists,then it will block
until deletetable finishs, then CreateTableHandler will set zk enabling, and
find no data in meta:
regionsInMeta = MetaReader.getTableRegions(this.ct, tableName, true);
int countOfRegionsInTable = regionsInMeta.size();
npe will be throwed here. And we could not create the same table anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6473) deleted table is not deleted completely, some region may be still online
[
https://issues.apache.org/jira/browse/HBASE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437633#comment-13437633
]
Zhou wenjian commented on HBASE-6473:
-
@stack
that is right, if a region is offline, it will not in
am.getRegionStates().getRegionsOfTable(tableName) and rit.
Nice of you to add the parens on commit.
deleted table is not deleted completely, some region may be still online
Key: HBASE-6473
URL: https://issues.apache.org/jira/browse/HBASE-6473
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Fix For: 0.96.0, 0.94.2
Attachments: HBASE-6473-trunk.patch, HBASE-6473-trunk-v2.patch,
HBASE-6473-trunk-v3.patch
consider such Scenario:
we have a table called T1, which has 1 regions: A
1. move A from rs1 to rs2,and A is now closed
2. disable T1,
3. delete T1.
when we disable T1, disable handler will just set the zk to disabled and A
will still be assigned. when Ais opened, A in transition will be clean out.
At that time, Deletetable found it is safe to delete all regions and table in
meta and fs , it will also delete the zk node of T1.
{code}
while (System.currentTimeMillis() done) {
AssignmentManager.RegionState rs = am.isRegionInTransition(region);
if (rs == null) break;
Threads.sleep(waitingTimeForEvents);
LOG.debug(Waiting on region to clear regions in transition; + rs);
}
if (am.isRegionInTransition(region) != null) {
throw new IOException(Waited hbase.master.wait.on.region ( +
waitTime + ms) for region to leave region +
region.getRegionNameAsString() + in transitions);
}
{code}
however A is still being unassigned, when it finished closed the A,it finds
that the disabled state in zk is deleted, and then A will be assigned again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6473) deleted table is not deleted completely, some region may be still online
[
https://issues.apache.org/jira/browse/HBASE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437635#comment-13437635
]
Zhou wenjian commented on HBASE-6473:
-
Look into the failed case ,found nothing with my case, may be caused by
unstable hudsun
deleted table is not deleted completely, some region may be still online
Key: HBASE-6473
URL: https://issues.apache.org/jira/browse/HBASE-6473
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Fix For: 0.96.0, 0.94.2
Attachments: HBASE-6473-trunk.patch, HBASE-6473-trunk-v2.patch,
HBASE-6473-trunk-v3.patch
consider such Scenario:
we have a table called T1, which has 1 regions: A
1. move A from rs1 to rs2,and A is now closed
2. disable T1,
3. delete T1.
when we disable T1, disable handler will just set the zk to disabled and A
will still be assigned. when Ais opened, A in transition will be clean out.
At that time, Deletetable found it is safe to delete all regions and table in
meta and fs , it will also delete the zk node of T1.
{code}
while (System.currentTimeMillis() done) {
AssignmentManager.RegionState rs = am.isRegionInTransition(region);
if (rs == null) break;
Threads.sleep(waitingTimeForEvents);
LOG.debug(Waiting on region to clear regions in transition; + rs);
}
if (am.isRegionInTransition(region) != null) {
throw new IOException(Waited hbase.master.wait.on.region ( +
waitTime + ms) for region to leave region +
region.getRegionNameAsString() + in transitions);
}
{code}
however A is still being unassigned, when it finished closed the A,it finds
that the disabled state in zk is deleted, and then A will be assigned again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6603) RegionMetricsStorage.incrNumericMetric is called too often
[ https://issues.apache.org/jira/browse/HBASE-6603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437636#comment-13437636 ] Lars Hofhansl commented on HBASE-6603: -- Sure. Just make a 0.94/0.96 patch there. Patch there looks good to me. (copykv is gone now, though). RegionMetricsStorage.incrNumericMetric is called too often -- Key: HBASE-6603 URL: https://issues.apache.org/jira/browse/HBASE-6603 Project: HBase Issue Type: Bug Components: performance Reporter: Lars Hofhansl Assignee: Lars Hofhansl Fix For: 0.96.0, 0.94.2 Running an HBase scan load through the profiler revealed that RegionMetricsStorage.incrNumericMetric is called way too often. It turns out that we make this call for *each* KV in StoreScanner.next(...). Incrementing AtomicLong requires expensive memory barriers. The observation here is that StoreScanner.next(...) can maintain a simple long in its internal loop and only update the metric upon exit. Thus the AtomicLong is not updated nearly as often. That cuts about 10% runtime from scan only load (I'll quantify this better soon). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6537) Balancer compete with disable table will lead to cluster inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou wenjian updated HBASE-6537: Status: Patch Available (was: Open) Balancer compete with disable table will lead to cluster inconsistent -- Key: HBASE-6537 URL: https://issues.apache.org/jira/browse/HBASE-6537 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Zhou wenjian Fix For: 0.94.2 Attachments: HBASE-6537-94.patch, HBASE-6537-94-v2.patch, HBASE-6537-trunk.patch Appear in 94. trunk is ok for the issue Balancer will collect the regionplans to move(unassign and then assign). before unassign, disable table appears, after close the region in rs, master will delete the znode, romove region from RIT, and then clean the region from the online regions. During romoving region from RIT and cleaning out the region from the online regions. balancer begins to unassign, it will get a NotServingRegionException and if the table is disabling, it will deal with the state in master and delete the znode . However the table is disabled now, so the RIT and znode will remain. TimeoutMonitor draws a blank on it. It will hold back enabling the table or balancer unless restart -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6537) Balancer compete with disable table will lead to cluster inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437639#comment-13437639 ] Hadoop QA commented on HBASE-6537: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540261/HBASE-6537-94-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2624//console This message is automatically generated. Balancer compete with disable table will lead to cluster inconsistent -- Key: HBASE-6537 URL: https://issues.apache.org/jira/browse/HBASE-6537 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Zhou wenjian Fix For: 0.94.2 Attachments: HBASE-6537-94.patch, HBASE-6537-94-v2.patch, HBASE-6537-trunk.patch Appear in 94. trunk is ok for the issue Balancer will collect the regionplans to move(unassign and then assign). before unassign, disable table appears, after close the region in rs, master will delete the znode, romove region from RIT, and then clean the region from the online regions. During romoving region from RIT and cleaning out the region from the online regions. balancer begins to unassign, it will get a NotServingRegionException and if the table is disabling, it will deal with the state in master and delete the znode . However the table is disabled now, so the RIT and znode will remain. TimeoutMonitor draws a blank on it. It will hold back enabling the table or balancer unless restart -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6584) Test flappers due to port 60000 already in use.
[ https://issues.apache.org/jira/browse/HBASE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6584: -- Attachment: HBASE-6584_trunk.patch Test flappers due to port 6 already in use. --- Key: HBASE-6584 URL: https://issues.apache.org/jira/browse/HBASE-6584 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Reporter: s...@hotmail.com Priority: Critical Attachments: HBASE-6584_trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6584) Test flappers due to port 60000 already in use.
[ https://issues.apache.org/jira/browse/HBASE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6584: -- Attachment: HBASE-6584_trunk.patch This problem exists in trunk only. While writing testDisablingTableRegionsAssignmentDuringCleanClusterStartup in TestAssignmentmanager created HMaster object with HTU.getConfiguration() instead of mocked master. Test flappers due to port 6 already in use. --- Key: HBASE-6584 URL: https://issues.apache.org/jira/browse/HBASE-6584 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Reporter: s...@hotmail.com Priority: Critical Attachments: HBASE-6584_trunk.patch, HBASE-6584_trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6537) Balancer compete with disable table will lead to cluster inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou wenjian updated HBASE-6537: Status: Open (was: Patch Available) Balancer compete with disable table will lead to cluster inconsistent -- Key: HBASE-6537 URL: https://issues.apache.org/jira/browse/HBASE-6537 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Zhou wenjian Fix For: 0.94.2 Attachments: HBASE-6537-94.patch, HBASE-6537-94-v2.patch, HBASE-6537-trunk.patch Appear in 94. trunk is ok for the issue Balancer will collect the regionplans to move(unassign and then assign). before unassign, disable table appears, after close the region in rs, master will delete the znode, romove region from RIT, and then clean the region from the online regions. During romoving region from RIT and cleaning out the region from the online regions. balancer begins to unassign, it will get a NotServingRegionException and if the table is disabling, it will deal with the state in master and delete the znode . However the table is disabled now, so the RIT and znode will remain. TimeoutMonitor draws a blank on it. It will hold back enabling the table or balancer unless restart -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6537) Balancer compete with disable table will lead to cluster inconsistent
[ https://issues.apache.org/jira/browse/HBASE-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhou wenjian updated HBASE-6537: Status: Patch Available (was: Open) Balancer compete with disable table will lead to cluster inconsistent -- Key: HBASE-6537 URL: https://issues.apache.org/jira/browse/HBASE-6537 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Zhou wenjian Fix For: 0.94.2 Attachments: HBASE-6537-94.patch, HBASE-6537-94-v2.patch, HBASE-6537-trunk.patch Appear in 94. trunk is ok for the issue Balancer will collect the regionplans to move(unassign and then assign). before unassign, disable table appears, after close the region in rs, master will delete the znode, romove region from RIT, and then clean the region from the online regions. During romoving region from RIT and cleaning out the region from the online regions. balancer begins to unassign, it will get a NotServingRegionException and if the table is disabling, it will deal with the state in master and delete the znode . However the table is disabled now, so the RIT and znode will remain. TimeoutMonitor draws a blank on it. It will hold back enabling the table or balancer unless restart -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6584) Test flappers due to port 60000 already in use.
[ https://issues.apache.org/jira/browse/HBASE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6584: -- Status: Patch Available (was: Open) Test flappers due to port 6 already in use. --- Key: HBASE-6584 URL: https://issues.apache.org/jira/browse/HBASE-6584 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Reporter: s...@hotmail.com Priority: Critical Attachments: HBASE-6584_trunk.patch, HBASE-6584_trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6473) deleted table is not deleted completely, some region may be still online
[
https://issues.apache.org/jira/browse/HBASE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Ted Yu reassigned HBASE-6473:
-
Assignee: Zhou wenjian
deleted table is not deleted completely, some region may be still online
Key: HBASE-6473
URL: https://issues.apache.org/jira/browse/HBASE-6473
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: zhou wenjian
Assignee: Zhou wenjian
Fix For: 0.96.0, 0.94.2
Attachments: HBASE-6473-trunk.patch, HBASE-6473-trunk-v2.patch,
HBASE-6473-trunk-v3.patch
consider such Scenario:
we have a table called T1, which has 1 regions: A
1. move A from rs1 to rs2,and A is now closed
2. disable T1,
3. delete T1.
when we disable T1, disable handler will just set the zk to disabled and A
will still be assigned. when Ais opened, A in transition will be clean out.
At that time, Deletetable found it is safe to delete all regions and table in
meta and fs , it will also delete the zk node of T1.
{code}
while (System.currentTimeMillis() done) {
AssignmentManager.RegionState rs = am.isRegionInTransition(region);
if (rs == null) break;
Threads.sleep(waitingTimeForEvents);
LOG.debug(Waiting on region to clear regions in transition; + rs);
}
if (am.isRegionInTransition(region) != null) {
throw new IOException(Waited hbase.master.wait.on.region ( +
waitTime + ms) for region to leave region +
region.getRegionNameAsString() + in transitions);
}
{code}
however A is still being unassigned, when it finished closed the A,it finds
that the disabled state in zk is deleted, and then A will be assigned again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6588) enable table throws npe and leaves trash in zk in competition with delete table
[
https://issues.apache.org/jira/browse/HBASE-6588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Ted Yu reassigned HBASE-6588:
-
Assignee: Zhou wenjian
enable table throws npe and leaves trash in zk in competition with delete
table
---
Key: HBASE-6588
URL: https://issues.apache.org/jira/browse/HBASE-6588
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.0
Reporter: Zhou wenjian
Assignee: Zhou wenjian
Fix For: 0.94.2
Attachments: HBASE-6588-trunk.patch, HBASE-6588-trunk-v2.patch,
HBASE-6588-trunk-v3.patch, HBASE-6588-trunk-v4.patch,
HBASE-6588-trunk-v5.patch
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:23:36,178 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key ''
2012-08-15 19:24:09,180 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at ''
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:24:09,183 DEBUG org.apache.hadoop.hbase.master.CatalogJanitor:
Scanned 2 catalog row(s) and gc'd 0 unreferenced parent region(s)
2012-08-15 19:25:12,260 DEBUG
org.apache.hadoop.hbase.master.handler.DeleteTableHandler: Deleting region
test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META and FS
2012-08-15 19:25:12,263 INFO org.apache.hadoop.hbase.catalog.MetaEditor:
Deleted region test,,1345029764571.d1e24b251ca6286c840a9a5f571b7db1. from META
2012-08-15 19:25:12,265 INFO
org.apache.hadoop.hbase.master.handler.EnableTableHandler: Attemping to
enable the table test
2012-08-15 19:25:12,265 WARN org.apache.hadoop.hbase.zookeeper.ZKTable:
Moving table test state to enabling but was not first in disabled state: null
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Creating scanner over .META. starting at key 'test,,'
2012-08-15 19:25:12,267 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Advancing internal scanner to startKey at 'test,,'
2012-08-15 19:25:12,270 DEBUG org.apache.hadoop.hbase.client.ClientScanner:
Finished with scanning at {NAME = '.META.,,1', STARTKEY = '', ENDKEY = '',
ENCODED = 1028785192,}
2012-08-15 19:25:12,270 ERROR org.apache.hadoop.hbase.executor.EventHandler:
Caught throwable while processing event C_M_ENABLE_TABLE
java.lang.NullPointerException
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.handleEnableTable(EnableTableHandler.java:116)
at
org.apache.hadoop.hbase.master.handler.EnableTableHandler.process(EnableTableHandler.java:97)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
table is disabled now, then we enable and delete the table at the same time.
Since the thread num of MASTER_TABLE_OPERATIONS is 1 by default. The two
operations are serial in master.Before deletetable deletes all the regions in
meta, CreateTableHandler ships the check of tableExists,then it will block
until deletetable finishs, then CreateTableHandler will set zk enabling, and
find no data in meta:
regionsInMeta = MetaReader.getTableRegions(this.ct, tableName, true);
int countOfRegionsInTable = regionsInMeta.size();
npe will be throwed here. And we could not create the same table anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6584) Test flappers due to port 60000 already in use.
[ https://issues.apache.org/jira/browse/HBASE-6584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13437659#comment-13437659 ] Hadoop QA commented on HBASE-6584: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12541557/HBASE-6584_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.master.TestAssignmentManager org.apache.hadoop.hbase.TestMultiVersions Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2625//console This message is automatically generated. Test flappers due to port 6 already in use. --- Key: HBASE-6584 URL: https://issues.apache.org/jira/browse/HBASE-6584 Project: HBase Issue Type: Bug Components: test Affects Versions: 0.94.0 Reporter: s...@hotmail.com Priority: Critical Attachments: HBASE-6584_trunk.patch, HBASE-6584_trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
