[jira] [Commented] (HBASE-7204) Make hbck ErrorReporter pluggable
[ https://issues.apache.org/jira/browse/HBASE-7204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503524#comment-13503524 ] Jimmy Xiang commented on HBASE-7204: -Dhbasefsck.errorreporter=x will set it in the system properties, right? Make hbck ErrorReporter pluggable - Key: HBASE-7204 URL: https://issues.apache.org/jira/browse/HBASE-7204 Project: HBase Issue Type: Improvement Components: hbck Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Minor Attachments: trunk-7204.patch Make hbck ErrorReporter pluggable so that it can be replaced dynamically. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503528#comment-13503528 ] Jimmy Xiang commented on HBASE-5954: Those methods should be removed. But old applications may not compile any more if someone happens to use them. Are we ok with that? Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503557#comment-13503557 ] Lars Hofhansl commented on HBASE-5954: -- I'd vote for removing them. If we keep these we have failed with wire compatibility and all the protofuf stuff was for nothing. Put/Delete is still used as writable at least in these cases: * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503558#comment-13503558 ] Andrew Purtell commented on HBASE-5954: --- bq. I'd vote for removing them. If we keep these we have failed with wire compatibility and all the protofuf stuff was for nothing. +1 Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7160) Improve IdLock and remove its minor defects
[ https://issues.apache.org/jira/browse/HBASE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-7160: - Attachment: HBASE-7160-V4.patch Patch v4 from review board. Improve IdLock and remove its minor defects --- Key: HBASE-7160 URL: https://issues.apache.org/jira/browse/HBASE-7160 Project: HBase Issue Type: Improvement Reporter: Hiroshi Ikeda Assignee: Hiroshi Ikeda Priority: Minor Attachments: HBASE-7160.patch, HBASE-7160-V2.patch, HBASE-7160-V3.patch, HBASE-7160-V4.patch Combination of synchronizations and concurrent collections complicates the code, and it is hard to trace the code and to confirm its correctness. We should re-create the class and make it more understandable. In the current code, I find the following minor defects: (1) In the case that there is a waiting thread for a lock in getLockEntry() and another thread is releasing the lock by calling releaseLockEntry(), trying to get the lock with a 3rd thread by calling getLockEntry() falls into a busy loop until the waiting thread wakes up and gets the lock. Even if notify() wakes up the blocked thread and causes a context switch to the waked thread immediately, synchronization might block the waked thread and cause another context switch, and the busy loop might continue for a while. (2) In the same case as (1), since releasing the lock is merely notifying without removing the lock-entry from the map, interrupting the waiting thread might leave an unused lock-entry (entry.numWaiters == 0) in the map. This is a memory leak unless the id of the lock is used again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7215) Put/Delete/Increment still implement Writable
Lars Hofhansl created HBASE-7215: Summary: Put/Delete/Increment still implement Writable Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put, Delete, and Increment still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Summary: Put, Delete, and Increment still implement Writable (was: Put and Delete still implement Writable) Put, Delete, and Increment still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put and Delete still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Summary: Put and Delete still implement Writable (was: Put/Delete/Increment still implement Writable) Put and Delete still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, and Increment still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503612#comment-13503612 ] Lars Hofhansl commented on HBASE-7215: -- Result.java also still implements Writable. The sticky part is that M/R expects Writables. So if we have Put/Delete not as Writable they cannot be used in M/R, but if we keep them as Writable we are not using protobufs everywhere and hence we do not have wire compatibility. Maybe readFields and write could be implemented in terms of protobuf, so that we still have Writables, but all encoding/decoding is done by protobuf. Put, Delete, and Increment still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5954) Allow proper fsync support for HBase
[ https://issues.apache.org/jira/browse/HBASE-5954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503613#comment-13503613 ] Lars Hofhansl commented on HBASE-5954: -- Filed HBASE-7215. Allow proper fsync support for HBase Key: HBASE-5954 URL: https://issues.apache.org/jira/browse/HBASE-5954 Project: HBase Issue Type: Improvement Reporter: Lars Hofhansl Assignee: Lars Hofhansl Attachments: 5954-trunk-hdfs-trunk.txt, 5954-trunk-hdfs-trunk-v2.txt, 5954-trunk-hdfs-trunk-v3.txt, 5954-trunk-hdfs-trunk-v4.txt, 5954-trunk-hdfs-trunk-v5.txt, 5954-trunk-hdfs-trunk-v6.txt, hbase-hdfs-744.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7160) Improve IdLock and remove its minor defects
[
https://issues.apache.org/jira/browse/HBASE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503614#comment-13503614
]
Hadoop QA commented on HBASE-7160:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554815/HBASE-7160-V4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3399//console
This message is automatically generated.
Improve IdLock and remove its minor defects
---
Key: HBASE-7160
URL: https://issues.apache.org/jira/browse/HBASE-7160
Project: HBase
Issue Type: Improvement
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
Attachments: HBASE-7160.patch, HBASE-7160-V2.patch,
HBASE-7160-V3.patch, HBASE-7160-V4.patch
Combination of synchronizations and concurrent collections complicates the
code, and it is hard to trace the code and to confirm its correctness. We
should re-create the class and make it more understandable.
In the current code, I find the following minor defects:
(1) In the case that there is a waiting thread for a lock in getLockEntry()
and another thread is releasing the lock by calling releaseLockEntry(),
trying to get the lock with a 3rd thread by calling getLockEntry() falls into
a busy loop until the waiting thread wakes up and gets the lock.
Even if notify() wakes up the blocked thread and causes a context switch to
the waked thread immediately, synchronization might block the waked thread
and cause another context switch, and the busy loop might continue for a
while.
(2) In the same case as (1), since releasing the lock is merely notifying
without removing the lock-entry from the map, interrupting the waiting thread
might leave an unused lock-entry (entry.numWaiters == 0) in the map. This is
a memory leak unless the id of the lock is used again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7216) sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync
binlijin created HBASE-7216: --- Summary: sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync Key: HBASE-7216 URL: https://issues.apache.org/jira/browse/HBASE-7216 Project: HBase Issue Type: Improvement Reporter: binlijin Current when hlog rollWriter: (1)createWriter will call ClientProtocol.create (2)first hlog sync will trigger ClientProtocol.addBlock and establish write pipeline we can sync the outputstream to force trigger ClientProtocol.addBlock and establish write pipeline before the first hlog sync -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7216) sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync
[ https://issues.apache.org/jira/browse/HBASE-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-7216: Attachment: HBASE-7216.patch sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync -- Key: HBASE-7216 URL: https://issues.apache.org/jira/browse/HBASE-7216 Project: HBase Issue Type: Improvement Reporter: binlijin Attachments: HBASE-7216.patch Current when hlog rollWriter: (1)createWriter will call ClientProtocol.create (2)first hlog sync will trigger ClientProtocol.addBlock and establish write pipeline we can sync the outputstream to force trigger ClientProtocol.addBlock and establish write pipeline before the first hlog sync -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7216) sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync
[ https://issues.apache.org/jira/browse/HBASE-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-7216: Attachment: HBASE-7216-94.patch sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync -- Key: HBASE-7216 URL: https://issues.apache.org/jira/browse/HBASE-7216 Project: HBase Issue Type: Improvement Reporter: binlijin Attachments: HBASE-7216-94.patch, HBASE-7216.patch Current when hlog rollWriter: (1)createWriter will call ClientProtocol.create (2)first hlog sync will trigger ClientProtocol.addBlock and establish write pipeline we can sync the outputstream to force trigger ClientProtocol.addBlock and establish write pipeline before the first hlog sync -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7215) Put, Delete, and Increment still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503624#comment-13503624 ] Lars Hofhansl commented on HBASE-7215: -- After some looking... If we want continue to support Put/Delete/Result in M/R, using protobufs to implement readFields and write is the only way out. In Writable.write(DataOutput) we'd create the write Mutate protobuf object, write that to a bytearray and then write that byte array into the DataOutput. It's too bad we only have a DataOutut here and not an OutputStream, othewise we could just stream the protobuf object through that stream, saving one copy of all the data. In Writable.readFields we do something akin to ProtoBufutil.toDelete after we read the Mutate object from the stream. That means the data is not streamed, but instead materialized in memory three(!) times in these cases (the object itself, its protobuf representation, and the byte array to be sent across the wire). I'd welcome any better ideas! Put, Delete, and Increment still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7216) sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync
[ https://issues.apache.org/jira/browse/HBASE-7216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503625#comment-13503625 ] Lars Hofhansl commented on HBASE-7216: -- Reasoning sounds good to me. Do you have some benchmark numbers by any chance [~aoxiang]? sync hlog writer just after create it, so trigger addBlock and establish write pipeline before the first hlog sync -- Key: HBASE-7216 URL: https://issues.apache.org/jira/browse/HBASE-7216 Project: HBase Issue Type: Improvement Reporter: binlijin Attachments: HBASE-7216-94.patch, HBASE-7216.patch Current when hlog rollWriter: (1)createWriter will call ClientProtocol.create (2)first hlog sync will trigger ClientProtocol.addBlock and establish write pipeline we can sync the outputstream to force trigger ClientProtocol.addBlock and establish write pipeline before the first hlog sync -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7215) Put, Delete, Increment, and Result still implement Writable
[ https://issues.apache.org/jira/browse/HBASE-7215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7215: - Summary: Put, Delete, Increment, and Result still implement Writable (was: Put, Delete, and Increment still implement Writable) Put, Delete, Increment, and Result still implement Writable --- Key: HBASE-7215 URL: https://issues.apache.org/jira/browse/HBASE-7215 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Priority: Blocker Fix For: 0.96.0 Making blocker as suggested by Stack. At least the following still use Put/Delete as writables. * IdentityTableReduce.java * MultiPut.java * HRegionServer.checkAndMutate -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
