date:20140527

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009310#comment-14009310
]

Hadoop QA commented on HBASE-11126:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12646663/HBASE-11126_new_3.patch
against trunk revision .
ATTACHMENT ID: 12646663

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9596//console

This message is automatically generated.

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0, 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126_1.patch, HBASE-11126_new_2.patch,
HBASE-11126_new_3.patch

The coprocessor hooks were placed outside of row locks. This was meant to
sidestep performance issues arising from significant work done within hook
invocations. However as the security code increases in sophistication we are
now running into concurrency issues trying to use them as a result of that
early decision. Since the initial introduction of coprocessor upcalls there
has been some significant refactoring done around them and concurrency
control in core has become more complex. This is potentially an issue for
many coprocessor users.
We should do either:\\
- Move all existing RegionObserver pre* hooks to execute under row lock.
- Introduce a new set of RegionObserver pre* hooks that execute under row
lock, named to indicate such.
The second option is less likely to lead to surprises.
All RegionObserver hook Javadoc should be updated with advice to the
coprocessor implementor not to take their own row locks in the hook. If the
current thread happens to already have a row lock and they try to take a lock
on another row, there is a deadlock risk.
As always a drawback of adding hooks is the potential for performance impact.
We should benchmark the impact and decide if the second option above is a
viable choice or if the first option is required.
Finally, we should introduce a higher level interface for managing the
registration of 'user' code for execution from the low level hooks. I filed
HBASE-11125 to discuss this further.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-11126:
---

Status: Open (was: Patch Available)

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.3, 0.99.0
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126_1.patch, HBASE-11126_new_2.patch,
HBASE-11126_new_3.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-11126:
---

Attachment: HBASE-11126.patch

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0, 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126.patch, HBASE-11126_1.patch,
HBASE-11126_new_2.patch, HBASE-11126_new_3.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-11126:
---

Status: Patch Available (was: Open)

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.98.3, 0.99.0
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126.patch, HBASE-11126_1.patch,
HBASE-11126_new_2.patch, HBASE-11126_new_3.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11252) Fixing new javadoc warnings in master branch


[ 
https://issues.apache.org/jira/browse/HBASE-11252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009468#comment-14009468
 ] 

Hudson commented on HBASE-11252:


SUCCESS: Integrated in HBase-TRUNK #5142 (See 
[https://builds.apache.org/job/HBase-TRUNK/5142/])
HBASE-11252 Fixing new javadoc warnings in master branch.(Anoop) (anoopsamjohn: 
rev 04b9a0a57376d6cae2b4fdc10a55dcedf5b983ad)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java


 Fixing new javadoc warnings in master branch
 

 Key: HBASE-11252
 URL: https://issues.apache.org/jira/browse/HBASE-11252
 Project: HBase
  Issue Type: Bug
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Priority: Minor
 Fix For: 0.99.0

 Attachments: HBASE-11252.patch


 {code}
 3 warnings
 [WARNING] Javadoc Warnings
 [WARNING] 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java:125:
  warning - Missing closing '}' character for inline tag: {@link 
 #ClientScanner(Configuration, Scan, TableName, HConnection, 
 RpcRetryingCallerFactory, RpcControllerFactory)
 [WARNING] instead.
 [WARNING] 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientSmallScanner.java:98:
  warning - Missing closing '}' character for inline tag: {@link 
 #ClientSmallScanner(Configuration, Scan, TableName, HConnection, 
 RpcRetryingCallerFactory, RpcControllerFactory)
 [WARNING] instead
 [WARNING] 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java:111:
  warning - Tag @link: can't find ScannerCallable(HConnection, TableName, 
 Scan, ScanMetrics) in org.apache.hadoop.hbase.client.ScannerCallable
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8970) [book] Filter language documentation is hidden


[ 
https://issues.apache.org/jira/browse/HBASE-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009476#comment-14009476
 ] 

Hadoop QA commented on HBASE-8970:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12646839/HBASE-8970.patch
  against trunk revision .
  ATTACHMENT ID: 12646839

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 31 release 
audit warnings (more than the trunk's current 0 warnings).

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+listitemparaThe unary operators codeSKIP/code and 
codeWHILE/code are next, and have the same
+paraThe client should use the symbols (lt;, lt;=, =, !=, gt;, 
gt;=) to express compare
+   $hbase = new HBase('server_name_running_thrift_server', port on which 
thrift server is running);
+   $result = $client-scannerOpenWithFilterString('table_name', (PrefixFilter 
('row2') AND (QualifierFilter (=, 'binary:xyz'))) AND (TimestampsFilter ( 123, 
456)));
+paracodeâPrefixFilter (âRowâ) AND PageFilter 
(1) AND FirstKeyOnlyFilter
+()â/code will return all key-value pairs 
that match the following
+paracodeâ(RowFilter (=, âbinary:Row 1â) 
AND TimeStampsFilter (74689,
+false))â/code will return all 
key-value pairs that match both
+paraThe key-value is in a row having row 
key âRow 1â /para
+paracodeâSKIP ValueFilter (0)â/code will 
skip the entire row if any of

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9597//console

This message is automatically generated.

 [book] Filter language documentation is hidden
 --

 Key: HBASE-8970
 URL: https://issues.apache.org/jira/browse/HBASE-8970
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
Priority: Minor
  Labels: noob
 Attachments: HBASE-8970.patch


 There is excellent documentation for how to use the Filter Language 
 (including examples!), implemented in HBASE-4176. Unfortunately, it's 
 [hidden|http://hbase.apache.org/book/thrift.html#thrift.filter-language] as 
 the only section in the otherwise empty Thrift chapter. I suggest it be moved 
 to it's own appendix and referenced

[jira] [Commented] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009484#comment-14009484
]

Hadoop QA commented on HBASE-11126:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646845/HBASE-11126.patch
against trunk revision .
ATTACHMENT ID: 12646845

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}. The applied patch generated 32 release
audit warnings (more than the trunk's current 0 warnings).

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9598//console

This message is automatically generated.

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0, 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126.patch, HBASE-11126_1.patch,
HBASE-11126_new_2.patch, HBASE-11126_new_3.patch

[jira] [Commented] (HBASE-6110) Fix TestInfoServers


[ 
https://issues.apache.org/jira/browse/HBASE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009575#comment-14009575
 ] 

Hudson commented on HBASE-6110:
---

FAILURE: Integrated in Hadoop-Yarn-trunk #565 (See 
[https://builds.apache.org/job/Hadoop-Yarn-trunk/565/])
HBASE-6110 adding more slow action log in critical write path (Liang Xie via 
stack) (stack: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1597633)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml


 Fix TestInfoServers
 ---

 Key: HBASE-6110
 URL: https://issues.apache.org/jira/browse/HBASE-6110
 Project: HBase
  Issue Type: Bug
  Components: build, master, regionserver
Affects Versions: 0.95.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: hbase-6110-v0.patch, hbase-6110-v1.patch, 
 hbase-6110-v2.patch, hbase-6110-v3.patch


 With the recent port to modules, we broke a couple of tests, including this 
 one. The fix needs to ensure that the webapp still works from the in-situ and 
 packaged running of HBase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise

[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

ramkrishna.s.vasudevan updated HBASE-11039:
---

Attachment: HBASE-11039_1.patch

This patch does not do what is described here, but uses the
IntegrationTestWithCellVisibilityLoadAndVerify to verify the deletes that
happens. May be we could make the deletes as configurable but in this patch
the puts that are added are tracked and in between few deletes are added to the
table. The counters are retrieved based on the number of deletes that were
performed while loading.

[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---

Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Priority: Critical
Fix For: 0.99.0, 0.98.4

Attachments: HBASE-11039_1.patch

Create an integration test for the VisibilityController that:
1. Create several tables of test data
2. Assign a set of auths to each table. Label all entries in the table with
appropriate visibility expressions. Insure that some data in every table
overlaps with data in other tables at common row/family/qualifier
coordinates. Generate data like ITBLL so we can verify all data present later.
3. Mix the data from the different tables into a new common table
4. Verify for each set of auths defined in step #2 that all entries found in
the source table can be found in the common table. Like the ITBLL
verification step but done N times for each set of auths defined in step #2.
5. Choose one of the source tables. Get its set of auths. Perform a deletion
with visibility expression from the common table using those auths.
6. Verify that no data in the common table with the auth set chosen in #5
remains. A simple row count with the set of auths chosen in #5 that should
return 0.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock

[
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009603#comment-14009603
]

ramkrishna.s.vasudevan commented on HBASE-11126:

This ready for commit. @apurtell?

Add RegionObserver pre hooks that operate under row lock

Key: HBASE-11126
URL: https://issues.apache.org/jira/browse/HBASE-11126
Project: HBase
Issue Type: Improvement
Affects Versions: 0.99.0, 0.98.3
Reporter: Andrew Purtell
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-11126.patch, HBASE-11126_1.patch,
HBASE-11126_new_2.patch, HBASE-11126_new_3.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise

[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009609#comment-14009609
]

ramkrishna.s.vasudevan commented on HBASE-11039:

To keep the intent of this JIRA unpolluted may be we can raise another JIRA and
attach the patch to that JIRA?

[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---

Attachments: HBASE-11039_1.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11232) Region fail to release the updatelock for illegal CF in multi row mutations


[ 
https://issues.apache.org/jira/browse/HBASE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009621#comment-14009621
 ] 

Liu Shaohui commented on HBASE-11232:
-

[~xieliang007]
0.96 and 0.98 branch don't have this issue.


 Region fail to release the updatelock for illegal CF in multi row mutations
 ---

 Key: HBASE-11232
 URL: https://issues.apache.org/jira/browse/HBASE-11232
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.19
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Attachments: HBASE-11232-0.94.diff


 The failback code in processRowsWithLocks did not check the column family. If 
 there is an illegal CF in the muation, it will  throw NullPointException and 
 the update lock will not be released.  So the region can not be flushed and 
 compacted. 
 HRegion #4946
 {code}
 if (!mutations.isEmpty()  !walSyncSuccessful) {
   LOG.warn(Wal sync failed. Roll back  + mutations.size() +
memstore keyvalues for row(s): +
   processor.getRowsToLock().iterator().next() + ...);
   for (KeyValue kv : mutations) {
 stores.get(kv.getFamily()).rollback(kv);
   }
 }
 // 11. Roll mvcc forward
 if (writeEntry != null) {
   mvcc.completeMemstoreInsert(writeEntry);
   writeEntry = null;
 }
 if (locked) {
   this.updatesLock.readLock().unlock();
   locked = false;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11232) Region fail to release the updatelock for illegal CF in multi row mutations


[ 
https://issues.apache.org/jira/browse/HBASE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009622#comment-14009622
 ] 

Hadoop QA commented on HBASE-11232:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12646175/HBASE-11232-0.94.diff
  against trunk revision .
  ATTACHMENT ID: 12646175

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 2 new 
or modified tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9599//console

This message is automatically generated.

 Region fail to release the updatelock for illegal CF in multi row mutations
 ---

 Key: HBASE-11232
 URL: https://issues.apache.org/jira/browse/HBASE-11232
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.19
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Attachments: HBASE-11232-0.94.diff


 The failback code in processRowsWithLocks did not check the column family. If 
 there is an illegal CF in the muation, it will  throw NullPointException and 
 the update lock will not be released.  So the region can not be flushed and 
 compacted. 
 HRegion #4946
 {code}
 if (!mutations.isEmpty()  !walSyncSuccessful) {
   LOG.warn(Wal sync failed. Roll back  + mutations.size() +
memstore keyvalues for row(s): +
   processor.getRowsToLock().iterator().next() + ...);
   for (KeyValue kv : mutations) {
 stores.get(kv.getFamily()).rollback(kv);
   }
 }
 // 11. Roll mvcc forward
 if (writeEntry != null) {
   mvcc.completeMemstoreInsert(writeEntry);
   writeEntry = null;
 }
 if (locked) {
   this.updatesLock.readLock().unlock();
   locked = false;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10596) Remove deprecated Filter#getNextKeyHint(KeyValue) method

2014-05-27 Thread Gustavo Anatoly (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009628#comment-14009628
 ] 

Gustavo Anatoly commented on HBASE-10596:
-

Thanks for review the patch.  I will fix it.

 Remove deprecated Filter#getNextKeyHint(KeyValue) method
 

 Key: HBASE-10596
 URL: https://issues.apache.org/jira/browse/HBASE-10596
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Trivial
 Attachments: HBASE-10596.patch


 Filter#getNextKeyHint(KeyValue) is deprecated.
 Filter#getNextCellHint(Cell) should be used instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10596) Remove deprecated Filter#getNextKeyHint(KeyValue) method

2014-05-27 Thread Gustavo Anatoly (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gustavo Anatoly updated HBASE-10596:


Attachment: HBASE-10596-v1.patch

 Remove deprecated Filter#getNextKeyHint(KeyValue) method
 

 Key: HBASE-10596
 URL: https://issues.apache.org/jira/browse/HBASE-10596
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Trivial
 Attachments: HBASE-10596-v1.patch, HBASE-10596.patch


 Filter#getNextKeyHint(KeyValue) is deprecated.
 Filter#getNextCellHint(Cell) should be used instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11255) Negative request num in region load

Liu Shaohui created HBASE-11255:
---

 Summary: Negative request num in region load
 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor


We found that the request number of region is negative in long-running hbase 
cluster.
The is because of improper cast in HRegionServer#createRegionLoad
{code}
... .setReadRequestsCount((int)r.readRequestsCount.get())
.setWriteRequestsCount((int) r.writeRequestsCount.get()) 
{code}

The patch is simple and just removes the cast.

 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11255) Negative request num in region load


 [ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-11255:


Affects Version/s: 0.99.0
   Status: Patch Available  (was: Open)

 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11255) Negative request num in region load


 [ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-11255:


Attachment: HBASE-11255-trunk-v1.diff

Patch for trunk

 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11255) Negative request num in region load


 [ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-11255:


Description: 
We found that the request number of region is negative in long-running hbase 
cluster.
The is because of improper cast in HRegionServer#createRegionLoad
{code}
  ...
  .setReadRequestsCount((int)r.readRequestsCount.get())
  .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
{code}

The patch is simple and just removes the cast.

 

  was:
We found that the request number of region is negative in long-running hbase 
cluster.
The is because of improper cast in HRegionServer#createRegionLoad
{code}
... .setReadRequestsCount((int)r.readRequestsCount.get())
.setWriteRequestsCount((int) r.writeRequestsCount.get()) 
{code}

The patch is simple and just removes the cast.

 


 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor

 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11215) Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)

2014-05-27 Thread Nicolas Liochon (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11215:


   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed, thanks for the patch, Carter.

 Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)
 -

 Key: HBASE-11215
 URL: https://issues.apache.org/jira/browse/HBASE-11215
 Project: HBase
  Issue Type: Bug
Reporter: Carter
Assignee: Carter
 Fix For: 0.99.0

 Attachments: HBASE_11215.patch


 The JavaDoc for HTableInterface#setAutoFlush(boolean autoFlush, boolean 
 clearBufferOnFail) says Setting clearBufferOnFail to false is deprecated 
 since 0.96.
 If that's the case, this method should be deprecated as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10596) Remove deprecated Filter#getNextKeyHint(KeyValue) method


[ 
https://issues.apache.org/jira/browse/HBASE-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009719#comment-14009719
 ] 

Hadoop QA commented on HBASE-10596:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12646906/HBASE-10596-v1.patch
  against trunk revision .
  ATTACHMENT ID: 12646906

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 32 release 
audit warnings (more than the trunk's current 0 warnings).

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9600//console

This message is automatically generated.

 Remove deprecated Filter#getNextKeyHint(KeyValue) method
 

 Key: HBASE-10596
 URL: https://issues.apache.org/jira/browse/HBASE-10596
 Project: HBase
  Issue Type: Task
Reporter: Ted Yu
Assignee: Gustavo Anatoly
Priority: Trivial
 Attachments: HBASE-10596-v1.patch, HBASE-10596.patch


 Filter#getNextKeyHint(KeyValue) is deprecated.
 Filter#getNextCellHint(Cell) should be used instead.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009736#comment-14009736
 ] 

Ted Yu commented on HBASE-11234:


Integrated addendum to trunk and 0.96.

Integrated to 0.94 and 0.98

Thanks

 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Resolved] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


 [ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu resolved HBASE-11234.


Resolution: Fixed

 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11255) Negative request num in region load


[ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009774#comment-14009774
 ] 

stack commented on HBASE-11255:
---

+1

 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-6110) Fix TestInfoServers


[ 
https://issues.apache.org/jira/browse/HBASE-6110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009779#comment-14009779
 ] 

Hudson commented on HBASE-6110:
---

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1783 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1783/])
HBASE-6110 adding more slow action log in critical write path (Liang Xie via 
stack) (stack: 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1597633)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockReceiver.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml


 Fix TestInfoServers
 ---

 Key: HBASE-6110
 URL: https://issues.apache.org/jira/browse/HBASE-6110
 Project: HBase
  Issue Type: Bug
  Components: build, master, regionserver
Affects Versions: 0.95.2
Reporter: Jesse Yates
Assignee: Jesse Yates
 Attachments: hbase-6110-v0.patch, hbase-6110-v1.patch, 
 hbase-6110-v2.patch, hbase-6110-v3.patch


 With the recent port to modules, we broke a couple of tests, including this 
 one. The fix needs to ensure that the webapp still works from the in-situ and 
 packaged running of HBase.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11255) Negative request num in region load


[ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009780#comment-14009780
 ] 

Hadoop QA commented on HBASE-11255:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12646908/HBASE-11255-trunk-v1.diff
  against trunk revision .
  ATTACHMENT ID: 12646908

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 32 release 
audit warnings (more than the trunk's current 0 warnings).

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   org.apache.hadoop.hbase.client.TestMultiParallel

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9601//console

This message is automatically generated.

 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11232) Region fail to release the updatelock for illegal CF in multi row mutations


[ 
https://issues.apache.org/jira/browse/HBASE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009787#comment-14009787
 ] 

stack commented on HBASE-11232:
---

Patch looks good.  Did you intend to add that big test class?  It seems to test 
more than this issue.

 Region fail to release the updatelock for illegal CF in multi row mutations
 ---

 Key: HBASE-11232
 URL: https://issues.apache.org/jira/browse/HBASE-11232
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.19
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Attachments: HBASE-11232-0.94.diff


 The failback code in processRowsWithLocks did not check the column family. If 
 there is an illegal CF in the muation, it will  throw NullPointException and 
 the update lock will not be released.  So the region can not be flushed and 
 compacted. 
 HRegion #4946
 {code}
 if (!mutations.isEmpty()  !walSyncSuccessful) {
   LOG.warn(Wal sync failed. Roll back  + mutations.size() +
memstore keyvalues for row(s): +
   processor.getRowsToLock().iterator().next() + ...);
   for (KeyValue kv : mutations) {
 stores.get(kv.getFamily()).rollback(kv);
   }
 }
 // 11. Roll mvcc forward
 if (writeEntry != null) {
   mvcc.completeMemstoreInsert(writeEntry);
   writeEntry = null;
 }
 if (locked) {
   this.updatesLock.readLock().unlock();
   locked = false;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise

[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Andrew Purtell reassigned HBASE-11039:
--

Assignee: (was: ramkrishna.s.vasudevan)

[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---

Key: HBASE-11039
URL: https://issues.apache.org/jira/browse/HBASE-11039
Project: HBase
Issue Type: Test
Affects Versions: 0.98.1
Reporter: Andrew Purtell
Priority: Critical
Fix For: 0.99.0, 0.98.4

Attachments: HBASE-11039_1.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11039) [VisibilityController] Integration test for labeled data set mixing and filtered excise

[
https://issues.apache.org/jira/browse/HBASE-11039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009817#comment-14009817
]

Andrew Purtell commented on HBASE-11039:

Why attach a patch here that doesn't satisfy what is required by the
description. -1

[VisibilityController] Integration test for labeled data set mixing and
filtered excise
---

Attachments: HBASE-11039_1.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-8970) [book] Filter language documentation is hidden


 [ 
https://issues.apache.org/jira/browse/HBASE-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8970:
-

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to master.  Thanks for the patch [~misty]  I tried it and there may 
be a few formatting issues that could be addressed in a subsequent patch but 
otherwise, looks great.  Thanks.

 [book] Filter language documentation is hidden
 --

 Key: HBASE-8970
 URL: https://issues.apache.org/jira/browse/HBASE-8970
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
Priority: Minor
  Labels: noob
 Fix For: 0.99.0

 Attachments: HBASE-8970.patch


 There is excellent documentation for how to use the Filter Language 
 (including examples!), implemented in HBASE-4176. Unfortunately, it's 
 [hidden|http://hbase.apache.org/book/thrift.html#thrift.filter-language] as 
 the only section in the otherwise empty Thrift chapter. I suggest it be moved 
 to it's own appendix and referenced by all the places that expose it 
 (currently just Thrift and the Shell, eventually REST -- pending completion 
 of HBASE-5417).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11255) Negative request num in region load


 [ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-11255:
--

   Resolution: Fixed
Fix Version/s: 0.98.4
   0.96.3
   0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Applied to master and 0.98, 0.96.  Patch does not seem applicable to 0.94. 
Thanks for the patch [~liushaohui]

 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.96.3, 0.98.4

 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (HBASE-11256) [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour


 [ 
https://issues.apache.org/jira/browse/HBASE-11256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan reassigned HBASE-11256:
--

Assignee: ramkrishna.s.vasudevan

 [Visibility Controller] Enhance 
 IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour
 -

 Key: HBASE-11256
 URL: https://issues.apache.org/jira/browse/HBASE-11256
 Project: HBase
  Issue Type: Test
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-11039_1.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11256) [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour


 [ 
https://issues.apache.org/jira/browse/HBASE-11256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-11256:
---

Attachment: HBASE-11039_1.patch

 [Visibility Controller] Enhance 
 IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour
 -

 Key: HBASE-11256
 URL: https://issues.apache.org/jira/browse/HBASE-11256
 Project: HBase
  Issue Type: Test
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-11039_1.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11256) [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour

ramkrishna.s.vasudevan created HBASE-11256:
--

 Summary: [Visibility Controller] Enhance 
IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour
 Key: HBASE-11256
 URL: https://issues.apache.org/jira/browse/HBASE-11256
 Project: HBase
  Issue Type: Test
Reporter: ramkrishna.s.vasudevan
 Attachments: HBASE-11039_1.patch





--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11256) [Visibility Controller] Enhance IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour


[ 
https://issues.apache.org/jira/browse/HBASE-11256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009845#comment-14009845
 ] 

ramkrishna.s.vasudevan commented on HBASE-11256:


IntegrationTestWithCellVisibilityLoadAndVerify to verify the deletes that 
happens.  May be we could make the deletes as configurable but in this patch 
the puts that are added are tracked and in between few deletes are added to the 
table.  The counters are retrieved based on the number of deletes that were 
performed while loading.

 [Visibility Controller] Enhance 
 IntegrationTestWithCellVisibilityLoadAndVerify to verify delete behaviour
 -

 Key: HBASE-11256
 URL: https://issues.apache.org/jira/browse/HBASE-11256
 Project: HBase
  Issue Type: Test
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Attachments: HBASE-11039_1.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009848#comment-14009848
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in HBase-0.94-security #486 (See 
[https://builds.apache.org/job/HBase-0.94-security/486/])
HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result 
(Chunhui) (tedyu: rev a616bb42c651ed7ab79f25c55d8f6baaea585676)
* src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11248) KeyOnlyKeyValue#toString() passes wrong offset to keyToString()


 [ 
https://issues.apache.org/jira/browse/HBASE-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ramkrishna.s.vasudevan updated HBASE-11248:
---

Attachment: HBASE-11248.patch

 KeyOnlyKeyValue#toString() passes wrong offset to keyToString()
 ---

 Key: HBASE-11248
 URL: https://issues.apache.org/jira/browse/HBASE-11248
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Attachments: HBASE-11248.patch


 When debugging for HBASE-11234, I found that debugger encountered exception 
 displaying the string value of KeyOnlyKeyValue.
 It turns out that KeyOnlyKeyValue#toString() passes wrong offset to 
 keyToString(): ROW_OFFSET should not be used.
 Another bug is that getValueLength() shouldn't be called - it throws 
 IllegalArgumentException.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11248) KeyOnlyKeyValue#toString() passes wrong offset to keyToString()


[ 
https://issues.apache.org/jira/browse/HBASE-11248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009860#comment-14009860
 ] 

Ted Yu commented on HBASE-11248:


lgtm
nit: 
'/vlen=' can be combined with the other constant strings.

 KeyOnlyKeyValue#toString() passes wrong offset to keyToString()
 ---

 Key: HBASE-11248
 URL: https://issues.apache.org/jira/browse/HBASE-11248
 Project: HBase
  Issue Type: Bug
Reporter: Ted Yu
Assignee: ramkrishna.s.vasudevan
Priority: Minor
 Attachments: HBASE-11248.patch


 When debugging for HBASE-11234, I found that debugger encountered exception 
 displaying the string value of KeyOnlyKeyValue.
 It turns out that KeyOnlyKeyValue#toString() passes wrong offset to 
 keyToString(): ROW_OFFSET should not be used.
 Another bug is that getValueLength() shouldn't be called - it throws 
 IllegalArgumentException.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11126) Add RegionObserver pre hooks that operate under row lock


[ 
https://issues.apache.org/jira/browse/HBASE-11126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009866#comment-14009866
 ] 

Andrew Purtell commented on HBASE-11126:


Patch is looking good.

{code}
@@ -587,6 +587,25 @@ public interface RegionObserver extends Coprocessor {
   void preDelete(final ObserverContextRegionCoprocessorEnvironment c, 
   final Delete delete, final WALEdit edit, final Durability durability)
 throws IOException;
+/**
+ * Called before the client updates the timestamp for delete Columns with 
latest timestamp.
+ * p
+ * Call CoprocessorEnvironment#bypass to skip default actions
+ * p
+ * Call CoprocessorEnvironment#complete to skip any subsequent chained
+ * coprocessors
+ * @param c the environment provided by the region server
+ * @param mutation - the parent mutation associated with this delete cell
+ * @param cell - The deleteColumn with latest version cell
+ * @param byteNow - timestamp bytes
+ * @param family - family bytes
+ * @param get - the get formed using the current cell's row.
+ * Note that the get does not specify the family and qualifier
+ * @throws IOException
+ */
+  void preGetForDeleteVersion(final 
ObserverContextRegionCoprocessorEnvironment c,
+  final Mutation mutation, final Cell cell, final byte[] byteNow, final 
byte[] family,
+  final Get get) throws IOException;
{code}

preGetForDeleteVersion needs a better name. What are you doing with this new 
hook? 

In the Javadoc did you mean Called before the *server* ... 

{code}
@@ -5064,12 +5090,18 @@ public class HRegion implements HeapSize { // , 
Writable{
   rowLock = getRowLock(row);
   try {
 lock(this.updatesLock.readLock());
-// wait for all prior MVCC transactions to finish - while we hold the 
row lock
-// (so that we are guaranteed to see the latest state)
-mvcc.completeMemstoreInsert(mvcc.beginMemstoreInsert());
-// now start my own transaction
-w = mvcc.beginMemstoreInsert();
 try {
+  // wait for all prior MVCC transactions to finish - while we hold 
the row lock
+  // (so that we are guaranteed to see the latest state)
+  mvcc.completeMemstoreInsert(mvcc.beginMemstoreInsert());
+  if (this.coprocessorHost != null) {
+Result r = this.coprocessorHost.preAppendAfterRowLock(append);
+if(r!= null) {
+  return r;
+}
+  }
+  // now start my own transaction
+  w = mvcc.beginMemstoreInsert();
   long now = EnvironmentEdgeManager.currentTimeMillis();
   // Process each family
   for (Map.Entrybyte[], ListCell family : 
append.getFamilyCellMap().entrySet()) {
{code}

Why are you moving completeMemstoreInsert and beginMemstoreInsert into the 
inner try? 

{code}
@@ -5254,12 +5286,18 @@ public class HRegion implements HeapSize { // , 
Writable{
   RowLock rowLock = getRowLock(row);
   try {
 lock(this.updatesLock.readLock());
-// wait for all prior MVCC transactions to finish - while we hold the 
row lock
-// (so that we are guaranteed to see the latest state)
-mvcc.completeMemstoreInsert(mvcc.beginMemstoreInsert());
-// now start my own transaction
-w = mvcc.beginMemstoreInsert();
 try {
+  // wait for all prior MVCC transactions to finish - while we hold 
the row lock
+  // (so that we are guaranteed to see the latest state)
+  mvcc.completeMemstoreInsert(mvcc.beginMemstoreInsert());
+  if (this.coprocessorHost != null) {
+Result r = 
this.coprocessorHost.preIncrementAfterRowLock(increment);
+if (r != null) {
+  return r;
+}
+  }
+  // now start my own transaction
+  w = mvcc.beginMemstoreInsert();
   long now = EnvironmentEdgeManager.currentTimeMillis();
   // Process each family
   for (Map.Entrybyte [], ListCell family:
{code}

Ditto.

{code}
diff --git 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
index 2e7b022..e0363b2 100644
--- 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
+++ 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
@@ -1730,7 +1730,7 @@ public class RSRpcServices implements 
HBaseRPCErrorHandler,
 ProtobufUtil.toComparator(condition.getComparator());
   if (region.getCoprocessorHost() != null) {
 processed = region.getCoprocessorHost().preCheckAndPut(
-  row, family, qualifier, compareOp, comparator, put);
+row, family, qualifier, compareOp, comparator, put);
   }
   if (processed == null) {
 boolean result =

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009874#comment-14009874
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in HBase-0.94-JDK7 #138 (See 
[https://builds.apache.org/job/HBase-0.94-JDK7/138/])
HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result 
(Chunhui) (tedyu: rev a616bb42c651ed7ab79f25c55d8f6baaea585676)
* src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11118) non environment variable solution for IllegalAccessError: class com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass com.google.protobuf.Literal


[ 
https://issues.apache.org/jira/browse/HBASE-8?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009887#comment-14009887
 ] 

Andrew Purtell commented on HBASE-8:


Comparing against master: 
https://github.com/apurtell/hbase/compare/master...HBASE-8
Comparing against 0.98: 
https://github.com/apurtell/hbase/compare/0.98...HBASE-8-0.98

 non environment variable solution for IllegalAccessError: class 
 com.google.protobuf.ZeroCopyLiteralByteString cannot access its superclass 
 com.google.protobuf.LiteralByteString
 --

 Key: HBASE-8
 URL: https://issues.apache.org/jira/browse/HBASE-8
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.2
Reporter: André Kelpe
Priority: Blocker
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-8-0.98.patch.gz, HBASE-8-trunk.patch.gz, 
 shade_attempt.patch


 I am running into the problem described in 
 https://issues.apache.org/jira/browse/HBASE-10304, while trying to use a 
 newer version within cascading.hbase 
 (https://github.com/cascading/cascading.hbase).
 One of the features of cascading.hbase is that you can use it from lingual 
 (http://www.cascading.org/projects/lingual/), our SQL layer for hadoop. 
 lingual has a notion of providers, which are fat jars that we pull down 
 dynamically at runtime. Those jars give users the ability to talk to any 
 system or format from SQL. They are added to the classpath  programmatically 
 before we submit jobs to a hadoop cluster.
 Since lingual does not know upfront , which providers are going to be used in 
 a given run, the HADOOP_CLASSPATH trick proposed in the JIRA above is really 
 clunky and breaks the ease of use we had before. No other provider requires 
 this right now.
 It would be great to have a programmatical way to fix this, when using fat 
 jars.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11257) [0.89-fb] Remove the timestamp from the annotation of Put

2014-05-27 Thread Gaurav Menghani (JIRA)

Gaurav Menghani created HBASE-11257:
---

 Summary: [0.89-fb] Remove the timestamp from the annotation of Put
 Key: HBASE-11257
 URL: https://issues.apache.org/jira/browse/HBASE-11257
 Project: HBase
  Issue Type: Improvement
Reporter: Gaurav Menghani
Assignee: Gaurav Menghani
Priority: Minor


David and I, recently found out, when discussing the C++ client with @csliu 
from the Search Team, that we have an extraneous timestamp field in the Put. 
Actually, that field is used when we create a Put object like:

Put p = new Put(row, ts);
And then, if you do:

p.add(cf, qualifier, value);
it would use ts as a timestamp for the KeyValue for the cf, qualifier. If you 
did not specify it, it will use HConstants.LATEST_TIMESTAMP.

One can also do this, where you explicitly state the timestamp to be used:

p.add(cf, qualifier, ts, value)
In either case, when the add() method is called, the KeyValue is constructed, 
and it has the proper timestamp. Therefore, once you have created the family 
map with all these KeyValues, you don't need to send the ts field provided 
during the construction. All the KVs will have the correct timestamp embedded 
by the time it will be sent across.

This diff removes the timestamp field from the Put object. This will save us 
some network bandwidth, hopefully :)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked


 [ 
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-11096:
---

Fix Version/s: 0.94.21

Committed to 0.94.

 stop method of Master and RegionServer coprocessor  is not invoked
 --

 Key: HBASE-11096
 URL: https://issues.apache.org/jira/browse/HBASE-11096
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.2, 0.98.1, 0.94.19
Reporter: Qiang Tian
Assignee: Qiang Tian
 Fix For: 0.99.0, 0.98.3, 0.94.21

 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, 
 HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, 
 HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, 
 HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.1.patch, 
 HBASE-11096-trunk-v3.patch, HBASE-11096-trunk-v3.patch, 
 TestCoprocessorStop-failed-output.txt


 the stop method of coprocessor specified by 
 hbase.coprocessor.master.classes and  
 hbase.coprocessor.regionserver.classes  is not invoked.
 If coprocessor allocates OS resources, it could cause master/regionserver 
 resource leak or hang during exit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11215) Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)

2014-05-27 Thread Nicolas Liochon (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicolas Liochon updated HBASE-11215:


  Component/s: Client
 Priority: Minor  (was: Major)
Affects Version/s: 0.99.0

 Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)
 -

 Key: HBASE-11215
 URL: https://issues.apache.org/jira/browse/HBASE-11215
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 0.99.0
Reporter: Carter
Assignee: Carter
Priority: Minor
 Fix For: 0.99.0

 Attachments: HBASE_11215.patch


 The JavaDoc for HTableInterface#setAutoFlush(boolean autoFlush, boolean 
 clearBufferOnFail) says Setting clearBufferOnFail to false is deprecated 
 since 0.96.
 If that's the case, this method should be deprecated as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10692) The Multi TableMap job don't support the security HBase cluster


 [ 
https://issues.apache.org/jira/browse/HBASE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-10692:
---

Fix Version/s: 0.98.4
   0.94.21
   0.99.0

Going to commit shortly to trunk, 0.98, and 0.94 unless objection

 The Multi TableMap job don't support the security HBase cluster
 ---

 Key: HBASE-10692
 URL: https://issues.apache.org/jira/browse/HBASE-10692
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.94.21, 0.98.4

 Attachments: HBASE-10692-0.94-v1.diff, HBASE-10692-trunk-v1.diff, 
 HBASE-10692-trunk-v2.diff


 HBASE-3996 adds the support of multiple tables and scanners as input to the 
 mapper in map/reduce jobs. But it don't support the security HBase cluster.
 [~erank] [~bbaugher]
 Ps: HBASE-3996 only support multiple tables from the same HBase cluster. 
 Should we support multiple tables from the different clusters?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009907#comment-14009907
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in HBase-0.94 #1370 (See 
[https://builds.apache.org/job/HBase-0.94/1370/])
HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result 
(Chunhui) (tedyu: rev a616bb42c651ed7ab79f25c55d8f6baaea585676)
* src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-8817) Enhance The Apache HBase Reference Guide

[
https://issues.apache.org/jira/browse/HBASE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-8817:
-

Attachment: HBASE-8817v2.patch

Here is what I applied. Changed '40% of heap' in the hbase-default.xml to 0.4.

Patch did not apply. Did you rebase [~misty] before making your patch?

I did the fixup.

Doc looks great.

There is a bunch of scan do spread all over the ref guide. We can collect it
into one location some day. For now, this is a nice addition (I gave it an id
of hregion.scan).

Enhance The Apache HBase Reference Guide

Key: HBASE-8817
URL: https://issues.apache.org/jira/browse/HBASE-8817
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Thomas Pan
Assignee: Misty Stanley-Jones
Priority: Critical
Attachments: HBASE-8817.patch, HBASE-8817v2.patch

I am reading the guide here: http://hbase.apache.org/book.html#regions.arch
I have noticed the following:
- There is lack of coverage on scan. Note that Lars has a very good
presentation from last year's HBaseCon:
http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final
- Under section 9.7.6.1, there is no mentioning of flush per region if a
memstore reached the size threshold (and a possible flush of all memstores if
over all memory usage of all the memstores reached memstore.upperlimit). Note
that people need to combine the information provided in section 9 with
section 2.3.1.1 to fully understand the situation.
- Compaction tuning strategies. Both Nicolas from Facebook and Sergey from
Hortonworks have presented various compaction algorithms. Sometime, once all
the necessary code is in master, we need to have the coverage.
Let me know how I can help on the matter.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-8817) Enhance The Apache HBase Reference Guide


 [ 
https://issues.apache.org/jira/browse/HBASE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-8817:
-

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Applied to trunk.  Thanks for the nice doc improvements [~misty]

 Enhance The Apache HBase Reference Guide
 

 Key: HBASE-8817
 URL: https://issues.apache.org/jira/browse/HBASE-8817
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Thomas Pan
Assignee: Misty Stanley-Jones
Priority: Critical
 Fix For: 0.99.0

 Attachments: HBASE-8817.patch, HBASE-8817v2.patch


 I am reading the guide here: http://hbase.apache.org/book.html#regions.arch
 I have noticed the following:
 - There is lack of coverage on scan. Note that Lars has a very good 
 presentation from last year's HBaseCon: 
 http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final
 - Under section 9.7.6.1, there is no mentioning of flush per region if a 
 memstore reached the size threshold (and a possible flush of all memstores if 
 over all memory usage of all the memstores reached memstore.upperlimit). Note 
 that people need to combine the information provided in section 9 with 
 section 2.3.1.1 to fully understand the situation.
 - Compaction tuning strategies. Both Nicolas from Facebook and Sergey from 
 Hortonworks have presented various compaction algorithms. Sometime, once all 
 the necessary code is in master, we need to have the coverage.
 Let me know how I can help on the matter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11215) Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)


[ 
https://issues.apache.org/jira/browse/HBASE-11215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009916#comment-14009916
 ] 

Hudson commented on HBASE-11215:


FAILURE: Integrated in HBase-TRUNK #5143 (See 
[https://builds.apache.org/job/HBase-TRUNK/5143/])
HBASE-11215 Deprecate void setAutoFlush(boolean autoFlush, boolean 
clearBufferOnFail) (nkeywal: rev f00d21179c76739ce8480c89fa9ad6b968fc6b7a)
* hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTableInterface.java


 Deprecate void setAutoFlush(boolean autoFlush, boolean clearBufferOnFail)
 -

 Key: HBASE-11215
 URL: https://issues.apache.org/jira/browse/HBASE-11215
 Project: HBase
  Issue Type: Bug
  Components: Client
Affects Versions: 0.99.0
Reporter: Carter
Assignee: Carter
Priority: Minor
 Fix For: 0.99.0

 Attachments: HBASE_11215.patch


 The JavaDoc for HTableInterface#setAutoFlush(boolean autoFlush, boolean 
 clearBufferOnFail) says Setting clearBufferOnFail to false is deprecated 
 since 0.96.
 If that's the case, this method should be deprecated as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8807) HBase MapReduce Job-Launch Documentation Misplaced


[ 
https://issues.apache.org/jira/browse/HBASE-8807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009918#comment-14009918
 ] 

stack commented on HBASE-8807:
--

Can you rebase and remake the patch [~misty] It doesn't apply as is.

This kind of operational doc probably belongs best yeah in the refguide.  The 
javadoc package-info was built up before the refguide was around and so we just 
kept on.  I can change the javadoc to point back into the refguide as the 
authoritative source.

 HBase MapReduce Job-Launch Documentation Misplaced
 --

 Key: HBASE-8807
 URL: https://issues.apache.org/jira/browse/HBASE-8807
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Affects Versions: 0.94.8
Reporter: Ambud Sharma
Assignee: Misty Stanley-Jones
 Attachments: HBASE-8807.patch


 Documentation on steps to properly launch a mapreduce job on an HBase + 
 Hadoop cluster is misplaced and is located in Javadocs: 
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath
  
 This must be extracted and placed in a separate page with rest of HBase MR 
 guide.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11094) Distributed log replay is incompatible for rolling restarts

[
https://issues.apache.org/jira/browse/HBASE-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009922#comment-14009922
]

Hadoop QA commented on HBASE-11094:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12646940/hbase-11094-v3.patch
against trunk revision .
ATTACHMENT ID: 12646940

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 13 new
or modified tests.

{color:red}-1 patch{color}. The patch command could not apply the patch.

Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9603//console

This message is automatically generated.

Distributed log replay is incompatible for rolling restarts
---

Key: HBASE-11094
URL: https://issues.apache.org/jira/browse/HBASE-11094
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Blocker
Fix For: 0.99.0

Attachments: hbase-11094-v2.patch, hbase-11094-v3.patch,
hbase-11094.patch

0.99.0 comes with dist log replay by default (HBASE-10888). However, reading
the code and discussing this with Jeffrey, we realized that the dist log
replay code is not compatible with rolling upgrades from 0.98.0 and 1.0.0.
The issue is that, the region server looks at it own configuration to decide
whether the region should be opened in replay mode or not. The open region
RPC does not contain that info. So if dist log replay is enabled on master,
the master will assign the region and schedule replay tasks. If the region is
opened in a RS that does not have this conf enabled, then it will happily
open the region in normal mode (not replay mode) causing possible (transient)
data loss.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer

2014-05-27 Thread Eric Charles (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009935#comment-14009935
 ] 

Eric Charles commented on HBASE-10336:
--

Thx [~tedyu] for your review.
[~stack], [~apurtell],... any other comment on it?

 Remove deprecated usage of Hadoop HttpServer in InfoServer
 --

 Key: HBASE-10336
 URL: https://issues.apache.org/jira/browse/HBASE-10336
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Eric Charles
Assignee: Eric Charles
 Attachments: 10336-v10.txt, HBASE-10336-1.patch, HBASE-10336-2.patch, 
 HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch, 
 HBASE-10336-6.patch, HBASE-10336-7.patch, HBASE-10336-8.patch, 
 HBASE-10336-9.patch, HBASE-10569-10.patch


 Recent changes in Hadoop HttpServer give NPE when running on hadoop 
 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be 
 not fixed (see HDFS-5760). We'd better move to the new proposed builder 
 pattern, which means we can no more use inheritance to build our nice 
 InfoServer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11232) Region fail to release the updatelock for illegal CF in multi row mutations

2014-05-27 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009934#comment-14009934
 ] 

Lars Hofhansl commented on HBASE-11232:
---

+1
Looks like we should also guard the unlock and the mvcc completion in another 
try finally block:
{code}
try {
   ... rollback memstore ...
} finally {
if (w != null) {
  mvcc.completeMemstoreInsert(w);
}
if (locked) {
  this.updatesLock.readLock().unlock();
}
if (acquiredLocks != null) {
  for (Integer lid : acquiredLocks) {
releaseRowLock(lid);
  }
}
}
{code}

 Region fail to release the updatelock for illegal CF in multi row mutations
 ---

 Key: HBASE-11232
 URL: https://issues.apache.org/jira/browse/HBASE-11232
 Project: HBase
  Issue Type: Bug
  Components: regionserver
Affects Versions: 0.94.19
Reporter: Liu Shaohui
Assignee: Liu Shaohui
 Attachments: HBASE-11232-0.94.diff


 The failback code in processRowsWithLocks did not check the column family. If 
 there is an illegal CF in the muation, it will  throw NullPointException and 
 the update lock will not be released.  So the region can not be flushed and 
 compacted. 
 HRegion #4946
 {code}
 if (!mutations.isEmpty()  !walSyncSuccessful) {
   LOG.warn(Wal sync failed. Roll back  + mutations.size() +
memstore keyvalues for row(s): +
   processor.getRowsToLock().iterator().next() + ...);
   for (KeyValue kv : mutations) {
 stores.get(kv.getFamily()).rollback(kv);
   }
 }
 // 11. Roll mvcc forward
 if (writeEntry != null) {
   mvcc.completeMemstoreInsert(writeEntry);
   writeEntry = null;
 }
 if (locked) {
   this.updatesLock.readLock().unlock();
   locked = false;
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10336) Remove deprecated usage of Hadoop HttpServer in InfoServer

2014-05-27 Thread Eric Charles (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009936#comment-14009936
 ] 

Eric Charles commented on HBASE-10336:
--

[~stack], [~apurtell]... it being https://reviews.apache.org/r/21705/
thx, Eric

 Remove deprecated usage of Hadoop HttpServer in InfoServer
 --

 Key: HBASE-10336
 URL: https://issues.apache.org/jira/browse/HBASE-10336
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Eric Charles
Assignee: Eric Charles
 Attachments: 10336-v10.txt, HBASE-10336-1.patch, HBASE-10336-2.patch, 
 HBASE-10336-3.patch, HBASE-10336-4.patch, HBASE-10336-5.patch, 
 HBASE-10336-6.patch, HBASE-10336-7.patch, HBASE-10336-8.patch, 
 HBASE-10336-9.patch, HBASE-10569-10.patch


 Recent changes in Hadoop HttpServer give NPE when running on hadoop 
 3.0.0-SNAPSHOT. This way we use HttpServer is deprecated and will probably be 
 not fixed (see HDFS-5760). We'd better move to the new proposed builder 
 pattern, which means we can no more use inheritance to build our nice 
 InfoServer.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-9149) javadoc cleanup of to reflect .META. rename to hbase:meta


 [ 
https://issues.apache.org/jira/browse/HBASE-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-9149:
-

Attachment: HBASE-9149v2.patch

Minor changes.  What I committed.

 javadoc cleanup of to reflect .META. rename to hbase:meta
 -

 Key: HBASE-9149
 URL: https://issues.apache.org/jira/browse/HBASE-9149
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Reporter: Francis Liu
Assignee: Misty Stanley-Jones
Priority: Critical
 Fix For: 0.99.0

 Attachments: HBASE-9149.patch, HBASE-9149v2.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-9149) javadoc cleanup of to reflect .META. rename to hbase:meta


 [ 
https://issues.apache.org/jira/browse/HBASE-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-9149:
-

   Resolution: Fixed
Fix Version/s: 0.99.0
 Hadoop Flags: Reviewed
   Status: Resolved  (was: Patch Available)

Committed to trunk.  Thanks [~misty]

 javadoc cleanup of to reflect .META. rename to hbase:meta
 -

 Key: HBASE-9149
 URL: https://issues.apache.org/jira/browse/HBASE-9149
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Reporter: Francis Liu
Assignee: Misty Stanley-Jones
Priority: Critical
 Fix For: 0.99.0

 Attachments: HBASE-9149.patch, HBASE-9149v2.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009941#comment-14009941
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #297 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/297/])
HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result 
(tedyu: rev 90ece25b6e480f51b05ff7330e59bdcc637e5c45)
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11220) Add listeners to ServerManager and AssignmentManager


[ 
https://issues.apache.org/jira/browse/HBASE-11220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009948#comment-14009948
 ] 

stack commented on HBASE-11220:
---

[~mbertozzi] Should it be regionOpened and regionClosed rather than 
regionLoaded and regionUnloaded?  Or onlined and offlined (that is what you 
call it in your distribution method).  Otherwise +1



 Add listeners to ServerManager and AssignmentManager
 

 Key: HBASE-11220
 URL: https://issues.apache.org/jira/browse/HBASE-11220
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-11220-v0.patch


 Add support for listeners to ServerManager and AssignmentManager.
 This will allows to get notified about servers added/removed or regions 
 added/removed/moved.
 I'm planning to use this in the MasterProcedureManager. Since we are starting 
 using the Procedures for distributed operations, we must add support for RS 
 joining or Regions moving. At the moment the operation on the moving set of 
 RSs is lost.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009964#comment-14009964
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in HBase-0.98 #316 (See 
[https://builds.apache.org/job/HBase-0.98/316/])
HBASE-11234 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result 
(tedyu: rev 90ece25b6e480f51b05ff7330e59bdcc637e5c45)
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
* 
hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileScanner.java
* 
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11250) Document: CLONE - Fix jersey serialization/deserialization of json objects


[ 
https://issues.apache.org/jira/browse/HBASE-11250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009969#comment-14009969
 ] 

stack commented on HBASE-11250:
---

Add youself with your real name -- e.g. Misty or MistyStanleyJones -- rather 
than mstan...@cloudera.com and I will then add you to this group, 
https://wiki.apache.org/hadoop/ContributorsGroup, so you can edit.

 Document: CLONE - Fix jersey serialization/deserialization of json objects
 --

 Key: HBASE-11250
 URL: https://issues.apache.org/jira/browse/HBASE-11250
 Project: HBase
  Issue Type: Bug
  Components: documentation, REST
Affects Versions: 0.92.2
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Blocker

 Stargate uses the default json marshaller/unmarshaller in natural mode. In 
 this mode the unmarshaller has trouble unmarshalling json instances. 
 This patch fixes this issue by using jackson as the marshaller/unmarshaller 
 instead. 
 I've also updated all the model unit tests to test json 
 serialization/deserialization. Backwards compatibilty can be verified by 
 modify the test base class to use the original marshaller/unmarshaller and 
 see that model tests pass.
 The patch is backward compatible except for StorageClusterStatusModel, which 
 is broken anyway. It only shows one node in the liveNodes field.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009974#comment-14009974
 ] 

Hudson commented on HBASE-11234:


FAILURE: Integrated in HBase-TRUNK #5144 (See 
[https://builds.apache.org/job/HBase-TRUNK/5144/])
HBASE-11234 Addendum for PrefixTreeArrayScanner (tedyu: rev 
56af657e5df8f17358c80c79d4aafa806a0f57b7)
* 
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8970) [book] Filter language documentation is hidden


[ 
https://issues.apache.org/jira/browse/HBASE-8970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009976#comment-14009976
 ] 

Hudson commented on HBASE-8970:
---

FAILURE: Integrated in HBase-TRUNK #5144 (See 
[https://builds.apache.org/job/HBase-TRUNK/5144/])
HBASE-8970 [book] Filter language documentation is hidden (Misty Stanley-Jones) 
(stack: rev 856892cf57c2b244a45e2b0b898e57078bdee749)
* src/main/docbkx/thrift_filter_language.xml
* src/main/docbkx/book.xml
* src/main/docbkx/external_apis.xml


 [book] Filter language documentation is hidden
 --

 Key: HBASE-8970
 URL: https://issues.apache.org/jira/browse/HBASE-8970
 Project: HBase
  Issue Type: Improvement
  Components: documentation
Reporter: Nick Dimiduk
Assignee: Misty Stanley-Jones
Priority: Minor
  Labels: noob
 Fix For: 0.99.0

 Attachments: HBASE-8970.patch


 There is excellent documentation for how to use the Filter Language 
 (including examples!), implemented in HBASE-4176. Unfortunately, it's 
 [hidden|http://hbase.apache.org/book/thrift.html#thrift.filter-language] as 
 the only section in the otherwise empty Thrift chapter. I suggest it be moved 
 to it's own appendix and referenced by all the places that expose it 
 (currently just Thrift and the Shell, eventually REST -- pending completion 
 of HBASE-5417).



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-8817) Enhance The Apache HBase Reference Guide


[ 
https://issues.apache.org/jira/browse/HBASE-8817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009978#comment-14009978
 ] 

Hudson commented on HBASE-8817:
---

FAILURE: Integrated in HBase-TRUNK #5144 (See 
[https://builds.apache.org/job/HBase-TRUNK/5144/])
HBASE-8817 Enhance The Apache HBase Reference Guide (Misty Stanley-Jones) 
(stack: rev bfbc6d48cd783ce0ea9a43fded0999839fc5d6c5)
* hbase-common/src/main/resources/hbase-default.xml
* src/main/docbkx/book.xml


 Enhance The Apache HBase Reference Guide
 

 Key: HBASE-8817
 URL: https://issues.apache.org/jira/browse/HBASE-8817
 Project: HBase
  Issue Type: Bug
  Components: documentation
Reporter: Thomas Pan
Assignee: Misty Stanley-Jones
Priority: Critical
 Fix For: 0.99.0

 Attachments: HBASE-8817.patch, HBASE-8817v2.patch


 I am reading the guide here: http://hbase.apache.org/book.html#regions.arch
 I have noticed the following:
 - There is lack of coverage on scan. Note that Lars has a very good 
 presentation from last year's HBaseCon: 
 http://www.slideshare.net/cloudera/3-learning-h-base-internals-lars-hofhansl-salesforce-final
 - Under section 9.7.6.1, there is no mentioning of flush per region if a 
 memstore reached the size threshold (and a possible flush of all memstores if 
 over all memory usage of all the memstores reached memstore.upperlimit). Note 
 that people need to combine the information provided in section 9 with 
 section 2.3.1.1 to fully understand the situation.
 - Compaction tuning strategies. Both Nicolas from Facebook and Sergey from 
 Hortonworks have presented various compaction algorithms. Sometime, once all 
 the necessary code is in master, we need to have the coverage.
 Let me know how I can help on the matter.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9149) javadoc cleanup of to reflect .META. rename to hbase:meta


[ 
https://issues.apache.org/jira/browse/HBASE-9149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009977#comment-14009977
 ] 

Hudson commented on HBASE-9149:
---

FAILURE: Integrated in HBase-TRUNK #5144 (See 
[https://builds.apache.org/job/HBase-TRUNK/5144/])
HBASE-9149 javadoc cleanup of to reflect .META. rename to hbase:meta (Misty 
Stanley-Jones) (stack: rev aa101ca97e8394e4ba293ae637cd0ca193040576)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/TruncateTableHandler.java
* src/main/site/xdoc/replication.xml
* src/main/docbkx/ops_mgt.xml
* conf/log4j.properties
* bin/region_status.rb
* bin/region_mover.rb
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java
* src/main/docbkx/troubleshooting.xml
* src/main/docbkx/book.xml


 javadoc cleanup of to reflect .META. rename to hbase:meta
 -

 Key: HBASE-9149
 URL: https://issues.apache.org/jira/browse/HBASE-9149
 Project: HBase
  Issue Type: Sub-task
  Components: documentation
Reporter: Francis Liu
Assignee: Misty Stanley-Jones
Priority: Critical
 Fix For: 0.99.0

 Attachments: HBASE-9149.patch, HBASE-9149v2.patch






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11255) Negative request num in region load


[ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009975#comment-14009975
 ] 

Hudson commented on HBASE-11255:


FAILURE: Integrated in HBase-TRUNK #5144 (See 
[https://builds.apache.org/job/HBase-TRUNK/5144/])
HBASE-11255 Negative request num in region load (Liu Shaohui) (stack: rev 
00d550ca13d7451f5a8db666f0d4cf311b1a1856)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.96.3, 0.98.4

 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag


[ 
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009979#comment-14009979
 ] 

Hadoop QA commented on HBASE-9531:
--

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  
http://issues.apache.org/jira/secure/attachment/12646812/HBASE-9531-trunk-v0.patch
  against trunk revision .
  ATTACHMENT ID: 12646812

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 3 new 
or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}.  The applied patch generated 31 release 
audit warnings (more than the trunk's current 0 warnings).

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+  result = result  (hasReplicationLoadSourceString() == 
other.hasReplicationLoadSourceString());
+  new java.lang.String[] { NumberOfRequests, 
TotalNumberOfRequests, UsedHeapMB, MaxHeapMB, RegionLoads, 
Coprocessors, ReportStartTime, ReportEndTime, InfoServerPort, 
ReplicationLoadSourceString, ReplicationLoadSinkString, });
+rsource = 
org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad::REPLICATIONLOADSOURCE
+rsink = 
org.apache.hadoop.hbase.replication.regionserver.ReplicationLoad::REPLICATIONLOADSINK

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//testReport/
Release audit warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/9602//console

This message is automatically generated.

 a command line (hbase shell) interface to retreive the replication metrics 
 and show replication lag
 ---

 Key: HBASE-9531
 URL: https://issues.apache.org/jira/browse/HBASE-9531
 Project: HBase
  Issue Type: New Feature
  Components: Replication
Affects Versions: 0.99.0
Reporter: Demai Ni
Assignee: Demai Ni
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-9531-trunk-v0.patch, HBASE-9531-trunk-v0.patch


 This jira is to provide a command line (hbase shell) interface to retreive 
 the replication metrics info such as:ageOfLastShippedOp, 
 timeStampsOfLastShippedOp, sizeOfLogQueue ageOfLastAppliedOp, and 
 timeStampsOfLastAppliedOp. And also to provide a point of time info of the 
 lag of replication(source only)
 Understand that hbase is using Hadoop 
 metrics(http://hbase.apache.org/metrics.html), which is a common way to 
 monitor metric info. This Jira is to serve as a light-weight client 
 interface, comparing to a completed(certainly better, but heavier)GUI 
 monitoring package. I made the code works on 0.94.9 now, and like to use this 
 jira to get opinions about whether the feature is valuable to other 
 users/workshop. If so, I

[jira] [Updated] (HBASE-11059) ZK-less region assignment


 [ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jimmy Xiang updated HBASE-11059:


Attachment: hbase-11059.patch

Attached the first patch, which is also on RB: 
https://reviews.apache.org/r/21934/

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009982#comment-14009982
 ] 

Jimmy Xiang edited comment on HBASE-11059 at 5/27/14 5:55 PM:
--

The patch is backward compatible, rolling upgradable. The performance depends 
on the setup and the number of regions of course. On my 6 nodes cluster, 
assigning 13,990 user regions, it took about 8minutes using ZK, and ~1minute 
with this patch (no ZK).


was (Author: jxiang):
The patch is backward compatible, rolling upgradable. The performance depends 
on the setup and the number of regions of course. On my 6 nodes cluster, 
assigning 13,990 user regions, it took about 8minutes using ZK, and ~1minute 
with this patch.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11218) Data loss in HBase standalone mode

[
https://issues.apache.org/jira/browse/HBASE-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009987#comment-14009987
]

stack commented on HBASE-11218:
---

[~liushaohui] Could you do your same trick on the RS side too if local fs (as
Enis suggests above)?

Thanks for doing the test compares. I will commit this unless objection later
today. We can look into fixing pseudo on local fs in a new issue?

Data loss in HBase standalone mode
--

Key: HBASE-11218
URL: https://issues.apache.org/jira/browse/HBASE-11218
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Fix For: 0.99.0

Attachments: HBASE-11218-trunk-v1.diff

Data loss in HBase standalone mode.
*How to produce it*
# Start HBase standalone mode.
# Create a table using hbase shell.
# Scan '.META.' and you will find data in meta table
# Kill the HBase process with -9 option
# Start the HBase agaion
# Scan '.META.' and you will find nothing in meta table.
*There are three main reasons.*
# FSDataOutputStream.sync should call flush() if the underlying wrapped
stream is not Syncable. See HADOOP-8861
# writeChecksum is ture in default LocalFileSystem and the
ChecksumFSOutputSummer will buffer the data, which make the waledits are not
written to os's filesystem with sync method immediately, and those edits will
be lost in regionserver's failover.
# The MiniZooKeeperCluster deletes the old zk data at startup which maye
cause data loss in meta table. The failover procedure is: split pre root
regionserver's hlog - assign root - split pre meta regionserver's hlog -
assign meta - split all other regionservers' hlogs - assign other regions.
If there is no data in zookeeper, we will get null for root regionserver and
then assign root table. Some data in root table maybe be lost for some root's
WalEdits have not been splited and replayed. So does the Meta table.
I finished the patch for 0.94 and am working on the patch for trunk.
Suggestions are welcomed.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14009982#comment-14009982
 ] 

Jimmy Xiang commented on HBASE-11059:
-

The patch is backward compatible, rolling upgradable. The performance depends 
on the setup and the number of regions of course. On my 6 nodes cluster, 
assigning 13,990 user regions, it took about 8minutes using ZK, and ~1minute 
with this patch.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-10692) The Multi TableMap job don't support the security HBase cluster


 [ 
https://issues.apache.org/jira/browse/HBASE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Purtell updated HBASE-10692:
---

  Resolution: Fixed
Hadoop Flags: Reviewed
  Status: Resolved  (was: Patch Available)

Committed

 The Multi TableMap job don't support the security HBase cluster
 ---

 Key: HBASE-10692
 URL: https://issues.apache.org/jira/browse/HBASE-10692
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.94.21, 0.98.4

 Attachments: HBASE-10692-0.94-v1.diff, HBASE-10692-trunk-v1.diff, 
 HBASE-10692-trunk-v2.diff


 HBASE-3996 adds the support of multiple tables and scanners as input to the 
 mapper in map/reduce jobs. But it don't support the security HBase cluster.
 [~erank] [~bbaugher]
 Ps: HBASE-3996 only support multiple tables from the same HBase cluster. 
 Should we support multiple tables from the different clusters?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11238) Add info about SlabCache and BucketCache to Ref Guide

[
https://issues.apache.org/jira/browse/HBASE-11238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010019#comment-14010019
]

stack commented on HBASE-11238:
---

I buy your argument that this better belongs in the refguide than in javadoc.

The features depends so much on code reading, having the doc in javadoc makes
it easier linking the code internals to the explanation. The doc is still
under development and configuration of this feature is likely to change in near
future too.

If we add doc to refguide, we need to change the package-info so it points into
refguide instead so we have doc in one place only.

In following, it should be fetching from BucketCache or SlabCache will be
slower than from native on heap LruBlockCache Fetching
+ will always be slower when fetching from BucketCache, but latencies
tend to be less
+ erratic over time, because there is less garbage collection.

Also experience seems to have it that BucketCache makes less garbage than
SlabCache so should be even less erratic (than SlabCache or LruBlockCache). No
hard numbers, just reports by a few of the lads.

Change '(CombinedBlockCache)' to '(DoubleBlockCache vs CombinedBlockCache)'

You lost the link to Nick's 'BlockCache 101'?

This is not right SlabCache is enabled by default. (and you contradict
yourself by writing later 'To enable it...) There are three block cache
options. The original implemenation which is in the java heap called
LruBlockCache. Then there are SlabCache and BucketCache, both of which are
mainly intended for keeping blockcache data offheap (but they can also keep
data onheap and in files, at least in the BucketCache case).

We should say what hbase.offheapcache.percentage is a multiplicand for (the JVM
setting). oh you say it later so there is some repeat going on here?

Otherwise, looks great.

Add info about SlabCache and BucketCache to Ref Guide
-

Key: HBASE-11238
URL: https://issues.apache.org/jira/browse/HBASE-11238
Project: HBase
Issue Type: Bug
Components: documentation
Affects Versions: 0.98.2
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Attachments: HBASE-11238.patch

Upstream issues: HBASE-11171 and HBASE-11098. Could back port some of what is
in these issues, the package-info.java class for instance.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11218) Data loss in HBase standalone mode

2014-05-27 Thread Enis Soztutar (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010023#comment-14010023
]

Enis Soztutar commented on HBASE-11218:
---

bq. I think the data loss in pseudo distributed mode caused by hardware failure
is accepted, data loss caused by the HBase's implement is not accepted and we
should fix it.
I think there is a misunderstanding here. I was referring to the same local
file system issue. The fix in the patch will only fix it for local mode
deployments, but not pseudo-distributed mode deployments.

Data loss in HBase standalone mode
--

Key: HBASE-11218
URL: https://issues.apache.org/jira/browse/HBASE-11218
Project: HBase
Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Fix For: 0.99.0

Attachments: HBASE-11218-trunk-v1.diff

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010037#comment-14010037
 ] 

stack commented on HBASE-11059:
---

[~jxiang] What if you did 1M regions?



 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11201) Enable global procedure members to return values to procedure master

2014-05-27 Thread Jerry He (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010046#comment-14010046
 ] 

Jerry He commented on HBASE-11201:
--

Ping [~tedyu] and [~mbertozzi]
Do you have any additional comment, concern?

 Enable global procedure members to return values to procedure master
 

 Key: HBASE-11201
 URL: https://issues.apache.org/jira/browse/HBASE-11201
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 0.99.0

 Attachments: HBASE-11201-trunk-v1.patch, HBASE-11201-trunk-v2.patch


 Currently in the global procedure framework, the procedure coordinator can 
 send data (procedure argument) to the members when starting procedure.
 But we don't support getting data returned from the procedure members back to 
 the master.
 Similar to RPC and normal procedure/function calls, in many cases, this is a 
 useful capability.
 The infrastructure is in place. We just need to plug in the holes and make it 
 happen.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11234) FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result


[ 
https://issues.apache.org/jira/browse/HBASE-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010047#comment-14010047
 ] 

Hudson commented on HBASE-11234:


SUCCESS: Integrated in hbase-0.96 #402 (See 
[https://builds.apache.org/job/hbase-0.96/402/])
HBASE-11234 Addendum for PrefixTreeArrayScanner (tedyu: rev 
191abb0b6b12670203feb13ecaf7aae20c307959)
* 
hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java


 FastDiffDeltaEncoder#getFirstKeyInBlock returns wrong result
 

 Key: HBASE-11234
 URL: https://issues.apache.org/jira/browse/HBASE-11234
 Project: HBase
  Issue Type: Bug
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
 Fix For: 0.99.0, 0.96.3, 0.94.21, 0.98.4

 Attachments: 11234-94.patch, 11234-96.patch, 
 11234-98-with-prefix-tree.txt, 11234-trunk.addendum, HBASE-11234.patch


 As Ted found, 
 With this change:
 {noformat}
 Index: 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
 ===
 --- 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(revision 1596579)
 +++ 
 hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestReversibleScanners.java
(working copy)
 @@ -51,6 +51,7 @@
  import org.apache.hadoop.hbase.filter.FilterList.Operator;
  import org.apache.hadoop.hbase.filter.PageFilter;
  import org.apache.hadoop.hbase.filter.SingleColumnValueFilter;
 +import org.apache.hadoop.hbase.io.encoding.DataBlockEncoding;
  import org.apache.hadoop.hbase.io.hfile.CacheConfig;
  import org.apache.hadoop.hbase.io.hfile.HFileContext;
  import org.apache.hadoop.hbase.io.hfile.HFileContextBuilder;
 @@ -90,6 +91,7 @@
  CacheConfig cacheConf = new CacheConfig(TEST_UTIL.getConfiguration());
  HFileContextBuilder hcBuilder = new HFileContextBuilder();
  hcBuilder.withBlockSize(2 * 1024);
 +hcBuilder.withDataBlockEncoding(DataBlockEncoding.FAST_DIFF);
  HFileContext hFileContext = hcBuilder.build();
  StoreFile.Writer writer = new StoreFile.WriterBuilder(
  TEST_UTIL.getConfiguration(), cacheConf, fs).withOutputDir(
 {noformat}
 I got:
 java.lang.AssertionError: 
 expected:testRow0197/testCf:testQual/1400712260004/Put/vlen=13/mvcc=5 
 but was:testRow0198/testCf:testQual/1400712260004/   Put/vlen=13/mvcc=0
   at org.junit.Assert.fail(Assert.java:88)
   at org.junit.Assert.failNotEquals(Assert.java:743)
   at org.junit.Assert.assertEquals(Assert.java:118)
   at org.junit.Assert.assertEquals(Assert.java:144)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.seekTestOfReversibleKeyValueScanner(TestReversibleScanners.java:533)
   at 
 org.apache.hadoop.hbase.regionserver.TestReversibleScanners.testReversibleStoreFileScanner(TestReversibleScanners.java:108)
 After debugging, it seems the method of 
 FastDiffDeltaEncoder#getFirstKeyInBlock become broken. And it will cause 
 hfilescanner#seekBefore returns wrong result.
 The solution is simple, see the patch.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment

2014-05-27 Thread Mikhail Antonov (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010051#comment-14010051
 ] 

Mikhail Antonov commented on HBASE-11059:
-

Great! [~jxiang] - will take a look today.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment

2014-05-27 Thread Matteo Bertozzi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010058#comment-14010058
 ] 

Matteo Bertozzi commented on HBASE-11059:
-

[~jxiang] is this patch just writing the state to meta instead of zookeeper 
without changing the assignment algorithm at all?

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag


[ 
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010064#comment-14010064
 ] 

Demai Ni commented on HBASE-9531:
-

'The patch introduces the following lines longer than 100' comes from two 
places: 
first two are from ClusterStatusProtos.java, which is generated
the 2nd two are from admin.rb, where there are a lot of lines longer than 100, 
probably for the ruby coding-style(?) 

do we need to keep the lines under 100 in both cases? 

Demai

 a command line (hbase shell) interface to retreive the replication metrics 
 and show replication lag
 ---

 Key: HBASE-9531
 URL: https://issues.apache.org/jira/browse/HBASE-9531
 Project: HBase
  Issue Type: New Feature
  Components: Replication
Affects Versions: 0.99.0
Reporter: Demai Ni
Assignee: Demai Ni
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-9531-trunk-v0.patch, HBASE-9531-trunk-v0.patch


 This jira is to provide a command line (hbase shell) interface to retreive 
 the replication metrics info such as:ageOfLastShippedOp, 
 timeStampsOfLastShippedOp, sizeOfLogQueue ageOfLastAppliedOp, and 
 timeStampsOfLastAppliedOp. And also to provide a point of time info of the 
 lag of replication(source only)
 Understand that hbase is using Hadoop 
 metrics(http://hbase.apache.org/metrics.html), which is a common way to 
 monitor metric info. This Jira is to serve as a light-weight client 
 interface, comparing to a completed(certainly better, but heavier)GUI 
 monitoring package. I made the code works on 0.94.9 now, and like to use this 
 jira to get opinions about whether the feature is valuable to other 
 users/workshop. If so, I will build a trunk patch. 
 All inputs are greatly appreciated. Thank you!
 The overall design is to reuse the existing logic which supports hbase shell 
 command 'status', and invent a new module, called ReplicationLoad.  In 
 HRegionServer.buildServerLoad() , use the local replication service objects 
 to get their loads  which could be wrapped in a ReplicationLoad object and 
 then simply pass it to the ServerLoad. In ReplicationSourceMetrics and 
 ReplicationSinkMetrics, a few getters and setters will be created, and ask 
 Replication to build a ReplicationLoad.  (many thanks to Jean-Daniel for 
 his kindly suggestions through dev email list)
 the replication lag will be calculated for source only, and use this formula: 
 {code:title=Replication lag|borderStyle=solid}
   if sizeOfLogQueue != 0 then max(ageOfLastShippedOp, (current time - 
 timeStampsOfLastShippedOp)) //err on the large side
   else if (current time - timeStampsOfLastShippedOp)  2* 
 ageOfLastShippedOp then lag = ageOfLastShippedOp // last shipped happen 
 recently 
   else lag = 0 // last shipped may happens last night, so NO real 
 lag although ageOfLastShippedOp is non-zero
 {code}
 External will look something like:
 {code:title=status 'replication'|borderStyle=solid}
 hbase(main):001:0 status 'replication'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:50:59 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
 hbase(main):002:0 status 'replication','source'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
 hbase(main):003:0 status 'replication','sink'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
     hdtest018.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:50:59 PDT 2013
     hdtest015.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04

[jira] [Commented] (HBASE-11201) Enable global procedure members to return values to procedure master

2014-05-27 Thread Matteo Bertozzi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010061#comment-14010061
 ] 

Matteo Bertozzi commented on HBASE-11201:
-

+1

 Enable global procedure members to return values to procedure master
 

 Key: HBASE-11201
 URL: https://issues.apache.org/jira/browse/HBASE-11201
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 0.99.0

 Attachments: HBASE-11201-trunk-v1.patch, HBASE-11201-trunk-v2.patch


 Currently in the global procedure framework, the procedure coordinator can 
 send data (procedure argument) to the members when starting procedure.
 But we don't support getting data returned from the procedure members back to 
 the master.
 Similar to RPC and normal procedure/function calls, in many cases, this is a 
 useful capability.
 The infrastructure is in place. We just need to plug in the holes and make it 
 happen.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11201) Enable global procedure members to return values to procedure master


[ 
https://issues.apache.org/jira/browse/HBASE-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010071#comment-14010071
 ] 

Ted Yu commented on HBASE-11201:


Good by me.

 Enable global procedure members to return values to procedure master
 

 Key: HBASE-11201
 URL: https://issues.apache.org/jira/browse/HBASE-11201
 Project: HBase
  Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
 Fix For: 0.99.0

 Attachments: HBASE-11201-trunk-v1.patch, HBASE-11201-trunk-v2.patch


 Currently in the global procedure framework, the procedure coordinator can 
 send data (procedure argument) to the members when starting procedure.
 But we don't support getting data returned from the procedure members back to 
 the master.
 Similar to RPC and normal procedure/function calls, in many cases, this is a 
 useful capability.
 The infrastructure is in place. We just need to plug in the holes and make it 
 happen.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked


[ 
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010072#comment-14010072
 ] 

Hudson commented on HBASE-11096:


SUCCESS: Integrated in HBase-0.94-security #487 (See 
[https://builds.apache.org/job/HBase-0.94-security/487/])
HBASE-11096 stop method of Master and RegionServer coprocessor is not invoked 
(Qiang Tian) (apurtell: rev 6ed3d0770202a497e9884d0347efa46121dc2808)
* src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java
* src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java


 stop method of Master and RegionServer coprocessor  is not invoked
 --

 Key: HBASE-11096
 URL: https://issues.apache.org/jira/browse/HBASE-11096
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.2, 0.98.1, 0.94.19
Reporter: Qiang Tian
Assignee: Qiang Tian
 Fix For: 0.99.0, 0.98.3, 0.94.21

 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, 
 HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, 
 HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, 
 HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.1.patch, 
 HBASE-11096-trunk-v3.patch, HBASE-11096-trunk-v3.patch, 
 TestCoprocessorStop-failed-output.txt


 the stop method of coprocessor specified by 
 hbase.coprocessor.master.classes and  
 hbase.coprocessor.regionserver.classes  is not invoked.
 If coprocessor allocates OS resources, it could cause master/regionserver 
 resource leak or hang during exit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010081#comment-14010081
 ] 

Jimmy Xiang commented on HBASE-11059:
-

The old assignment logic is still there for rolling upgrade. If the new logic 
is turned on, we will store states to meta, and regionserver reports region 
transitions to master via RPC instead of ZK. There are still room for 
improvement, for example, report aggregation, skip some heart-beat, etc.. 
Client side logic is not changed very much for compatibility.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag


[ 
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010084#comment-14010084
 ] 

Demai Ni commented on HBASE-9531:
-

about the release audit warning(31 of them), this file 
(https://builds.apache.org/job/PreCommit-HBASE-Build/9602//artifact/patchprocess/patchReleaseAuditWarnings.txt)
 pointes the problems for all the files under patchprocess, such as: 
{quote}
31 Unknown Licenses

***

Unapproved licenses:

  patchprocess/newPatchFindbugsWarningshbase-examples.xml
  patchprocess/newPatchFindbugsWarningshbase-thrift.xml
  patchprocess/patchFindbugsWarningshbase-prefix-tree.xml
  patchprocess/newPatchFindbugsWarningshbase-server.xml
  patchprocess/patchFindbugsWarningshbase-hadoop-compat.xml
  patchprocess/patchFindbugsWarningshbase-common.xml
  patchprocess/newPatchFindbugsWarningshbase-thrift.html

{quote}

probably a side-effect when switching to git? 

 a command line (hbase shell) interface to retreive the replication metrics 
 and show replication lag
 ---

 Key: HBASE-9531
 URL: https://issues.apache.org/jira/browse/HBASE-9531
 Project: HBase
  Issue Type: New Feature
  Components: Replication
Affects Versions: 0.99.0
Reporter: Demai Ni
Assignee: Demai Ni
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-9531-trunk-v0.patch, HBASE-9531-trunk-v0.patch


 This jira is to provide a command line (hbase shell) interface to retreive 
 the replication metrics info such as:ageOfLastShippedOp, 
 timeStampsOfLastShippedOp, sizeOfLogQueue ageOfLastAppliedOp, and 
 timeStampsOfLastAppliedOp. And also to provide a point of time info of the 
 lag of replication(source only)
 Understand that hbase is using Hadoop 
 metrics(http://hbase.apache.org/metrics.html), which is a common way to 
 monitor metric info. This Jira is to serve as a light-weight client 
 interface, comparing to a completed(certainly better, but heavier)GUI 
 monitoring package. I made the code works on 0.94.9 now, and like to use this 
 jira to get opinions about whether the feature is valuable to other 
 users/workshop. If so, I will build a trunk patch. 
 All inputs are greatly appreciated. Thank you!
 The overall design is to reuse the existing logic which supports hbase shell 
 command 'status', and invent a new module, called ReplicationLoad.  In 
 HRegionServer.buildServerLoad() , use the local replication service objects 
 to get their loads  which could be wrapped in a ReplicationLoad object and 
 then simply pass it to the ServerLoad. In ReplicationSourceMetrics and 
 ReplicationSinkMetrics, a few getters and setters will be created, and ask 
 Replication to build a ReplicationLoad.  (many thanks to Jean-Daniel for 
 his kindly suggestions through dev email list)
 the replication lag will be calculated for source only, and use this formula: 
 {code:title=Replication lag|borderStyle=solid}
   if sizeOfLogQueue != 0 then max(ageOfLastShippedOp, (current time - 
 timeStampsOfLastShippedOp)) //err on the large side
   else if (current time - timeStampsOfLastShippedOp)  2* 
 ageOfLastShippedOp then lag = ageOfLastShippedOp // last shipped happen 
 recently 
   else lag = 0 // last shipped may happens last night, so NO real 
 lag although ageOfLastShippedOp is non-zero
 {code}
 External will look something like:
 {code:title=status 'replication'|borderStyle=solid}
 hbase(main):001:0 status 'replication'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:50:59 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
 hbase(main):002:0 status 'replication','source'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04

[jira] [Updated] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag


 [ 
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Demai Ni updated HBASE-9531:


Description: 
This jira is to provide a command line (hbase shell) interface to retreive the 
replication metrics info such as:ageOfLastShippedOp, timeStampsOfLastShippedOp, 
sizeOfLogQueue ageOfLastAppliedOp, and timeStampsOfLastAppliedOp. And also to 
provide a point of time info of the lag of replication(source only)

Understand that hbase is using Hadoop 
metrics(http://hbase.apache.org/metrics.html), which is a common way to monitor 
metric info. This Jira is to serve as a light-weight client interface, 
comparing to a completed(certainly better, but heavier)GUI monitoring package. 
I made the code works on 0.94.9 now, and like to use this jira to get opinions 
about whether the feature is valuable to other users/workshop. If so, I will 
build a trunk patch. 

All inputs are greatly appreciated. Thank you!

The overall design is to reuse the existing logic which supports hbase shell 
command 'status', and invent a new module, called ReplicationLoad.  In 
HRegionServer.buildServerLoad() , use the local replication service objects to 
get their loads  which could be wrapped in a ReplicationLoad object and then 
simply pass it to the ServerLoad. In ReplicationSourceMetrics and 
ReplicationSinkMetrics, a few getters and setters will be created, and ask 
Replication to build a ReplicationLoad.  (many thanks to Jean-Daniel for his 
kindly suggestions through dev email list)

the replication lag will be calculated for source only, and use this formula: 
{code:title=Replication lag|borderStyle=solid}
if sizeOfLogQueue != 0 then max(ageOfLastShippedOp, (current time - 
timeStampsOfLastShippedOp)) //err on the large side
else if (current time - timeStampsOfLastShippedOp)  2* 
ageOfLastShippedOp then lag = ageOfLastShippedOp // last shipped happen 
recently 
else lag = 0 // last shipped may happens last night, so NO real lag 
although ageOfLastShippedOp is non-zero
{code}

External will look something like:
{code:title=status 'replication'|borderStyle=solid}
hbase(main):001:0 status 'replication'
version 0.94.9
3 live servers
    hdtest017.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
    SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:48:48 PDT 2013
    hdtest018.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
    SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:50:59 PDT 2013
    hdtest015.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
    SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:48:48 PDT 2013

hbase(main):002:0 status 'replication','source'
version 0.94.9
3 live servers
    hdtest017.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
    hdtest018.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
    hdtest015.svl.ibm.com:
    SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013

hbase(main):003:0 status 'replication','sink'
version 0.94.9
3 live servers
    hdtest017.svl.ibm.com:
    SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:48:48 PDT 2013
    hdtest018.svl.ibm.com:
    SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:50:59 PDT 2013
    hdtest015.svl.ibm.com:
    SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
14:48:48 PDT 2013

hbase(main):003:0 status 'replication','lag' 
version 0.94.9
3 live servers
    hdtest017.svl.ibm.com: lag = 0
    hdtest018.svl.ibm.com: lag = 14
    hdtest015.svl.ibm.com: lag = 0
{code}



  was:
This jira is to provide a command line (hbase shell) interface to retreive the 
replication metrics info such as:ageOfLastShippedOp, timeStampsOfLastShippedOp, 
sizeOfLogQueue ageOfLastAppliedOp, and timeStampsOfLastAppliedOp. And also to 
provide a point of time info of the lag of replication(source only)

Understand that hbase is using Hadoop 
metrics(http://hbase.apache.org/metrics.html), which is a common way to monitor 
metric info. This Jira is to serve as a light-weight client interface, 
comparing to a completed(certainly better, but heavier)GUI monitoring package. 
I made the code works on 0.94.9 now, and like to use this jira to get opinions 
about whether the feature is valuable to other users/workshop. If so, I will 
build a trunk patch. 

All inputs are greatly appreciated. Thank you!

The overall design is to reuse

[jira] [Commented] (HBASE-11059) ZK-less region assignment


[ 
https://issues.apache.org/jira/browse/HBASE-11059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010092#comment-14010092
 ] 

Jimmy Xiang commented on HBASE-11059:
-

By the way, I am going to try 1M regions now. I need a bigger cluster.

 ZK-less region assignment
 -

 Key: HBASE-11059
 URL: https://issues.apache.org/jira/browse/HBASE-11059
 Project: HBase
  Issue Type: Improvement
  Components: master, Region Assignment
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
 Attachments: hbase-11059.patch, zk-less_am.pdf


 It seems that most people don't like region assignment with ZK (HBASE-5487), 
 which causes many uncertainties. This jira is to support ZK-less region 
 assignment. We need to make sure this patch doesn't break backward 
 compatibility/rolling upgrade.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked


[ 
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010150#comment-14010150
 ] 

Hudson commented on HBASE-11096:


FAILURE: Integrated in HBase-0.94-JDK7 #139 (See 
[https://builds.apache.org/job/HBase-0.94-JDK7/139/])
HBASE-11096 stop method of Master and RegionServer coprocessor is not invoked 
(Qiang Tian) (apurtell: rev 6ed3d0770202a497e9884d0347efa46121dc2808)
* 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
* src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java
* src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 stop method of Master and RegionServer coprocessor  is not invoked
 --

 Key: HBASE-11096
 URL: https://issues.apache.org/jira/browse/HBASE-11096
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.2, 0.98.1, 0.94.19
Reporter: Qiang Tian
Assignee: Qiang Tian
 Fix For: 0.99.0, 0.98.3, 0.94.21

 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, 
 HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, 
 HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, 
 HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.1.patch, 
 HBASE-11096-trunk-v3.patch, HBASE-11096-trunk-v3.patch, 
 TestCoprocessorStop-failed-output.txt


 the stop method of coprocessor specified by 
 hbase.coprocessor.master.classes and  
 hbase.coprocessor.regionserver.classes  is not invoked.
 If coprocessor allocates OS resources, it could cause master/regionserver 
 resource leak or hang during exit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11096) stop method of Master and RegionServer coprocessor is not invoked


[ 
https://issues.apache.org/jira/browse/HBASE-11096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010176#comment-14010176
 ] 

Hudson commented on HBASE-11096:


SUCCESS: Integrated in HBase-0.94 #1371 (See 
[https://builds.apache.org/job/HBase-0.94/1371/])
HBASE-11096 stop method of Master and RegionServer coprocessor is not invoked 
(Qiang Tian) (apurtell: rev 6ed3d0770202a497e9884d0347efa46121dc2808)
* 
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerCoprocessorHost.java
* src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
* src/main/java/org/apache/hadoop/hbase/master/MasterCoprocessorHost.java
* src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorStop.java
* src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


 stop method of Master and RegionServer coprocessor  is not invoked
 --

 Key: HBASE-11096
 URL: https://issues.apache.org/jira/browse/HBASE-11096
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.96.2, 0.98.1, 0.94.19
Reporter: Qiang Tian
Assignee: Qiang Tian
 Fix For: 0.99.0, 0.98.3, 0.94.21

 Attachments: HBASE-11096-0.94.patch, HBASE-11096-0.96.patch, 
 HBASE-11096-0.98.patch, HBASE-11096-trunk-v0.patch, 
 HBASE-11096-trunk-v0.patch, HBASE-11096-trunk-v1.patch, 
 HBASE-11096-trunk-v2.patch, HBASE-11096-trunk-v3.1.patch, 
 HBASE-11096-trunk-v3.patch, HBASE-11096-trunk-v3.patch, 
 TestCoprocessorStop-failed-output.txt


 the stop method of coprocessor specified by 
 hbase.coprocessor.master.classes and  
 hbase.coprocessor.regionserver.classes  is not invoked.
 If coprocessor allocates OS resources, it could cause master/regionserver 
 resource leak or hang during exit.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11094) Distributed log replay is incompatible for rolling restarts

[
https://issues.apache.org/jira/browse/HBASE-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010180#comment-14010180
]

stack commented on HBASE-11094:
---

I took a quick looks. Seems good to me.

How to proceed? Commit this with fat release note and a note in the 1.0 doc
that we need to include in the upgrade doc the steps this issues requires
rolling upgrading and then discuss on dev in 1.0 thread if we should turn off
distributed log replay for 1.0?

Distributed log replay is incompatible for rolling restarts
---

Key: HBASE-11094
URL: https://issues.apache.org/jira/browse/HBASE-11094
Project: HBase
Issue Type: Sub-task
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Blocker
Fix For: 0.99.0

Attachments: hbase-11094-v2.patch, hbase-11094-v3.patch,
hbase-11094.patch

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11201) Enable global procedure members to return values to procedure master

[
https://issues.apache.org/jira/browse/HBASE-11201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010207#comment-14010207
]

Hadoop QA commented on HBASE-11201:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12645945/HBASE-11201-trunk-v2.patch
against trunk revision .
ATTACHMENT ID: 12645945

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.

{color:red}-1 release audit{color}. The applied patch generated 32 release
audit warnings (more than the trunk's current 0 warnings).

{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+throw new IOException(Failed to get data from finished node
or data is illegally formatted:
+assertEquals(Incorrect number of members returnd data, expected.size(),
dataFromMembers.size());

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9604//console

This message is automatically generated.

Enable global procedure members to return values to procedure master

Key: HBASE-11201
URL: https://issues.apache.org/jira/browse/HBASE-11201
Project: HBase
Issue Type: Improvement
Reporter: Jerry He
Assignee: Jerry He
Fix For: 0.99.0

Attachments: HBASE-11201-trunk-v1.patch, HBASE-11201-trunk-v2.patch

Currently in the global procedure framework, the procedure coordinator can
send data (procedure argument) to the members when starting procedure.
But we don't support getting data returned from the procedure members back to
the master.
Similar to RPC and normal procedure/function calls, in many cases, this is a
useful capability.
The infrastructure is in place. We just need to plug in the holes and make it
happen.

--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11220) Add listeners to ServerManager and AssignmentManager

2014-05-27 Thread Matteo Bertozzi (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matteo Bertozzi updated HBASE-11220:


Attachment: HBASE-11220-v1.patch

renamed to opened/closed

 Add listeners to ServerManager and AssignmentManager
 

 Key: HBASE-11220
 URL: https://issues.apache.org/jira/browse/HBASE-11220
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-11220-v0.patch, HBASE-11220-v1.patch


 Add support for listeners to ServerManager and AssignmentManager.
 This will allows to get notified about servers added/removed or regions 
 added/removed/moved.
 I'm planning to use this in the MasterProcedureManager. Since we are starting 
 using the Procedures for distributed operations, we must add support for RS 
 joining or Regions moving. At the moment the operation on the moving set of 
 RSs is lost.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HBASE-11258) [0.89-fb] Pull in Integration Tests from open source 0.96/trunk

2014-05-27 Thread Elliott Clark (JIRA)

Elliott Clark created HBASE-11258:
-

 Summary: [0.89-fb] Pull in Integration Tests from open source 
0.96/trunk
 Key: HBASE-11258
 URL: https://issues.apache.org/jira/browse/HBASE-11258
 Project: HBase
  Issue Type: Bug
  Components: test
Affects Versions: 0.89-fb
Reporter: Elliott Clark
Assignee: Elliott Clark
 Fix For: 0.89-fb






--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-10692) The Multi TableMap job don't support the security HBase cluster


[ 
https://issues.apache.org/jira/browse/HBASE-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010256#comment-14010256
 ] 

Hudson commented on HBASE-10692:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #298 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/298/])
HBASE-10692 The Multi TableMap job don't support the security HBase cluster 
(Liu Shaohui) (apurtell: rev d3ecf7523d0de335f0f95b36447c891365e18c6b)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java


 The Multi TableMap job don't support the security HBase cluster
 ---

 Key: HBASE-10692
 URL: https://issues.apache.org/jira/browse/HBASE-10692
 Project: HBase
  Issue Type: Bug
  Components: mapreduce
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.94.21, 0.98.4

 Attachments: HBASE-10692-0.94-v1.diff, HBASE-10692-trunk-v1.diff, 
 HBASE-10692-trunk-v2.diff


 HBASE-3996 adds the support of multiple tables and scanners as input to the 
 mapper in map/reduce jobs. But it don't support the security HBase cluster.
 [~erank] [~bbaugher]
 Ps: HBASE-3996 only support multiple tables from the same HBase cluster. 
 Should we support multiple tables from the different clusters?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11255) Negative request num in region load


[ 
https://issues.apache.org/jira/browse/HBASE-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010255#comment-14010255
 ] 

Hudson commented on HBASE-11255:


FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #298 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/298/])
HBASE-11255 Negative request num in region load (Liu Shaohui) (stack: rev 
2e0ad596fd5874f25e558beb42568c276f82)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


 Negative request num in region load
 ---

 Key: HBASE-11255
 URL: https://issues.apache.org/jira/browse/HBASE-11255
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 0.96.3, 0.98.4

 Attachments: HBASE-11255-trunk-v1.diff


 We found that the request number of region is negative in long-running hbase 
 cluster.
 The is because of improper cast in HRegionServer#createRegionLoad
 {code}
   ...
   .setReadRequestsCount((int)r.readRequestsCount.get())
   .setWriteRequestsCount((int) r.writeRequestsCount.get()) 
 {code}
 The patch is simple and just removes the cast.
  



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag


[ 
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010262#comment-14010262
 ] 

Demai Ni commented on HBASE-9531:
-

[~jdcryans], may I know your takes about this feature? thanks... Demai

 a command line (hbase shell) interface to retreive the replication metrics 
 and show replication lag
 ---

 Key: HBASE-9531
 URL: https://issues.apache.org/jira/browse/HBASE-9531
 Project: HBase
  Issue Type: New Feature
  Components: Replication
Affects Versions: 0.99.0
Reporter: Demai Ni
Assignee: Demai Ni
 Fix For: 0.99.0, 0.98.4

 Attachments: HBASE-9531-trunk-v0.patch, HBASE-9531-trunk-v0.patch


 This jira is to provide a command line (hbase shell) interface to retreive 
 the replication metrics info such as:ageOfLastShippedOp, 
 timeStampsOfLastShippedOp, sizeOfLogQueue ageOfLastAppliedOp, and 
 timeStampsOfLastAppliedOp. And also to provide a point of time info of the 
 lag of replication(source only)
 Understand that hbase is using Hadoop 
 metrics(http://hbase.apache.org/metrics.html), which is a common way to 
 monitor metric info. This Jira is to serve as a light-weight client 
 interface, comparing to a completed(certainly better, but heavier)GUI 
 monitoring package. I made the code works on 0.94.9 now, and like to use this 
 jira to get opinions about whether the feature is valuable to other 
 users/workshop. If so, I will build a trunk patch. 
 All inputs are greatly appreciated. Thank you!
 The overall design is to reuse the existing logic which supports hbase shell 
 command 'status', and invent a new module, called ReplicationLoad.  In 
 HRegionServer.buildServerLoad() , use the local replication service objects 
 to get their loads  which could be wrapped in a ReplicationLoad object and 
 then simply pass it to the ServerLoad. In ReplicationSourceMetrics and 
 ReplicationSinkMetrics, a few getters and setters will be created, and ask 
 Replication to build a ReplicationLoad.  (many thanks to Jean-Daniel for 
 his kindly suggestions through dev email list)
 the replication lag will be calculated for source only, and use this formula: 
 {code:title=Replication lag|borderStyle=solid}
   if sizeOfLogQueue != 0 then max(ageOfLastShippedOp, (current time - 
 timeStampsOfLastShippedOp)) //err on the large side
   else if (current time - timeStampsOfLastShippedOp)  2* 
 ageOfLastShippedOp then lag = ageOfLastShippedOp // last shipped happen 
 recently 
 else lag = 0 // last shipped may happens last night, so NO real lag 
 although ageOfLastShippedOp is non-zero
 {code}
 External will look something like:
 {code:title=status 'replication'|borderStyle=solid}
 hbase(main):001:0 status 'replication'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:50:59 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
 hbase(main):002:0 status 'replication','source'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
     hdtest018.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
     hdtest015.svl.ibm.com:
     SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0, 
 timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
 hbase(main):003:0 status 'replication','sink'
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
     hdtest018.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:50:59 PDT 2013
     hdtest015.svl.ibm.com:
     SINK  :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04 
 14:48:48 PDT 2013
 hbase(main):003:0 status 'replication','lag' 
 version 0.94.9
 3 live servers
     hdtest017.svl.ibm.com: lag = 0
     hdtest018.svl.ibm.com: lag = 14
     hdtest015.svl.ibm.com: lag = 0
 {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (HBASE-11220) Add listeners to ServerManager and AssignmentManager


[ 
https://issues.apache.org/jira/browse/HBASE-11220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14010279#comment-14010279
 ] 

stack commented on HBASE-11220:
---

+1

 Add listeners to ServerManager and AssignmentManager
 

 Key: HBASE-11220
 URL: https://issues.apache.org/jira/browse/HBASE-11220
 Project: HBase
  Issue Type: Improvement
  Components: master
Affects Versions: 0.99.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Minor
 Attachments: HBASE-11220-v0.patch, HBASE-11220-v1.patch


 Add support for listeners to ServerManager and AssignmentManager.
 This will allows to get notified about servers added/removed or regions 
 added/removed/moved.
 I'm planning to use this in the MasterProcedureManager. Since we are starting 
 using the Procedures for distributed operations, we must add support for RS 
 joining or Regions moving. At the moment the operation on the moving set of 
 RSs is lost.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (HBASE-11085) Incremental Backup Restore support

[
https://issues.apache.org/jira/browse/HBASE-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Demai Ni updated HBASE-11085:
-

Attachment: HBASE-11085-trunk-v2.patch

Incremental Backup Restore support
--

Key: HBASE-11085
URL: https://issues.apache.org/jira/browse/HBASE-11085
Project: HBase
Issue Type: New Feature
Reporter: Demai Ni
Assignee: Demai Ni
Fix For: 0.99.0

Attachments:
HBASE-11085-trunk-v1-contains-HBASE-10900-trunk-v4.patch,
HBASE-11085-trunk-v1.patch, HBASE-11085-trunk-v2.patch

h2. Feature Description
the jira is part of
[HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on
full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900].
for the detail layout and frame work, please reference to [HBASE-10900|
https://issues.apache.org/jira/browse/HBASE-10900].
When client issues an incremental backup request, BackupManager will check
the request and then kicks of a global procedure via HBaseAdmin for all the
active regionServer to roll log. Each Region server will record their log
number into zookeeper. Then we determine which log need to be included in
this incremental backup, and use DistCp to copy them to target location. At
the same time, a dependency of backup image will be recorded, and later on
saved in Backup Manifest file.
Restore is to replay the backuped WAL logs on target HBase instance. The
replay will occur after full backup.
As incremental backup image depends on prior full backup image and
incremental images if exists. Manifest file will be used to store the
dependency lineage during backup, and used during restore time for PIT
restore.
h2. Use case(i.e example)
{code:title=Incremental Backup Restore example|borderStyle=solid}
/***/
/* STEP1: FULL backup from sourcecluster to targetcluster
/* if no table name specified, all tables from source cluster will be
backuped
/***/
[sourcecluster]$ hbase backup create full
hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
...
14/05/09 13:35:46 INFO backup.BackupManager: Backup request
backup_1399667695966 has been executed.
/***/
/* STEP2: In HBase Shell, put a few rows

/***/
hbase(main):002:0 put 't1_dn','row100','cf1:q1','value100_0509_increm1'
hbase(main):003:0 put 't2_dn','row100','cf1:q1','value100_0509_increm1'
hbase(main):004:0 put 't3_dn','row100','cf1:q1','value100_0509_increm1'
/***/
/* STEP3: Take the 1st incremental backup

/***/
[sourcecluster]$ hbase backup create incremental
hdfs://hostname.targetcluster.org:9000/userid/backupdir
...
14/05/09 13:37:45 INFO backup.BackupManager: Backup request
backup_1399667851020 has been executed.
/***/
/* STEP4: In HBase Shell, put a few more rows.

/* update 'row100', and create new 'row101'

/***/
hbase(main):005:0 put 't3_dn','row100','cf1:q1','value101_0509_increm2'
hbase(main):006:0 put 't2_dn','row100','cf1:q1','value101_0509_increm2'
hbase(main):007:0 put 't1_dn','row100','cf1:q1','value101_0509_increm2'
hbase(main):009:0 put 't1_dn','row101','cf1:q1','value101_0509_increm2'
hbase(main):010:0 put 't2_dn','row101','cf1:q1','value101_0509_increm2'
hbase(main):011:0 put 't3_dn','row101','cf1:q1','value101_0509_increm2'
/***/
/* STEP5: Take the 2nd incremental backup

/***/
[sourcecluster]$ hbase backup create incremental
hdfs://hostname.targetcluster.org:9000/userid/backupdir
...
14/05/09 13:39:33 INFO backup.BackupManager: Backup request
backup_1399667959165 has been executed.
/***/
/* STEP7: Restore from PIT of the 1st

[jira] [Updated] (HBASE-11085) Incremental Backup Restore support


 [ 
https://issues.apache.org/jira/browse/HBASE-11085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Demai Ni updated HBASE-11085:
-

Attachment: HBASE-11085-trunk-v2-contain-HBASE-10900-trunk-v4.patch

attached V2 patch, which contains:
1) a unit testcase (thanks to [~tianq], who implemented it)
2) address comments from [~tedyu]
3) fix a few long line warning, and java doc warnings

 Incremental Backup Restore support
 --

 Key: HBASE-11085
 URL: https://issues.apache.org/jira/browse/HBASE-11085
 Project: HBase
  Issue Type: New Feature
Reporter: Demai Ni
Assignee: Demai Ni
 Fix For: 0.99.0

 Attachments: 
 HBASE-11085-trunk-v1-contains-HBASE-10900-trunk-v4.patch, 
 HBASE-11085-trunk-v1.patch, 
 HBASE-11085-trunk-v2-contain-HBASE-10900-trunk-v4.patch, 
 HBASE-11085-trunk-v2.patch


 h2. Feature Description
 the jira is part of  
 [HBASE-7912|https://issues.apache.org/jira/browse/HBASE-7912], and depend on 
 full backup [HBASE-10900| https://issues.apache.org/jira/browse/HBASE-10900]. 
 for the detail layout and frame work, please reference to  [HBASE-10900| 
 https://issues.apache.org/jira/browse/HBASE-10900].
 When client issues an incremental backup request, BackupManager will check 
 the request and then kicks of a global procedure via HBaseAdmin for all the 
 active regionServer to roll log. Each Region server will record their log 
 number into zookeeper. Then we determine which log need to be included in 
 this incremental backup, and use DistCp to copy them to target location. At 
 the same time, a dependency of backup image will be recorded, and later on 
 saved in Backup Manifest file.
 Restore is to replay the backuped WAL logs on target HBase instance. The 
 replay will occur after full backup.
 As incremental backup image depends on prior full backup image and 
 incremental images if exists. Manifest file will be used to store the 
 dependency lineage during backup, and used during restore time for PIT 
 restore.  
 h2. Use case(i.e  example)
 {code:title=Incremental Backup Restore example|borderStyle=solid}
 /***/
 /* STEP1:  FULL backup from sourcecluster to targetcluster  
 /* if no table name specified, all tables from source cluster will be 
 backuped 
 /***/
 [sourcecluster]$ hbase backup create full 
 hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
 ...
 14/05/09 13:35:46 INFO backup.BackupManager: Backup request 
 backup_1399667695966 has been executed.
 /***/
 /* STEP2:   In HBase Shell, put a few rows
 
 /***/
 hbase(main):002:0 put 't1_dn','row100','cf1:q1','value100_0509_increm1'
 hbase(main):003:0 put 't2_dn','row100','cf1:q1','value100_0509_increm1'
 hbase(main):004:0 put 't3_dn','row100','cf1:q1','value100_0509_increm1'
 /***/
 /* STEP3:   Take the 1st incremental backup   
  
 /***/
 [sourcecluster]$ hbase backup create incremental 
 hdfs://hostname.targetcluster.org:9000/userid/backupdir
 ...
 14/05/09 13:37:45 INFO backup.BackupManager: Backup request 
 backup_1399667851020 has been executed.
 /***/
 /* STEP4:   In HBase Shell, put a few more rows.  
 
 /*   update 'row100', and create new 'row101' 
   
 /***/
 hbase(main):005:0 put 't3_dn','row100','cf1:q1','value101_0509_increm2'
 hbase(main):006:0 put 't2_dn','row100','cf1:q1','value101_0509_increm2'
 hbase(main):007:0 put 't1_dn','row100','cf1:q1','value101_0509_increm2'
 hbase(main):009:0 put 't1_dn','row101','cf1:q1','value101_0509_increm2'
 hbase(main):010:0 put 't2_dn','row101','cf1:q1','value101_0509_increm2'
 hbase(main):011:0 put 't3_dn','row101','cf1:q1','value101_0509_increm2'
 /***/
 /* STEP5:   Take the 2nd incremental backup   
 
 /***/
 [sourcecluster]$ hbase backup create incremental

[jira] [Updated] (HBASE-11085) Incremental Backup Restore support