date:20140905

[
https://issues.apache.org/jira/browse/HBASE-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122498#comment-14122498
]

Hudson commented on HBASE-11882:

FAILURE: Integrated in HBase-TRUNK #5467 (See
[https://builds.apache.org/job/HBase-TRUNK/5467/])
HBASE-11882 Row level consistency may not be maintained with bulk load and
(ramkrishna: rev 8de30d32d4d5c86650effadbda72f7ef32a4f15f)
*
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/Compactor.java

Row level consistency may not be maintained with bulk load and compaction
-

Key: HBASE-11882
URL: https://issues.apache.org/jira/browse/HBASE-11882
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.99.0, 2.0.0
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
Fix For: 0.99.0, 2.0.0

Attachments: HBASE-11882-master-v1.patch,
HBASE-11882-master-v2.patch, HBASE-11882-master-v3.patch,
TestHRegionServerBulkLoad.java.patch

While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I
found the root cause is that row level atomicity may not be maintained with
bulk load together with compation.
TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses
multiple threads to do bulk load and scan continuously and do compactions
periodically.
It verifies row level data is always consistent across column families.
After HBASE-11591, we added readpoint checks for bulkloaded data using the
seqId at the time of bulk load. Now a scanner will not see the data from a
bulk load if the scanner's readpoint is earlier than the bulk load seqId.
Previously, the atomic bulk load result is visible immediately to all
scanners.
The problem is with compaction after bulk load. Compaction does not lock the
region and it is done one store (column family) at a time. It also compact
away the seqId marker of bulk load.
Here is an event sequence where the row level consistency is broken.
1. A scanner is started to scan a region with cf1 and cf2. The readpoint is
10.
2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is
11. Bulk load is guarded by region write lock. So it is atomic.
3. There is a compaction that compacts cf1. It compacts away the seqId marker
of the bulk load.
4. The scanner tries to next to row-1001. It gets the bulk load data for cf1
since there is no seqId preventing it. It does not get the bulk load data
for cf2 since the scanner's readpoint (10) is less than the bulk load seqId
(11).
Now the row level consistency is broken in this case.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11877) Make TableSplit more readable


[ 
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122499#comment-14122499
 ] 

Hudson commented on HBASE-11877:


SUCCESS: Integrated in HBase-0.98 #497 (See 
[https://builds.apache.org/job/HBase-0.98/497/])
HBASE-11877 Make TableSplit more readable (Liu Shaohui) (stack: rev 
817f30110e6fdcba468c848f4dc458ecf164ce1e)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSplit.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSplit.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestSplitTable.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java


 Make TableSplit more readable
 -

 Key: HBASE-11877
 URL: https://issues.apache.org/jira/browse/HBASE-11877
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff


 When debugging MR jobs reading from hbase table, it's import to figure out 
 which region a map task is reading from.
 But the table split object is hard to read.
 eg:
 {code}
 2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask: 
 Processing split: lg-hadoop-prc-st40.bj:,0
 {code}
 See: TableSplit.java 
 {code}
   @Override
   public String toString() {
 return m_regionLocation + : +
   Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
   }
 {code}
 We should make it more readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source

2014-09-05 Thread Dima Spivak (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dima Spivak updated HBASE-11885:

Attachment: HBASE-11885_v2.patch

Added hbase_docker.sh into dev-support, a driver script that lets users specify 
locations of the Oracle JDK and Maven tarballs before running docker build 
(repeatedly, if necessary), and then starting the docker image.

 Provide a Dockerfile to easily build and run HBase from source
 --

 Key: HBASE-11885
 URL: https://issues.apache.org/jira/browse/HBASE-11885
 Project: HBase
  Issue Type: New Feature
Reporter: Dima Spivak
Assignee: Dima Spivak
 Attachments: HBASE-11885.patch, HBASE-11885_v2.patch


 [A recent email to 
 dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E]
  highlighted the difficulty that new users can face in getting HBase compiled 
 from source and running locally. I'd like to provide a Dockerfile that would 
 allow anyone with Docker running on a machine with a reasonably current Linux 
 kernel to do so with ease.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

2014-09-05 Thread Anoop Sam John (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122520#comment-14122520
 ] 

Anoop Sam John commented on HBASE-11772:


In your eg: you were telling hfile named 'abc_SeqId_10_' can exist in HBase
But I got it now.  The file can be already a bulk loaded one and so seqId_ part 
already in name. Again when u load this to another table, second seqId_ part 
can come in. So yes, for getting the actual seq_id of the file, go with 
lastIndexOf() and there is no issue with this check to know whether this is a 
bulk loaded one or not.
{code}
+String fileName = this.getPath().getName();
+int startPos = fileName.indexOf(SeqId_);
+if (startPos != -1) {
+  bulkLoadedHFile = true;
+}{code}
This is what ur thinking?

[~tedyu] was asking why the change to lastIndexOf().


 Bulk load mvcc and seqId issues with native hfiles
 --

 Key: HBASE-11772
 URL: https://issues.apache.org/jira/browse/HBASE-11772
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.5
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
 Fix For: 0.99.0, 1.0.0, 2.0.0, 0.98.7

 Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch


 There are mvcc and seqId issues when bulk load native hfiles -- meaning 
 hfiles that are direct file copy-out from hbase, not from HFileOutputFormat 
 job.
 There are differences between these two types of hfiles.
 Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero 
 mvcc values in cells. 
 Native hfiles also have MAX_SEQ_ID_KEY.
 Native hfiles do not have BULKLOAD_TIME_KEY.
 Here are a couple of problems I observed when bulk load native hfiles.
 1.  Cells in newly bulk loaded hfiles can be invisible to scan.
 It is easy to re-create.
 Bulk load a native hfile that has a larger mvcc value in cells, e.g 10
 If the current readpoint when initiating a scan is less than 10, the cells in 
 the new hfile are skipped, thus become invisible.
 We don't reset the readpoint of a region after bulk load.
 2. The current StoreFile.isBulkLoadResult() is implemented as:
 {code}
 return metadataMap.containsKey(BULKLOAD_TIME_KEY)
 {code}
 which does not detect bulkloaded native hfiles.
 3. Another observed problem is possible data loss during log recovery. 
 It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create 
 steps from HBASE-10958.
 1) Create an empty table
 2) Put one row in it (let's say it gets seqid 1)
 3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile 
 can be obtained by copying out from existing table.
 4) Kill the region server that holds the table's region.
 Scan the table once the region is made available again. The first row, at 
 seqid 1, will be missing since the HFile with seqid 100 makes us believe that 
 everything that came before it was flushed. 
 The problem 3 is probably related to 2. We will be ok if we get the appended 
 seqId during bulk load instead of 100 from inside the file.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11877) Make TableSplit more readable


[ 
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122557#comment-14122557
 ] 

Hudson commented on HBASE-11877:


SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #471 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/471/])
HBASE-11877 Make TableSplit more readable (Liu Shaohui) (stack: rev 
817f30110e6fdcba468c848f4dc458ecf164ce1e)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSplit.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSplit.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestSplitTable.java


 Make TableSplit more readable
 -

 Key: HBASE-11877
 URL: https://issues.apache.org/jira/browse/HBASE-11877
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff


 When debugging MR jobs reading from hbase table, it's import to figure out 
 which region a map task is reading from.
 But the table split object is hard to read.
 eg:
 {code}
 2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask: 
 Processing split: lg-hadoop-prc-st40.bj:,0
 {code}
 See: TableSplit.java 
 {code}
   @Override
   public String toString() {
 return m_regionLocation + : +
   Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
   }
 {code}
 We should make it more readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source

[
https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122582#comment-14122582
]

Hadoop QA commented on HBASE-11885:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1274/HBASE-11885_v2.patch
against trunk revision .
ATTACHMENT ID: 1274

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+# # ./hbase_docker.sh -j
http://download.oracle.com/otn-pub/java/jdk/7u67-b01/jdk-7u67-linux-x64.tar.gz \
+ENV PATH
/usr/java/bin:/usr/local/apache-maven/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ENV PATH
/root/hbase-bin/bin:/usr/java/bin:/usr/local/apache-maven/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10720//console

This message is automatically generated.

Provide a Dockerfile to easily build and run HBase from source
--

Key: HBASE-11885
URL: https://issues.apache.org/jira/browse/HBASE-11885
Project: HBase
Issue Type: New Feature
Reporter: Dima Spivak
Assignee: Dima Spivak
Attachments: HBASE-11885.patch, HBASE-11885_v2.patch

[A recent email to
dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E]
highlighted the difficulty that new users can face in getting HBase compiled
from source and running locally. I'd like to provide a Dockerfile that would
allow anyone with Docker running on a machine with a reasonably current Linux
kernel to do so with ease.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11877) Make TableSplit more readable


[ 
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122593#comment-14122593
 ] 

Hudson commented on HBASE-11877:


SUCCESS: Integrated in HBase-1.0 #153 (See 
[https://builds.apache.org/job/HBase-1.0/153/])
HBASE-11877 Make TableSplit more readable (Liu Shaohui) (stack: rev 
c0d81e9ad014609c7a5e118e66b902f828818edd)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSplit.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestSplitTable.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSplit.java


 Make TableSplit more readable
 -

 Key: HBASE-11877
 URL: https://issues.apache.org/jira/browse/HBASE-11877
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff


 When debugging MR jobs reading from hbase table, it's import to figure out 
 which region a map task is reading from.
 But the table split object is hard to read.
 eg:
 {code}
 2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask: 
 Processing split: lg-hadoop-prc-st40.bj:,0
 {code}
 See: TableSplit.java 
 {code}
   @Override
   public String toString() {
 return m_regionLocation + : +
   Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
   }
 {code}
 We should make it more readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11440) Make KeyValueCodecWithTags as the default codec for replication in trunk


[ 
https://issues.apache.org/jira/browse/HBASE-11440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122592#comment-14122592
 ] 

Hudson commented on HBASE-11440:


SUCCESS: Integrated in HBase-1.0 #153 (See 
[https://builds.apache.org/job/HBase-1.0/153/])
HBASE-11440 Make KeyValueCodecWithTags as the default codec for (ramkrishna: 
rev 0eddf0c984c2e4169a3f312747ff279fe44ded00)
* hbase-common/src/main/resources/hbase-default.xml


 Make KeyValueCodecWithTags as the default codec for replication in trunk
 

 Key: HBASE-11440
 URL: https://issues.apache.org/jira/browse/HBASE-11440
 Project: HBase
  Issue Type: Improvement
Affects Versions: 0.99.0
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
 Fix For: 0.99.0, 2.0.0

 Attachments: HBASE-11440.patch, HBASE-11440_1.patch, 
 HBASE-11440_2.patch


 Set 
 {code}
 property
 namehbase.replication.rpc.codec/name  
 valueorg.apache.hadoop.hbase.codec.KeyValueCodecWithTags/value
 /property
 {code}
 in hbase-default.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11072) Abstract WAL splitting from ZK


[ 
https://issues.apache.org/jira/browse/HBASE-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122594#comment-14122594
 ] 

Hudson commented on HBASE-11072:


SUCCESS: Integrated in HBase-1.0 #153 (See 
[https://builds.apache.org/job/HBase-1.0/153/])
HBASE-11072 Abstract WAL splitting from ZK (Sergey Soldatov) (stack: rev 
66220e492979751824ec47dd85174c5d5924af50)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogManagerCoordination.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/BaseCoordinatedStateManager.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/HLogSplitterHandler.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java
* hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/SplitLogWorkerCoordination.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZKSplitLogManagerCoordination.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkCoordinatedStateManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogReaderOnSecureHLog.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestWALReplay.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogMethods.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coordination/ZkSplitLogWorkerCoordination.java
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java


 Abstract WAL splitting from ZK
 --

 Key: HBASE-11072
 URL: https://issues.apache.org/jira/browse/HBASE-11072
 Project: HBase
  Issue Type: Sub-task
  Components: Consensus, Zookeeper
Affects Versions: 0.99.0
Reporter: Mikhail Antonov
Assignee: Sergey Soldatov
 Fix For: 0.99.0, 2.0.0

 Attachments: HBASE-11072-1_v2.patch, HBASE-11072-1_v3.patch, 
 HBASE-11072-1_v4.patch, HBASE-11072-2_v2.patch, HBASE-11072-branch1.patch, 
 HBASE-11072-v1.patch, HBASE-11072-v10.patch, HBASE-11072-v10.patch, 
 HBASE-11072-v11.patch, HBASE-11072-v11.patch, HBASE-11072-v11.patch, 
 HBASE-11072-v11.patch, HBASE-11072-v11.patch, HBASE-11072-v2.patch, 
 HBASE-11072-v3.patch, HBASE-11072-v4.patch, HBASE-11072-v5.patch, 
 HBASE-11072-v6.patch, HBASE-11072-v7.patch, HBASE-11072-v8.patch, 
 HBASE-11072-v8.patch, HBASE-11072-v9.patch, HBASE_11072-1.patch


 HM side:
  - SplitLogManager
 RS side:
  - SplitLogWorker
  - HLogSplitter and a few handler classes.
 This jira may need to be split further apart into smaller ones.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11646) Handle the MOB in compaction

2014-09-05 Thread Jingcheng Du (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jingcheng Du updated HBASE-11646:
-
Attachment: HBASE-11646-V5.diff

Update the patch after the value size of a reference cell in a mob-enabled 
column is changed from a long to an int.

 Handle the MOB in compaction
 

 Key: HBASE-11646
 URL: https://issues.apache.org/jira/browse/HBASE-11646
 Project: HBase
  Issue Type: Sub-task
  Components: Compaction
Reporter: Jingcheng Du
Assignee: Jingcheng Du
 Attachments: HBASE-11646-V2.diff, HBASE-11646-V3.diff, 
 HBASE-11646-V4.diff, HBASE-11646-V5.diff, HBASE-11646.diff


 In the updated MOB design however, admins can set CF level thresholds that 
 would force cell values  the threshold to use the MOB write path instead of 
 the traditional path.  There are two cases where mobs need to interact with 
 this threshold
 1) How do we handle the case when the threshold size is changed?
 2) Today, you can bulkload hfiles that contain MOBs.  These cells will work 
 as normal inside hbase.  Unfortunately the cells with MOBs in them will never 
 benefit form the MOB write path.
 The proposal here is to modify compaction in mob enabled cf's such that the 
 threshold value is honored with compactions.  This handles case #1 -- 
 elements that should be moved out of the normal hfiles get 'compacted' into 
 refs and mob hfiles, and values that should be pulled into the cf get derefed 
 and written out wholy in the compaction.  For case #2, we can maintain the 
 same behavior and compaction would move data into the mob writepath/lifecycle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys


[ 
https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122604#comment-14122604
 ] 

rajeshbabu commented on HBASE-10576:


[~tedyu]
Thanks for review.
bq. Why is check of shadow tied to offline status ?
There is no relation to offline status..That need to be separated. It's 
mistake. I will correct it.
bq. Can you add javadoc for the above method ?
I will add
bq. What's the purpose for adding trailing delimiter ?
The delimeter is not required, will remove it.

Once I complete the patch I will put it in RB.


 Custom load balancer to co-locate the regions of two tables which are having 
 same split keys
 

 Key: HBASE-10576
 URL: https://issues.apache.org/jira/browse/HBASE-10576
 Project: HBase
  Issue Type: Sub-task
  Components: Balancer
Reporter: rajeshbabu
Assignee: rajeshbabu
 Attachments: HBASE-10536_v2.patch, HBASE-10576.patch, 
 HBASE-10576_shadow_regions_wip.patch


 To support local indexing both user table and index table should have same 
 split keys. This issue to provide custom balancer to colocate the regions of 
 two tables which are having same split keys. 
 This helps in Phoenix as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11877) Make TableSplit more readable


[ 
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122610#comment-14122610
 ] 

Hudson commented on HBASE-11877:


FAILURE: Integrated in HBase-TRUNK #5468 (See 
[https://builds.apache.org/job/HBase-TRUNK/5468/])
HBASE-11877 Make TableSplit more readable (Liu Shaohui) (stack: rev 
aa05ad1291130919383455d39a7b3679c7b636d6)
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSplit.java
* 
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSplit.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSplit.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestSplitTable.java


 Make TableSplit more readable
 -

 Key: HBASE-11877
 URL: https://issues.apache.org/jira/browse/HBASE-11877
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff


 When debugging MR jobs reading from hbase table, it's import to figure out 
 which region a map task is reading from.
 But the table split object is hard to read.
 eg:
 {code}
 2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask: 
 Processing split: lg-hadoop-prc-st40.bj:,0
 {code}
 See: TableSplit.java 
 {code}
   @Override
   public String toString() {
 return m_regionLocation + : +
   Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
   }
 {code}
 We should make it more readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11902) RegionServer was blocked while aborting

2014-09-05 Thread Victor Xu (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122682#comment-14122682
 ] 

Victor Xu commented on HBASE-11902:
---

Yes, stack. The rs main thread is waiting at 
org.apache.hadoop.hbase.util.DrainBarrier.stopAndDrainOps, but the main cause 
of the aborting is DataNodes. You can find the details in the log:
2014-09-03 13:38:03,789 FATAL org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Error while AsyncSyncer sync, request close of hlog 
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,799 ERROR 
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flush failed for 
region page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c.
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,801 ERROR 
org.apache.hadoop.hbase.regionserver.wal.ProtobufLogWriter: Got IOException 
while writing trailer
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,802 ERROR org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Failed close of HLog writer
java.io.IOException: All datanodes 10.246.2.103:50010 are bad. Aborting...
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1127)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:924)
at 
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:486)
2014-09-03 13:38:03,802 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Riding over HLog close failure! error count=1
2014-09-03 13:38:03,804 INFO org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
Rolled WAL 
/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409722420708
 with entries=32565, filesize=118.6 M; new WAL 
/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409722683780
2014-09-03 13:38:03,804 DEBUG org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
log file is ready for archiving 
hdfs://hadoopnnvip.cm6:9000/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409707475254
2014-09-03 13:38:03,804 DEBUG org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
log file is ready for archiving 
hdfs://hadoopnnvip.cm6:9000/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409707722202
2014-09-03 13:38:03,804 DEBUG org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
log file is ready for archiving 
hdfs://hadoopnnvip.cm6:9000/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409707946159
2014-09-03 13:38:03,804 DEBUG org.apache.hadoop.hbase.regionserver.wal.FSHLog: 
log file is ready for archiving 
hdfs://hadoopnnvip.cm6:9000/hbase/WALs/hadoop461.cm6.tbsite.net,60020,1409003284950/hadoop461.cm6.tbsite.net%2C60020%2C1409003284950.1409708155788
2014-09-03 13:38:03,839 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Flush requested on 
page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c.
2014-09-03 13:38:03,839 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Started memstore flush for 
page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c., 
current region memstore size 218.5 M
2014-09-03 13:38:03,887 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Flush requested on 
page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c.
2014-09-03 13:38:03,887 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
Started memstore flush for 
page_content_queue,00166,1408946731655.8671b8a0f82565f88eb2ab8a5b53e84c., 
current region memstore size 218.5 M
2014-09-03 13:38:03,897 DEBUG

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance

2014-09-05 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122696#comment-14122696
 ] 

Lars Hofhansl commented on HBASE-11898:
---

That one looks suspicious.

 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys

[
https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122708#comment-14122708
]

rajeshbabu commented on HBASE-10576:

[~stack]
bq. How is this balancer patch supposed to work?
Once this shadow region feature goes in the balancer patch is not required.
We can ensure the co-location using coprocessors.
bq. I turn on this feature for a table? Or is it for whole cluster?
It's region wise Stack. The shadow region is like offline region but not
completely
(Offline at master side and can be opened/closed at region server through
coprocessors).
If we make a region shadow then master considers it as offline and will not
move/reassign
in any case like master/RS failover. At the same time the region serves reads
and writes
from external client if it's online at RS.

In case of secondary index,to ensure co-location we can make the index
region(s) shadow
and open/close while opening/closing data region(s) through coprocessors.
Since the regions cannot be moved by master they will be served by the RS
holding data region always.

The shadow region operation can be performed similar like split or merge on a
region.
If we want to make the table shadow we need to make all the regions shadow.

Thanks.

Custom load balancer to co-locate the regions of two tables which are having
same split keys

Key: HBASE-10576
URL: https://issues.apache.org/jira/browse/HBASE-10576
Project: HBase
Issue Type: Sub-task
Components: Balancer
Reporter: rajeshbabu
Assignee: rajeshbabu
Attachments: HBASE-10536_v2.patch, HBASE-10576.patch,
HBASE-10576_shadow_regions_wip.patch

To support local indexing both user table and index table should have same
split keys. This issue to provide custom balancer to colocate the regions of
two tables which are having same split keys.
This helps in Phoenix as well.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-5699) Run with 1 WAL in HRegionServer

2014-09-05 Thread Anoop Sam John (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anoop Sam John updated HBASE-5699:
--
Component/s: wal

 Run with  1 WAL in HRegionServer
 -

 Key: HBASE-5699
 URL: https://issues.apache.org/jira/browse/HBASE-5699
 Project: HBase
  Issue Type: Improvement
  Components: Performance, wal
Reporter: binlijin
Assignee: Li Pi
Priority: Critical
 Attachments: PerfHbase.txt






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11877) Make TableSplit more readable

2014-09-05 Thread Jean-Marc Spaggiari (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122832#comment-14122832
 ] 

Jean-Marc Spaggiari commented on HBASE-11877:
-

Sorry, I prepared a comment yesterday but forgot to press Add :( Here it is, 
even if to lte.

I checked the Bytes. code, etc. Looks good to me. Thanks fo the details and 
for the patch

+1


 Make TableSplit more readable
 -

 Key: HBASE-11877
 URL: https://issues.apache.org/jira/browse/HBASE-11877
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff


 When debugging MR jobs reading from hbase table, it's import to figure out 
 which region a map task is reading from.
 But the table split object is hard to read.
 eg:
 {code}
 2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask: 
 Processing split: lg-hadoop-prc-st40.bj:,0
 {code}
 See: TableSplit.java 
 {code}
   @Override
   public String toString() {
 return m_regionLocation + : +
   Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
   }
 {code}
 We should make it more readable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11869) Support snapshot owner

2014-09-05 Thread Liu Shaohui (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-11869:

Attachment: HBASE-11869-trunk-v4.diff

Update the patch. Thanks for [~anoop.hbase]'s review.

 Support snapshot owner
 --

 Key: HBASE-11869
 URL: https://issues.apache.org/jira/browse/HBASE-11869
 Project: HBase
  Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Fix For: 2.0.0

 Attachments: HBASE-11869-trunk-v1.diff, HBASE-11869-trunk-v3.diff, 
 HBASE-11869-trunk-v4.diff


 In current codebase, the table snapshot operations only can be done by the 
 global admin , not by  the table admin.
 There is a multi-tenant hbase cluster, each table has different snapshot 
 policies, eg: do snapshot per week, or snapshot after the new data are 
 imported. 
 We want to release the snapshot permission to each table admin.
 According to [~mbertozzi]'s suggestion, we implement the snapshot owner 
 feature.
 * The user with table admin permission can create snapshot and the owner of 
 this snapshot is this user.
 * The owner of snapshot can delete and restore the snapshot.
 * Only the user with global admin permission can clone a snapshot, for this 
 operation creates a new table.
   



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11897) Add append and remove peer table-cfs cmds for replication

2014-09-05 Thread Liu Shaohui (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liu Shaohui updated HBASE-11897:

Attachment: HBASE-11897-trunk-v2.diff

Fix the failed tests

 Add append and remove peer table-cfs cmds for replication
 -

 Key: HBASE-11897
 URL: https://issues.apache.org/jira/browse/HBASE-11897
 Project: HBase
  Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
 Attachments: HBASE-11897-trunk-v1.diff, HBASE-11897-trunk-v2.diff


 HBASE-8751 introduces the tables/table-column families config for a 
 replication peer. It's very flexible for practical replication in hbase 
 clusters.
 But it is easy to make mistakes during add or remove a table/table-column 
 family for a existing peer, especially when the table-cfs is very long, for 
 we need to copy the current table-cfs of the peer first,  and then add or 
 remove a table/table-column family to/from the table-cfs, at last set  back 
 the table-cfs using the cmd: set_peer_tableCFs. 
 So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs 
 to do the operation of adding and removing a table/table-column family. They 
 are useful operation tools.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11869) Support snapshot owner

[
https://issues.apache.org/jira/browse/HBASE-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122925#comment-14122925
]

Hadoop QA commented on HBASE-11869:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12666782/HBASE-11869-trunk-v4.diff
against trunk revision .
ATTACHMENT ID: 12666782

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestRegionRebalancing

{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.crypto.CryptoStreamsTestBase.testCombinedOp(CryptoStreamsTestBase.java:583)

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10721//console

This message is automatically generated.

Support snapshot owner
--

Key: HBASE-11869
URL: https://issues.apache.org/jira/browse/HBASE-11869
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0

Attachments: HBASE-11869-trunk-v1.diff, HBASE-11869-trunk-v3.diff,
HBASE-11869-trunk-v4.diff

In current codebase, the table snapshot operations only can be done by the
global admin , not by the table admin.
There is a multi-tenant hbase cluster, each table has different snapshot
policies, eg: do snapshot per week, or snapshot after the new data are
imported.
We want to release the snapshot permission to each table admin.
According to [~mbertozzi]'s suggestion, we implement the snapshot owner
feature.
* The user with table admin permission can create snapshot and the owner of
this snapshot is this user.
* The owner of snapshot can delete and restore the snapshot.
* Only the user with global admin permission can clone a snapshot, for this
operation creates a new table.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified


 [ 
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-11896:
---
Attachment: 11896-v2.txt

 LoadIncrementalHFiles fails in secure mode if the namespace is specified
 

 Key: HBASE-11896
 URL: https://issues.apache.org/jira/browse/HBASE-11896
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: 11896-v1.txt, 11896-v2.txt, HBASE-11896.patch


 completebulkload tool is failing in secure mode when namespace is specified.
 {noformat}
 Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: 
 Relative path in absolute URI: 
 hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
 at org.apache.hadoop.fs.Path.initialize(Path.java:206)
 at org.apache.hadoop.fs.Path.init(Path.java:172)
 at org.apache.hadoop.fs.Path.init(Path.java:94)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
 at 
 org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
 {noformat}
 Here I feel we should replace the ':' in table name
 {code}
 bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
 {code}
 Because at the later part of code we are creating staging directory using 
 table name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified


[ 
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123015#comment-14123015
 ] 

Hadoop QA commented on HBASE-11896:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12666795/11896-v2.txt
  against trunk revision .
  ATTACHMENT ID: 12666795

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
 

 {color:red}-1 core zombie tests{color}.  There are 1 zombie test(s):   
at 
org.apache.hadoop.hbase.http.TestHttpServerLifecycle.testStartedServerWithRequestLog(TestHttpServerLifecycle.java:92)

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10723//console

This message is automatically generated.

 LoadIncrementalHFiles fails in secure mode if the namespace is specified
 

 Key: HBASE-11896
 URL: https://issues.apache.org/jira/browse/HBASE-11896
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: 11896-v1.txt, 11896-v2.txt, HBASE-11896.patch


 completebulkload tool is failing in secure mode when namespace is specified.
 {noformat}
 Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: 
 Relative path in absolute URI: 
 hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
 at org.apache.hadoop.fs.Path.initialize(Path.java:206)
 at org.apache.hadoop.fs.Path.init(Path.java:172)
 at org.apache.hadoop.fs.Path.init(Path.java:94)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
 at 
 org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
 at

[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified


[ 
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123023#comment-14123023
 ] 

Ted Yu commented on HBASE-11896:


If HadoopSecurityEnabledUserProviderForTesting#isHBaseSecurityEnabled() returns 
true, I get:
{code}
2014-09-05 07:58:17,367 WARN  [RS:0;192.168.0.12:60646] 
regionserver.HRegionServer(2065): error telling master we are up
com.google.protobuf.ServiceException: java.io.IOException: java.io.IOException: 
Failed to specify server's Kerberos principal name
at 
org.apache.hadoop.hbase.ipc.RpcClient.callBlockingMethod(RpcClient.java:1699)
at 
org.apache.hadoop.hbase.ipc.RpcClient$BlockingRpcChannelImplementation.callBlockingMethod(RpcClient.java:1749)
at 
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8277)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2055)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:748)
at 
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156)
{code}

 LoadIncrementalHFiles fails in secure mode if the namespace is specified
 

 Key: HBASE-11896
 URL: https://issues.apache.org/jira/browse/HBASE-11896
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: 11896-v1.txt, 11896-v2.txt, HBASE-11896.patch


 completebulkload tool is failing in secure mode when namespace is specified.
 {noformat}
 Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: 
 Relative path in absolute URI: 
 hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
 at org.apache.hadoop.fs.Path.initialize(Path.java:206)
 at org.apache.hadoop.fs.Path.init(Path.java:172)
 at org.apache.hadoop.fs.Path.init(Path.java:94)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
 at 
 org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
 {noformat}
 Here I feel we should replace the ':' in table name
 {code}
 bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
 {code}
 Because at the later part of code we are creating staging directory using 
 table name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified


 [ 
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ted Yu updated HBASE-11896:
---
Attachment: 11896-v2.txt

 LoadIncrementalHFiles fails in secure mode if the namespace is specified
 

 Key: HBASE-11896
 URL: https://issues.apache.org/jira/browse/HBASE-11896
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: 11896-v1.txt, 11896-v2.txt, 11896-v2.txt, 
 HBASE-11896.patch


 completebulkload tool is failing in secure mode when namespace is specified.
 {noformat}
 Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: 
 Relative path in absolute URI: 
 hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
 at org.apache.hadoop.fs.Path.initialize(Path.java:206)
 at org.apache.hadoop.fs.Path.init(Path.java:172)
 at org.apache.hadoop.fs.Path.init(Path.java:94)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
 at 
 org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
 at 
 org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
 {noformat}
 Here I feel we should replace the ':' in table name
 {code}
 bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
 {code}
 Because at the later part of code we are creating staging directory using 
 table name.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


 [ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter updated HBASE-11679:
---
Attachment: HBASE_11679_v4.patch

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


 [ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter updated HBASE-11679:
---
Status: Patch Available  (was: Open)

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


[ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123042#comment-14123042
 ] 

Carter commented on HBASE-11679:


Patch added.  There are many other things that can be cleaned up, but I wanted 
to grab all the uncontroversial, low-hanging fruit in one pass.  Should be 
fairly easy to review, despite it being long.

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11897) Add append and remove peer table-cfs cmds for replication

[
https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123037#comment-14123037
]

Hadoop QA commented on HBASE-11897:
---

{color:red}-1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12666786/HBASE-11897-trunk-v2.diff
against trunk revision .
ATTACHMENT ID: 12666786

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestReplicaWithCluster
org.apache.hadoop.hbase.mapreduce.TestTableSnapshotInputFormat

{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10722//console

This message is automatically generated.

Add append and remove peer table-cfs cmds for replication
-

Key: HBASE-11897
URL: https://issues.apache.org/jira/browse/HBASE-11897
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-11897-trunk-v1.diff, HBASE-11897-trunk-v2.diff

HBASE-8751 introduces the tables/table-column families config for a
replication peer. It's very flexible for practical replication in hbase
clusters.
But it is easy to make mistakes during add or remove a table/table-column
family for a existing peer, especially when the table-cfs is very long, for
we need to copy the current table-cfs of the peer first, and then add or
remove a table/table-column family to/from the table-cfs, at last set back
the table-cfs using the cmd: set_peer_tableCFs.
So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs
to do the operation of adding and removing a table/table-column family. They
are useful operation tools.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


[ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123049#comment-14123049
 ] 

Hadoop QA commented on HBASE-11679:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12666804/HBASE_11679_v4.patch
  against trunk revision .
  ATTACHMENT ID: 12666804

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 495 
new or modified tests.

{color:red}-1 javac{color}.  The patch appears to cause mvn compile goal to 
fail.

Compilation errors resume:
[ERROR] COMPILATION ERROR : 
[ERROR] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java:[84,8]
 error: method render in class MasterStatusTmpl cannot be applied to given 
types;
[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) 
on project hbase-server: Compilation failure
[ERROR] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java:[84,8]
 error: method render in class MasterStatusTmpl cannot be applied to given 
types;
[ERROR] - [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e 
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please 
read the following articles:
[ERROR] [Help 1] 
http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn goals -rf :hbase-server


Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10725//console

This message is automatically generated.

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123058#comment-14123058
]

Jerry He commented on HBASE-11772:
--

Hi, [~anoop.hbase]

Yes. You explained it!

Re-based with attachment 'master-v2'

Bulk load mvcc and seqId issues with native hfiles
--

Key: HBASE-11772
URL: https://issues.apache.org/jira/browse/HBASE-11772
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.5
Reporter: Jerry He
Assignee: Jerry He
Priority: Critical
Fix For: 0.99.0, 1.0.0, 2.0.0, 0.98.7

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch

There are mvcc and seqId issues when bulk load native hfiles -- meaning
hfiles that are direct file copy-out from hbase, not from HFileOutputFormat
job.
There are differences between these two types of hfiles.
Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero
mvcc values in cells.
Native hfiles also have MAX_SEQ_ID_KEY.
Native hfiles do not have BULKLOAD_TIME_KEY.
Here are a couple of problems I observed when bulk load native hfiles.
1. Cells in newly bulk loaded hfiles can be invisible to scan.
It is easy to re-create.
Bulk load a native hfile that has a larger mvcc value in cells, e.g 10
If the current readpoint when initiating a scan is less than 10, the cells in
the new hfile are skipped, thus become invisible.
We don't reset the readpoint of a region after bulk load.
2. The current StoreFile.isBulkLoadResult() is implemented as:
{code}
return metadataMap.containsKey(BULKLOAD_TIME_KEY)
{code}
which does not detect bulkloaded native hfiles.
3. Another observed problem is possible data loss during log recovery.
It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create
steps from HBASE-10958.
1) Create an empty table
2) Put one row in it (let's say it gets seqid 1)
3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile
can be obtained by copying out from existing table.
4) Kill the region server that holds the table's region.
Scan the table once the region is made available again. The first row, at
seqid 1, will be missing since the HFile with seqid 100 makes us believe that
everything that came before it was flushed.
The problem 3 is probably related to 2. We will be ok if we get the appended
seqId during bulk load instead of 100 from inside the file.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jerry He updated HBASE-11772:
-
Attachment: HBASE-11772-master-v2.patch

Bulk load mvcc and seqId issues with native hfiles
--

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611

2014-09-05 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123060#comment-14123060
 ] 

Jimmy Xiang commented on HBASE-11894:
-

Can we invoke these coprocessor hooks in master(AM/RegionStateStore)?  We'd 
like to make sure only master updates the meta.

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.

2014-09-05 Thread Sean Busbey (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123067#comment-14123067
 ] 

Sean Busbey commented on HBASE-11891:
-

patch should apply cleanly on top of the patch from HBASE-11858

 Introduce HBaseInterfaceAudience level to denote class names that appear in 
 configs.
 

 Key: HBASE-11891
 URL: https://issues.apache.org/jira/browse/HBASE-11891
 Project: HBase
  Issue Type: Improvement
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Minor
 Attachments: HBASE-11891.patch


 We have several classes that are private use as far as APIs are concerned, 
 but the class name appear in user provided config files.
 We should add an HBaseInterfaceAudience level CONFIG and use it to denote 
 these classes as LimitedPrivate.
 HBase contributors can then use this audience marking to see when changes to 
 a FQCN will require a release note to help upgrading users correct their 
 configuration files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611


 [ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

rajeshbabu reassigned HBASE-11894:
--

Assignee: rajeshbabu

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611


[ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123068#comment-14123068
 ] 

rajeshbabu commented on HBASE-11894:


Let me try moving the hooks to master [~jxiang]. Thanks.

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611


[ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123085#comment-14123085
 ] 

rajeshbabu commented on HBASE-11894:


[~jxiang]
Currently the hooks are region level but moving them to system level hooks make 
not be good.
instead what about having new interface to pass any meta mutations while 
reporting region transition
{code}
  /**
   * Notify master that a handler requests to change a region state
   */
  boolean reportRegionStateTransition(TransitionCode code, HRegionInfo... 
hris,ListMutation mutations);
{code}

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11786) Document web UI for tracking time spent in coprocessors

2014-09-05 Thread Misty Stanley-Jones (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Misty Stanley-Jones updated HBASE-11786:

Attachment: HBASE-11786.patch

Integrated [~apurtell]'s suggestions

 Document web UI for tracking time spent in coprocessors
 ---

 Key: HBASE-11786
 URL: https://issues.apache.org/jira/browse/HBASE-11786
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Minor
 Attachments: HBASE-11786.patch, HBASE-11786.patch, HBASE-11786.patch, 
 HBASE-11786.patch, HBASE-11786.patch, coprocessor_stats.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance

2014-09-05 Thread Lars Hofhansl (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123103#comment-14123103
 ] 

Lars Hofhansl commented on HBASE-11898:
---

Alhough I do not see how this change can possibly call this... And test passes 
locally.
Going to commit this.

 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11786) Document web UI for tracking time spent in coprocessors


[ 
https://issues.apache.org/jira/browse/HBASE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123106#comment-14123106
 ] 

Hadoop QA commented on HBASE-11786:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12666813/HBASE-11786.patch
  against trunk revision .
  ATTACHMENT ID: 12666813

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+0 tests included{color}.  The patch appears to be a 
documentation patch that doesn't require tests.

{color:red}-1 patch{color}.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10727//console

This message is automatically generated.

 Document web UI for tracking time spent in coprocessors
 ---

 Key: HBASE-11786
 URL: https://issues.apache.org/jira/browse/HBASE-11786
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Minor
 Attachments: HBASE-11786.patch, HBASE-11786.patch, HBASE-11786.patch, 
 HBASE-11786.patch, HBASE-11786.patch, coprocessor_stats.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


[ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123112#comment-14123112
 ] 

Carter commented on HBASE-11679:


Odd.  Perhaps patch diff is against a slightly out-of-date master.  Will fix 
and resubmit.

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11786) Document web UI for tracking time spent in coprocessors

2014-09-05 Thread Misty Stanley-Jones (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123114#comment-14123114
 ] 

Misty Stanley-Jones commented on HBASE-11786:
-

I tried creating the last patch with git format-patch. Hudson says it can't 
apply the patch. Let me know if I should regenerate it using git diff.

 Document web UI for tracking time spent in coprocessors
 ---

 Key: HBASE-11786
 URL: https://issues.apache.org/jira/browse/HBASE-11786
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Minor
 Attachments: HBASE-11786.patch, HBASE-11786.patch, HBASE-11786.patch, 
 HBASE-11786.patch, HBASE-11786.patch, coprocessor_stats.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-11903) Directly invoking split merge of secondary regions should be disallowed

2014-09-05 Thread Devaraj Das (JIRA)

Devaraj Das created HBASE-11903:
---

 Summary: Directly invoking split  merge of secondary regions 
should be disallowed
 Key: HBASE-11903
 URL: https://issues.apache.org/jira/browse/HBASE-11903
 Project: HBase
  Issue Type: Sub-task
Reporter: Devaraj Das
Assignee: Devaraj Das


When the primary is split/merged the secondaries follow suit. We should 
disallow calling split/merge on the secondaries directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys

2014-09-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123129#comment-14123129
 ] 

stack commented on HBASE-10576:
---

What is the 'shadow region feature'?

You do not answer how I turn on/off colocation.

I have read the various designs on colocated index regions. What is the plan 
for landing this secondary index feature?  Is it to come in piecemeal via stuff 
like this intermediate change to the balancer? What changes to core are 
required?  Where do I read about it?  Thanks.

 Custom load balancer to co-locate the regions of two tables which are having 
 same split keys
 

 Key: HBASE-10576
 URL: https://issues.apache.org/jira/browse/HBASE-10576
 Project: HBase
  Issue Type: Sub-task
  Components: Balancer
Reporter: rajeshbabu
Assignee: rajeshbabu
 Attachments: HBASE-10536_v2.patch, HBASE-10576.patch, 
 HBASE-10576_shadow_regions_wip.patch


 To support local indexing both user table and index table should have same 
 split keys. This issue to provide custom balancer to colocate the regions of 
 two tables which are having same split keys. 
 This helps in Phoenix as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611

2014-09-05 Thread Jimmy Xiang (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123128#comment-14123128
 ] 

Jimmy Xiang commented on HBASE-11894:
-

I see. Ok with me.  Do we need to handle similar requirements for other 
transition codes? I was wondering how to make the interface extendable. 

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11903) Directly invoking split merge of replica regions should be disallowed

2014-09-05 Thread stack (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stack updated HBASE-11903:
--
Summary: Directly invoking split  merge of replica regions should be 
disallowed  (was: Directly invoking split  merge of secondary regions should 
be disallowed)

I edited the subject to make it more clear (undo if I have it wrong DD)

 Directly invoking split  merge of replica regions should be disallowed
 ---

 Key: HBASE-11903
 URL: https://issues.apache.org/jira/browse/HBASE-11903
 Project: HBase
  Issue Type: Sub-task
Reporter: Devaraj Das
Assignee: Devaraj Das

 When the primary is split/merged the secondaries follow suit. We should 
 disallow calling split/merge on the secondaries directly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified


[ 
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123130#comment-14123130
 ] 

Hadoop QA commented on HBASE-11896:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12666800/11896-v2.txt
  against trunk revision .
  ATTACHMENT ID: 12666800

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:red}-1 tests included{color}.  The patch doesn't appear to include 
any new or modified tests.
Please justify why no new tests are needed for this 
patch.
Also please list what manual steps were performed to 
verify this patch.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:green}+1 lineLengths{color}.  The patch does not introduce lines 
longer than 100

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}.  The patch passed unit tests in .

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10724//console

This message is automatically generated.

 LoadIncrementalHFiles fails in secure mode if the namespace is specified
 

 Key: HBASE-11896
 URL: https://issues.apache.org/jira/browse/HBASE-11896
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: 11896-v1.txt, 11896-v2.txt, 11896-v2.txt, 
 HBASE-11896.patch


 completebulkload tool is failing in secure mode when namespace is specified.
 {noformat}
 Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException: 
 Relative path in absolute URI: 
 hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
 at org.apache.hadoop.fs.Path.initialize(Path.java:206)
 at org.apache.hadoop.fs.Path.init(Path.java:172)
 at org.apache.hadoop.fs.Path.init(Path.java:94)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
 at 
 org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
 at 
 org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
 at 
 org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
 at 
 org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
 at

[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611

2014-09-05 Thread stack (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123139#comment-14123139
 ] 

stack commented on HBASE-11894:
---

bq. to pass any meta mutations while reporting region transition

How to describe these transitions we report the master such that it can make 
sense of them?  Is master recording these ops critical or is it ok if they are 
dropped?

 MetaEntries from coprocessor hooks in split and merge are not getting added 
 to hbase:meta after HBASE-11611
 ---

 Key: HBASE-11894
 URL: https://issues.apache.org/jira/browse/HBASE-11894
 Project: HBase
  Issue Type: Bug
  Components: Region Assignment
Reporter: rajeshbabu
Assignee: rajeshbabu
 Fix For: 2.0.0


 As part of HBASE-9249  HBASE-9489, added new hooks in split and merge which 
 take meta entries from coprocessor hooks if any. These meta entries helps to 
 ensure atomicity of split(merge) of regions by server and split(merge) of the 
 regions handled in coprocessors(This is required in secondary indexing case).
 After HBASE-11611 the meta entries are not getting added to meta both in 
 split and merge.
 {code}
 @MetaMutationAnnotation
 ListMutation metaEntries = new ArrayListMutation();
 if (rsCoprocessorHost != null) {
   if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b, 
 metaEntries)) {
 throw new IOException(Coprocessor bypassing regions  + 
 this.region_a +  
 + this.region_b +  merge.);
   }
   try {
 for (Mutation p : metaEntries) {
   HRegionInfo.parseRegionName(p.getRow());
 }
   } catch (IOException e) {
 LOG.error(Row key of mutation from coprocessor is not parsable as 
 region name.
 + Mutations from coprocessor should only be for hbase:meta 
 table., e);
 throw e;
   }
 }
 // This is the point of no return. Similar with SplitTransaction.
 // IF we reach the PONR then subsequent failures need to crash out this
 // regionserver
 this.journal.add(JournalEntry.PONR);
 // Add merged region and delete region_a and region_b
 // as an atomic update. See HBASE-7721. This update to hbase:meta makes 
 the region
 // will determine whether the region is merged or not in case of failures.
 // If it is successful, master will roll-forward, if not, master will
 // rollback
 if (services != null  
 !services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
 mergedRegionInfo, region_a.getRegionInfo(), 
 region_b.getRegionInfo())) {
   // Passed PONR, let SSH clean it up
   throw new IOException(Failed to notify master that merge passed PONR: 
 + region_a.getRegionInfo().getRegionNameAsString() +  and 
 + region_b.getRegionInfo().getRegionNameAsString());
 }
 {code}
 I think while reporting region state transition to master we need to pass 
 meta entries also so that we can add them to meta along with split or merge 
 updates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance

2014-09-05 Thread Lars Hofhansl (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lars Hofhansl updated HBASE-11898:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

comimtted to 0.98+

 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


 [ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter updated HBASE-11679:
---
Attachment: HBASE_11679_v5.patch

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch, HBASE_11679_v5.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


 [ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter updated HBASE-11679:
---
Status: Open  (was: Patch Available)

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch, HBASE_11679_v5.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


 [ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Carter updated HBASE-11679:
---
Status: Patch Available  (was: Open)

Found some issues with Jamon integration that didn't manifest in a simple 
compile.  Reverted two servlets back to master and cutting a new patch.

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch, HBASE_11679_v5.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method 
 return values, since that would violate criteria #1 above and break 
 compatibility.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (HBASE-11900) Optimization for incremental load reducer

2014-09-05 Thread Yi Deng (JIRA)

[
https://issues.apache.org/jira/browse/HBASE-11900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yi Deng updated HBASE-11900:

Description:
In current implementation, the key of reducer configured by
HFileOutputFormat.configureIncrementalLoad, is row. So, the reducer has to do
an in-memory sort before writing key values to the disk. When we meet with
some rows with a huge number of columns/versions, there could be OOM.

A better way is:
Use the KeyValue as the key, value can be a NullWritable. Partitioner
partitions the KeyValue only by it's row part. Set a sort comparator that sort
KeyValue with KeyValue.COMPARATOR

was:
In current implementation, the key of reducer configured by
HFileOutputFormat.configureIncrementalLoad, is row. So, the reducer has to do
an in-memory sort before writing key values to the disk. When we meet with
some rows with a huge number of comlumns/versions, there could be OOM.

A better way is:
Use the KeyValue as the key, value can be a NullWritable. Partioner partitions
the KeyValue only by it's row part. Set a sort comparator that sort KeyValue
with KeyValue.COMPARATOR

Optimization for incremental load reducer
-

Key: HBASE-11900
URL: https://issues.apache.org/jira/browse/HBASE-11900
Project: HBase
Issue Type: Improvement
Components: HFile, mapreduce
Affects Versions: 0.98.6
Reporter: Yi Deng
Priority: Minor

In current implementation, the key of reducer configured by
HFileOutputFormat.configureIncrementalLoad, is row. So, the reducer has to do
an in-memory sort before writing key values to the disk. When we meet with
some rows with a huge number of columns/versions, there could be OOM.
A better way is:
Use the KeyValue as the key, value can be a NullWritable. Partitioner
partitions the KeyValue only by it's row part. Set a sort comparator that
sort KeyValue with KeyValue.COMPARATOR

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123169#comment-14123169
]

Hadoop QA commented on HBASE-11772:
---

{color:green}+1 overall{color}. Here are the results of testing the latest
attachment

http://issues.apache.org/jira/secure/attachment/12666807/HBASE-11772-master-v2.patch
against trunk revision .
ATTACHMENT ID: 12666807

{color:green}+1 @author{color}. The patch does not contain any @author
tags.

{color:green}+1 tests included{color}. The patch appears to include 10 new
or modified tests.

{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.

{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.

{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.

{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100

{color:green}+1 site{color}. The mvn site goal succeeds with this patch.

{color:green}+1 core tests{color}. The patch passed unit tests in .

Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10726//console

This message is automatically generated.

Bulk load mvcc and seqId issues with native hfiles
--

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123186#comment-14123186
]

Ted Yu commented on HBASE-11772:

lgtm

nit: in the new test
{code}
+scanner = table.getScanner(scan);
{code}
scanner should be closed at the end.

Bulk load mvcc and seqId issues with native hfiles
--

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (HBASE-11862) Get rid of Writables in HTableDescriptor, HColumnDescriptor

2014-09-05 Thread Andrey Stepachev (JIRA)


 [ 
https://issues.apache.org/jira/browse/HBASE-11862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Stepachev reassigned HBASE-11862:


Assignee: Andrey Stepachev

 Get rid of Writables in HTableDescriptor, HColumnDescriptor
 ---

 Key: HBASE-11862
 URL: https://issues.apache.org/jira/browse/HBASE-11862
 Project: HBase
  Issue Type: Improvement
Reporter: Andrey Stepachev
Assignee: Andrey Stepachev
Priority: Minor
  Labels: beginner

 Currently we have protobuf for encoding this structures. Existence of 
 Writable is misleading and need to be removed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11889) Turn distributedLogReplay off by default for HBase1.0

2014-09-05 Thread Jeffrey Zhong (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-11889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123259#comment-14123259
 ] 

Jeffrey Zhong commented on HBASE-11889:
---

That's a good idea! [~enis] How do you think? Thanks.

 Turn distributedLogReplay off by default for HBase1.0
 -

 Key: HBASE-11889
 URL: https://issues.apache.org/jira/browse/HBASE-11889
 Project: HBase
  Issue Type: Bug
Affects Versions: 0.99.0
Reporter: Jeffrey Zhong

 The reason is that we want to support rolling upgrade while 
 distributedLogReplay supports rolling upgrade only since 
 hbase0.98.4(HBASE-11094). So keep it off by default for 1.0 because we want 
 to rolling upgrade from 0.98.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys

2014-09-05 Thread Jeffrey Zhong (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123280#comment-14123280
 ] 

Jeffrey Zhong commented on HBASE-10576:
---

[~rajesh23]  
{quote}
is this approach fine for you?
{quote}
I skimmed through the patch and basic idea looks good to me.  Thanks.

For stack's question
{quote}
I turn on this feature for a table? Or is it for whole cluster?
{quote}
In my opinion, it's better to turn on/off the feature in a table level because 
regions are always in changing state like split/merge. In addition, the feature 
requires 'shadow' region has the same boundary of the prime region. It's may be 
hard to turn on the feature for each individual region.


 Custom load balancer to co-locate the regions of two tables which are having 
 same split keys
 

 Key: HBASE-10576
 URL: https://issues.apache.org/jira/browse/HBASE-10576
 Project: HBase
  Issue Type: Sub-task
  Components: Balancer
Reporter: rajeshbabu
Assignee: rajeshbabu
 Attachments: HBASE-10536_v2.patch, HBASE-10576.patch, 
 HBASE-10576_shadow_regions_wip.patch


 To support local indexing both user table and index table should have same 
 split keys. This issue to provide custom balancer to colocate the regions of 
 two tables which are having same split keys. 
 This helps in Phoenix as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123312#comment-14123312
 ] 

Hudson commented on HBASE-11898:


SUCCESS: Integrated in HBase-TRUNK #5469 (See 
[https://builds.apache.org/job/HBase-TRUNK/5469/])
HBASE-11898 CoprocessorHost.Environment should cache class loader instance. 
(Vladimir Rodionov) (larsh: rev edac84567f83287f3b053e818401d7012a6fe44e)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible


[ 
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123322#comment-14123322
 ] 

Hadoop QA commented on HBASE-11679:
---

{color:red}-1 overall{color}.  Here are the results of testing the latest 
attachment 
  http://issues.apache.org/jira/secure/attachment/12666822/HBASE_11679_v5.patch
  against trunk revision .
  ATTACHMENT ID: 12666822

{color:green}+1 @author{color}.  The patch does not contain any @author 
tags.

{color:green}+1 tests included{color}.  The patch appears to include 492 
new or modified tests.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javac{color}.  The applied patch does not increase the 
total number of javac compiler warnings.

{color:green}+1 javadoc{color}.  The javadoc tool did not generate any 
warning messages.

{color:green}+1 findbugs{color}.  The patch does not introduce any new 
Findbugs (version 2.0.3) warnings.

{color:green}+1 release audit{color}.  The applied patch does not increase 
the total number of release audit warnings.

{color:red}-1 lineLengths{color}.  The patch introduces the following lines 
longer than 100:
+  private static MapString,ListPut createRsPutMap(RegionLocator 
htable, ListPut puts) throws IOException {
+  private static MapString,ListRow createRsRowMap(RegionLocator htable, 
ListRow rows) throws IOException {
+   * Test for {@link 
HFileOutputFormat#configureDataBlockEncoding(org.apache.hadoop.hbase.client.Table,
+   * Test for {@link 
HFileOutputFormat2#configureDataBlockEncoding(org.apache.hadoop.hbase.client.Table,

  {color:green}+1 site{color}.  The mvn site goal succeeds with this patch.

 {color:red}-1 core tests{color}.  The patch failed these unit tests:
   
org.apache.hadoop.hbase.replication.regionserver.TestReplicationThrottler

Test results: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//testReport/
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output: 
https://builds.apache.org/job/PreCommit-HBASE-Build/10728//console

This message is automatically generated.

 Replace HTable with HTableInterface where backwards-compatible
 --

 Key: HBASE-11679
 URL: https://issues.apache.org/jira/browse/HBASE-11679
 Project: HBase
  Issue Type: Improvement
Reporter: Carter
Assignee: Carter
 Attachments: HBASE_11679.patch, HBASE_11679.patch, 
 HBASE_11679_v3.patch, HBASE_11679_v4.patch, HBASE_11679_v5.patch


 This is a refactor to move more of the code towards using interfaces for 
 proper encapsulation of logic.
 The amount of code touched is large, but it should be fairly easy to review.  
 It changes variable declarations from HTable to HTableInterface where the 
 following holds:
 # The declaration being updated won't break assignment
 # The declaration change does not break the compile (eg trying to access 
 non-interface methods)
 The two main situations are to change something like this:
 {code}
 HTable h = new HTable(c, tn);
 {code}
 to
 {code}
 HTableInterface h = new HTable(c, tn);
 {code}
 and this:
 {code}
 public void doSomething(HTable h) { ... }
 {code}
 to this:
 {code}
 public void doSomething(HTableInterface h) { ... }
 {code}
 This gets most of the obvious cases out of the way and prepares for more 
 complicated interface refactors in the future.  In method signatures, I 
 changed parameters, but did _not_ change any public or protected method

[jira] [Updated] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jerry He updated HBASE-11772:
-
Attachment: HBASE-11772-master-v3.patch

Bulk load mvcc and seqId issues with native hfiles
--

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch, HBASE-11772-master-v3.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles

[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123351#comment-14123351
]

Jerry He commented on HBASE-11772:
--

Hi, [~tedyu]

Updated as attachment master-v3.

Bulk load mvcc and seqId issues with native hfiles
--

Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch,
HBASE-11772-master-v2.patch, HBASE-11772-master-v3.patch

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-8495) Change ownership of the directory to bulk load

2014-09-05 Thread Jeffrey Zhong (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123387#comment-14123387
 ] 

Jeffrey Zhong commented on HBASE-8495:
--

[~mbertozzi] We also see the same issue. It seems that the fix can only work 
for cases when bulk loaded files needs split as shown the code below. Thanks.
{code}
if (!lastKeyInRange) {
  ListLoadQueueItem lqis = splitStoreFile(item, table,
  startEndKeys.getFirst()[indexForCallable],
  startEndKeys.getSecond()[indexForCallable]);
  return lqis;
}
{code} 

 Change ownership of the directory to bulk load
 --

 Key: HBASE-8495
 URL: https://issues.apache.org/jira/browse/HBASE-8495
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.94.7, 0.95.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
 Fix For: 0.99.0, 0.94.21, 0.98.4

 Attachments: HBASE-8495-0.94-v0.patch


 To bulk load something you need to change the ownership of the data directory 
 to allow the hbase user to read and move the files, also in the split case 
 you must use the hbase user to run the LoadIncrementalHFiles tool, since 
 internally some directories _tmp are created to add the split reference 
 files.
 In a secure cluster, the SecureBulkLoadEndPoint will take care of this 
 problem by doing a chmod 777 on the directory to bulk load.
 NOTE that a chown is not possible since you must be a super user to change 
 the ownership, a change group may be possible but the user must be in the 
 hbase group... and anyway it will require a chmod to allow the group to 
 perform the move.
 {code}
 Caused by: org.apache.hadoop.security.AccessControlException: Permission 
 denied: user=hbase, access=WRITE, inode=/test/cf:th30z:supergroup:drwxr-xr-x
   at 
 org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)
 Caused by: 
 org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
 java.io.IOException: Exception in rename
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:928)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:340)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:414)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.


[ 
https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123401#comment-14123401
 ] 

Andrew Purtell commented on HBASE-11891:


Will commit to 0.98+ shortly.


 Introduce HBaseInterfaceAudience level to denote class names that appear in 
 configs.
 

 Key: HBASE-11891
 URL: https://issues.apache.org/jira/browse/HBASE-11891
 Project: HBase
  Issue Type: Improvement
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Minor
 Attachments: HBASE-11891.patch


 We have several classes that are private use as far as APIs are concerned, 
 but the class name appear in user provided config files.
 We should add an HBaseInterfaceAudience level CONFIG and use it to denote 
 these classes as LimitedPrivate.
 HBase contributors can then use this audience marking to see when changes to 
 a FQCN will require a release note to help upgrading users correct their 
 configuration files.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11858) Audit regionserver classes that are missing InterfaceAudience


[ 
https://issues.apache.org/jira/browse/HBASE-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123400#comment-14123400
 ] 

Andrew Purtell commented on HBASE-11858:


Will commit to 0.98+ shortly.

 Audit regionserver classes that are missing InterfaceAudience
 -

 Key: HBASE-11858
 URL: https://issues.apache.org/jira/browse/HBASE-11858
 Project: HBase
  Issue Type: Task
  Components: regionserver
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Minor
 Attachments: HBASE-11858.patch


 There are ~20 classes in regionserver that are missing InterfaceAudience 
 markers.
 Most are probably private, but HBASE-10378 has already hit atleast one that 
 should be LimitedPrivate(COPROC), so it's worth checking and cleaning things 
 up.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11786) Document web UI for tracking time spent in coprocessors


[ 
https://issues.apache.org/jira/browse/HBASE-11786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123404#comment-14123404
 ] 

Andrew Purtell commented on HBASE-11786:


+1

 Document web UI for tracking time spent in coprocessors
 ---

 Key: HBASE-11786
 URL: https://issues.apache.org/jira/browse/HBASE-11786
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors, documentation
Reporter: Misty Stanley-Jones
Assignee: Misty Stanley-Jones
Priority: Minor
 Attachments: HBASE-11786.patch, HBASE-11786.patch, HBASE-11786.patch, 
 HBASE-11786.patch, HBASE-11786.patch, coprocessor_stats.png






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source


[ 
https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123409#comment-14123409
 ] 

Andrew Purtell commented on HBASE-11885:


What SCM branches have been tested with this? Master? Can we check if this 
works with master, branch-1, and 0.98? If so I will defer to those with Docker 
expertise and commit this to the respective branches.

 Provide a Dockerfile to easily build and run HBase from source
 --

 Key: HBASE-11885
 URL: https://issues.apache.org/jira/browse/HBASE-11885
 Project: HBase
  Issue Type: New Feature
Reporter: Dima Spivak
Assignee: Dima Spivak
 Attachments: HBASE-11885.patch, HBASE-11885_v2.patch


 [A recent email to 
 dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E]
  highlighted the difficulty that new users can face in getting HBase compiled 
 from source and running locally. I'd like to provide a Dockerfile that would 
 allow anyone with Docker running on a machine with a reasonably current Linux 
 kernel to do so with ease.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123410#comment-14123410
 ] 

Hudson commented on HBASE-11898:


FAILURE: Integrated in HBase-0.98 #498 (See 
[https://builds.apache.org/job/HBase-0.98/498/])
HBASE-11898 CoprocessorHost.Environment should cache class loader instance. 
(Vladimir Rodionov) (larsh: rev 8ed1ff0979f8e96c90f6101ebe9fb9bd0f1c917d)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-8495) Change ownership of the directory to bulk load

2014-09-05 Thread Matteo Bertozzi (JIRA)


[ 
https://issues.apache.org/jira/browse/HBASE-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123442#comment-14123442
 ] 

Matteo Bertozzi commented on HBASE-8495:


correct the fix is only on the dir/files that the tool generates.
you need to open the directory if you want allow to read files from that dir, i 
think is documented.
this jira solves the problem when you allow the tool to pick the files, but the 
generated files were not loadable.
Adding a chmod 777 doesn't look the right choice, since not everyone may want 
to open the dir to everyone

 Change ownership of the directory to bulk load
 --

 Key: HBASE-8495
 URL: https://issues.apache.org/jira/browse/HBASE-8495
 Project: HBase
  Issue Type: Improvement
  Components: mapreduce
Affects Versions: 0.94.7, 0.95.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Priority: Trivial
 Fix For: 0.99.0, 0.94.21, 0.98.4

 Attachments: HBASE-8495-0.94-v0.patch


 To bulk load something you need to change the ownership of the data directory 
 to allow the hbase user to read and move the files, also in the split case 
 you must use the hbase user to run the LoadIncrementalHFiles tool, since 
 internally some directories _tmp are created to add the split reference 
 files.
 In a secure cluster, the SecureBulkLoadEndPoint will take care of this 
 problem by doing a chmod 777 on the directory to bulk load.
 NOTE that a chown is not possible since you must be a super user to change 
 the ownership, a change group may be possible but the user must be in the 
 hbase group... and anyway it will require a chmod to allow the group to 
 perform the move.
 {code}
 Caused by: org.apache.hadoop.security.AccessControlException: Permission 
 denied: user=hbase, access=WRITE, inode=/test/cf:th30z:supergroup:drwxr-xr-x
   at 
 org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:205)
 Caused by: 
 org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException): 
 java.io.IOException: Exception in rename
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.rename(HRegionFileSystem.java:928)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.commitStoreFile(HRegionFileSystem.java:340)
   at 
 org.apache.hadoop.hbase.regionserver.HRegionFileSystem.bulkLoadStoreFile(HRegionFileSystem.java:414)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HBASE-11904) Coprocessor for AlterTable

2014-09-05 Thread Hyeonjong (JIRA)

Hyeonjong created HBASE-11904:
-

 Summary: Coprocessor for AlterTable
 Key: HBASE-11904
 URL: https://issues.apache.org/jira/browse/HBASE-11904
 Project: HBase
  Issue Type: Sub-task
  Components: Coprocessors
Affects Versions: 0.89-fb
Reporter: Hyeonjong
Priority: Minor
 Fix For: 0.89-fb


Added Endpoint to support AlterTable query



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123449#comment-14123449
 ] 

Hudson commented on HBASE-11898:


SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #472 (See 
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/472/])
HBASE-11898 CoprocessorHost.Environment should cache class loader instance. 
(Vladimir Rodionov) (larsh: rev 8ed1ff0979f8e96c90f6101ebe9fb9bd0f1c917d)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance


[ 
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123472#comment-14123472
 ] 

Hudson commented on HBASE-11898:


SUCCESS: Integrated in HBase-1.0 #154 (See 
[https://builds.apache.org/job/HBase-1.0/154/])
HBASE-11898 CoprocessorHost.Environment should cache class loader instance. 
(Vladimir Rodionov) (larsh: rev b117a8d3b477b569b377f2df9bb4a8a2c5329095)
* 
hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java


 CoprocessorHost.Environment should cache class loader instance
 --

 Key: HBASE-11898
 URL: https://issues.apache.org/jira/browse/HBASE-11898
 Project: HBase
  Issue Type: Bug
  Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
 Fix For: 0.99.0, 2.0.0, 0.98.7

 Attachments: HBASE_11898.patch


 HBASE-9941 fix causes additional overhead... 
 It looks like for every invocation it does a getClassLoader call.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (HBASE-11772) Bulk load mvcc and seqId issues with native hfiles