[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source
[ https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121002#comment-14121002 ] Dima Spivak commented on HBASE-11885: - Thanks for the feedback, Srikanth; I'll update documentation tomorrow. The error you saw is because of what seems to be buggy implementation of the devicemapper driver. Unfortunately, this tends to get fixed by just rerunning docker build (sometimes repeatedly). The fact that the image didn't complete is why the second error about being unable to find hbase_docker showed up (Docker always tries to pull images from the central registry if they aren't present locally). Provide a Dockerfile to easily build and run HBase from source -- Key: HBASE-11885 URL: https://issues.apache.org/jira/browse/HBASE-11885 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Attachments: HBASE-11885.patch [A recent email to dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E] highlighted the difficulty that new users can face in getting HBase compiled from source and running locally. I'd like to provide a Dockerfile that would allow anyone with Docker running on a machine with a reasonably current Linux kernel to do so with ease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121038#comment-14121038 ] Mikhail Antonov commented on HBASE-11165: - [~stack] that's kind of hard to compare the relative complexity without proposed detailed designs for both, I believe both options worth research and comparison (also we should be able to combine them). W.r.t multiple masters, we do try to streamline the design (with that incremental approach for cold-warm-hot- active-active transition) to make it less intrusive change + provide positive side effects like simplifying current workflow management around region splits, log splitting etc. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121051#comment-14121051 ] Elliott Clark commented on HBASE-11165: --- bq.There is a largish gap at the moment. Then we should be focusing there on places where we have slowness. Not on adding more complexity. It seems like everyone is rushing to following the example of HDFS to federation (that's what split meta, split masters gets us). I for one am terrified of going that way. From where I sit that was just a failure and following it isn't something I'm ready to do without tangible benefits. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11858) Audit regionserver classes that are missing InterfaceAudience
[ https://issues.apache.org/jira/browse/HBASE-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121059#comment-14121059 ] Anoop Sam John commented on HBASE-11858: +1 after going through the RB patch Audit regionserver classes that are missing InterfaceAudience - Key: HBASE-11858 URL: https://issues.apache.org/jira/browse/HBASE-11858 Project: HBase Issue Type: Task Components: regionserver Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11858.patch There are ~20 classes in regionserver that are missing InterfaceAudience markers. Most are probably private, but HBASE-10378 has already hit atleast one that should be LimitedPrivate(COPROC), so it's worth checking and cleaning things up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11777) Find a way to set sequenceId on Cells on the server
[ https://issues.apache.org/jira/browse/HBASE-11777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11777: --- Resolution: Fixed Fix Version/s: (was: 0.98.7) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) When trying to rebase this for 0.98, most of the patch parts not at all needed in 0.98. Whichever part is applicable that also might be not really needed as we can not remove KeyValueUtil#ensureKeyValue there. In 0.98 still the MemStore, Store in write paths and Scanners in read path deal with KeyValue only.Also for HBASE-11805 to be in 0.98 , this issue is not needed. (In branch-1+ this was a mandatory item for HBASE-11805) Am closing this issue with branch-1+ commit. Find a way to set sequenceId on Cells on the server --- Key: HBASE-11777 URL: https://issues.apache.org/jira/browse/HBASE-11777 Project: HBase Issue Type: Improvement Affects Versions: 0.99.0 Reporter: ramkrishna.s.vasudevan Assignee: Anoop Sam John Fix For: 0.99.0, 2.0.0 Attachments: CellWithSequenceNumber.java, HBASE-11777.patch, HBASE-11777_V2.patch, HBASE-11777_V3.patch, HBASE-11777_V4.patch, HBASE-11777_V5.patch Over in HBASE-11591 there was a need to set the sequenceId of the HFile to the bulk loaded KVs. Since we are trying to use the concept of Cells in the read path if we need to use setSequenceId(), then the Cell has to be converted to KV and only KeyValue impl has the accessor setSequenceId(). [~anoop.hbase] suggested if we can use a Server side impl of Cell and have these accessors in them. This JIRA aims to solve this and see the related code changes that needs to be carried out for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11892) configs contain stale entries
Sean Busbey created HBASE-11892:
---
Summary: configs contain stale entries
Key: HBASE-11892
URL: https://issues.apache.org/jira/browse/HBASE-11892
Project: HBase
Issue Type: Bug
Components: documentation
Reporter: Sean Busbey
Priority: Trivial
Configuration details that need to be cleaned up
* the documentation around configuring cleaner plugins have stale class names
for customizing behavior.
** {{hbase.master.logcleaner.plugins}} has LogCleanerDelegate and I think the
correct class is BaseLogCleanerDelegate.
** {{hbase.master.hfilecleaner.plugin}} has HFileCleanerDelegate instead of
BaseHFileCleanerDelegate.
* {{hbase.rpc.server.engine}} doesn't appear anywhere other than
hdfs-default.xml and the classes it references were removed by HBASE-8214
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11893) RowTooBigException should be in hbase-client module
Sean Busbey created HBASE-11893: --- Summary: RowTooBigException should be in hbase-client module Key: HBASE-11893 URL: https://issues.apache.org/jira/browse/HBASE-11893 Project: HBase Issue Type: Bug Components: Client Reporter: Sean Busbey Priority: Minor RowTooBigException is Public get thrown from client calls. It should be in hbase-client instead of hbase-server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-8674) JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
[
https://issues.apache.org/jira/browse/HBASE-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121136#comment-14121136
]
Nicolas Liochon commented on HBASE-8674:
+1 to everything that has been said. (yes, we need to keep Gary's repo up for
the previous releases.)
JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
Key: HBASE-8674
URL: https://issues.apache.org/jira/browse/HBASE-8674
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.98.0, 0.94.8, 0.95.1
Reporter: Andrew Purtell
Assignee: Gary Helmling
Attachments: HBASE-8674.patch
people.apache.org cannot currently host personal or transient Maven repos.
{noformat}
$ curl --connect-timeout 60 -v
http://people.apache.org/~garyh/mvn/org/apache/maven/plugins/maven-remote-resources-plugin/1.4/maven-remote-resources-plugin-1.4.pom
* About to connect() to people.apache.org port 80 (#0)
* Trying 140.211.11.9...
* Connection timed out after 60064 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 60064 milliseconds
{noformat}
All builds are at the moment broken if the HBase custom junit or surefire
jars are not already in cache. Even if this is a temporary condition, we
should find a new home for these artifacts, upgrade to versions that include
our submitted changes (if any), or fall back to release versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
rajeshbabu created HBASE-11894:
--
Summary: MetaEntries from coprocessor hooks in split and merge are
not getting added to hbase:meta after HBASE-11611
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in split
and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions + this.region_a
+
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes the
region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(), region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass meta
entries also so that we can add them to meta along with split or merge updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.
[ https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11891: Attachment: HBASE-11891.patch Introduce HBaseInterfaceAudience level to denote class names that appear in configs. Key: HBASE-11891 URL: https://issues.apache.org/jira/browse/HBASE-11891 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11891.patch We have several classes that are private use as far as APIs are concerned, but the class name appear in user provided config files. We should add an HBaseInterfaceAudience level CONFIG and use it to denote these classes as LimitedPrivate. HBase contributors can then use this audience marking to see when changes to a FQCN will require a release note to help upgrading users correct their configuration files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
[
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121139#comment-14121139
]
rajeshbabu commented on HBASE-11894:
ping [~jxiang], what do you say?
MetaEntries from coprocessor hooks in split and merge are not getting added
to hbase:meta after HBASE-11611
---
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in
split and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions +
this.region_a +
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes
the region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(),
region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass
meta entries also so that we can add them to meta along with split or merge
updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.
[ https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-11891: Status: Patch Available (was: Open) Introduce HBaseInterfaceAudience level to denote class names that appear in configs. Key: HBASE-11891 URL: https://issues.apache.org/jira/browse/HBASE-11891 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11891.patch We have several classes that are private use as far as APIs are concerned, but the class name appear in user provided config files. We should add an HBaseInterfaceAudience level CONFIG and use it to denote these classes as LimitedPrivate. HBase contributors can then use this audience marking to see when changes to a FQCN will require a release note to help upgrading users correct their configuration files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.
[
https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121155#comment-14121155
]
Hadoop QA commented on HBASE-11891:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666440/HBASE-11891.patch
against trunk revision .
ATTACHMENT ID: 12666440
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10710//console
This message is automatically generated.
Introduce HBaseInterfaceAudience level to denote class names that appear in
configs.
Key: HBASE-11891
URL: https://issues.apache.org/jira/browse/HBASE-11891
Project: HBase
Issue Type: Improvement
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Minor
Attachments: HBASE-11891.patch
We have several classes that are private use as far as APIs are concerned,
but the class name appear in user provided config files.
We should add an HBaseInterfaceAudience level CONFIG and use it to denote
these classes as LimitedPrivate.
HBase contributors can then use this audience marking to see when changes to
a FQCN will require a release note to help upgrading users correct their
configuration files.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
[ https://issues.apache.org/jira/browse/HBASE-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11805: --- Attachment: HBASE-11805_0.98.patch 0.98 version of the patch. Ping [~apurtell] Same trunk patch has to go into branch-1 as well. Ping [~enis] KeyValue to Cell Convert in WALEdit APIs Key: HBASE-11805 URL: https://issues.apache.org/jira/browse/HBASE-11805 Project: HBase Issue Type: Improvement Components: wal Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE-11805.patch, HBASE-11805_0.98.patch, HBASE-11805_V2.patch In almost all other main interface class/APIs we have changed KeyValue to Cell. But missing in WALEdit. This is public marked for Replication (Well it should be for CP also) These 2 APIs deal with KVs add(KeyValue kv) ArrayListKeyValue getKeyValues() Suggest deprecate them and add for 0.98 add(Cell kv) ListCell getCells() And just replace from 1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
[
https://issues.apache.org/jira/browse/HBASE-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121195#comment-14121195
]
Hadoop QA commented on HBASE-11805:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666448/HBASE-11805_0.98.patch
against trunk revision .
ATTACHMENT ID: 12666448
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 24 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10711//console
This message is automatically generated.
KeyValue to Cell Convert in WALEdit APIs
Key: HBASE-11805
URL: https://issues.apache.org/jira/browse/HBASE-11805
Project: HBase
Issue Type: Improvement
Components: wal
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: HBASE-11805.patch, HBASE-11805_0.98.patch,
HBASE-11805_V2.patch
In almost all other main interface class/APIs we have changed KeyValue to
Cell. But missing in WALEdit. This is public marked for Replication (Well it
should be for CP also)
These 2 APIs deal with KVs
add(KeyValue kv)
ArrayListKeyValue getKeyValues()
Suggest deprecate them and add for 0.98
add(Cell kv)
ListCell getCells()
And just replace from 1.0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11895) Change Writer/Readers factory to create CellComparators rather than KVComparator
ramkrishna.s.vasudevan created HBASE-11895: -- Summary: Change Writer/Readers factory to create CellComparators rather than KVComparator Key: HBASE-11895 URL: https://issues.apache.org/jira/browse/HBASE-11895 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan In HBASE-10800 we would try to create CellComparators so that later when we deal with cells we could add the comparison logic into the comparators of those cells. This JIRA is aimed at changing the writers and readers factory methods where we always accept a KVComparator. It would be better if we could create a CellComparator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11869) Support snapshot owner
[ https://issues.apache.org/jira/browse/HBASE-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121284#comment-14121284 ] Liu Shaohui commented on HBASE-11869: - [~tedyu] [~apurtell] [~misty] Could you help to review this issue? We need another +1 at least. Thanks. Support snapshot owner -- Key: HBASE-11869 URL: https://issues.apache.org/jira/browse/HBASE-11869 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-11869-trunk-v1.diff, HBASE-11869-trunk-v3.diff In current codebase, the table snapshot operations only can be done by the global admin , not by the table admin. There is a multi-tenant hbase cluster, each table has different snapshot policies, eg: do snapshot per week, or snapshot after the new data are imported. We want to release the snapshot permission to each table admin. According to [~mbertozzi]'s suggestion, we implement the snapshot owner feature. * The user with table admin permission can create snapshot and the owner of this snapshot is this user. * The owner of snapshot can delete and restore the snapshot. * Only the user with global admin permission can clone a snapshot, for this operation creates a new table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11877) Make TableSplit more readable
[
https://issues.apache.org/jira/browse/HBASE-11877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121287#comment-14121287
]
Liu Shaohui commented on HBASE-11877:
-
[~jmspaggi] [~stack]
What's your suggestions about the new patch?
Make TableSplit more readable
-
Key: HBASE-11877
URL: https://issues.apache.org/jira/browse/HBASE-11877
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-11877-trunk-v1.diff, HBASE-11877-trunk-v2.diff
When debugging MR jobs reading from hbase table, it's import to figure out
which region a map task is reading from.
But the table split object is hard to read.
eg:
{code}
2014-09-01 20:58:39,783 INFO [main] org.apache.hadoop.mapred.MapTask:
Processing split: lg-hadoop-prc-st40.bj:,0
{code}
See: TableSplit.java
{code}
@Override
public String toString() {
return m_regionLocation + : +
Bytes.toStringBinary(m_startRow) + , + Bytes.toStringBinary(m_endRow);
}
{code}
We should make it more readable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
Ashish Singhi created HBASE-11896:
-
Summary: LoadIncrementalHFiles fails in secure mode if the
namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using table
name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11339: --- Affects Version/s: 2.0.0 Fix Version/s: 2.0.0 HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: 2.0.0 Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user guide_v2.docx, MOB user guide_v3.docx, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11897) Add append and remove peer table-cfs cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-11897: Summary: Add append and remove peer table-cfs cmds for replication (was: Add append and remove table-cfs cmds for replication) Add append and remove peer table-cfs cmds for replication - Key: HBASE-11897 URL: https://issues.apache.org/jira/browse/HBASE-11897 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Priority: Minor HBASE-8751 introduces the tables/table-column families config for a replication peer. It's very flexible for practical replication in hbase clusters. But it is easy to make mistakes during add or remove a table/table-column family for a existing peer, especially when the table-cfs is very long, for we need to copy the current table-cfs of the peer first, and then add or remove a table/table-column family to/from the table-cfs, at last set back the table-cfs using the cmd: set_peer_tableCFs. So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs to do the operation of adding and removing a table/table-column family. They are useful operation tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11897) Add append and remove table-cfs cmds for replication
Liu Shaohui created HBASE-11897: --- Summary: Add append and remove table-cfs cmds for replication Key: HBASE-11897 URL: https://issues.apache.org/jira/browse/HBASE-11897 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Priority: Minor HBASE-8751 introduces the tables/table-column families config for a replication peer. It's very flexible for practical replication in hbase clusters. But it is easy to make mistakes during add or remove a table/table-column family for a existing peer, especially when the table-cfs is very long, for we need to copy the current table-cfs of the peer first, and then add or remove a table/table-column family to/from the table-cfs, at last set back the table-cfs using the cmd: set_peer_tableCFs. So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs to do the operation of adding and removing a table/table-column family. They are useful operation tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11643) Read and write MOB in HBase
[ https://issues.apache.org/jira/browse/HBASE-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11643: --- Resolution: Fixed Fix Version/s: hbase-11339 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I have created a new version, hbase-11339, and a new branch with the same name off of master in the repo. We will commit changes under the HBASE-11339 umbrella to this branch. The last commit before this branch has this hash bcfc6d65af. I have committed this to the hbase-11339 branch. Thanks for working through many reviews Jingcheng and thanks for the reviews Anoop and Ram. Read and write MOB in HBase --- Key: HBASE-11643 URL: https://issues.apache.org/jira/browse/HBASE-11643 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11643-V10.diff, HBASE-11643-V11.diff, HBASE-11643-V12.diff, HBASE-11643-V13.diff, HBASE-11643-V14.diff, HBASE-11643-V15.diff, HBASE-11643-V16.diff, HBASE-11643-V17.diff, HBASE-11643-V18.diff, HBASE-11643-V19.diff, HBASE-11643-V2.diff, HBASE-11643-V20.diff, HBASE-11643-V3.diff, HBASE-11643-V4.diff, HBASE-11643-V5.diff, HBASE-11643-V6.diff, HBASE-11643-V7.diff, HBASE-11643-V8.diff, HBASE-11643-V9.diff, HBase-11643.diff The read/write MOB in HBase are implemented in this JIRA. Normally, the Cells are saved in the MemStore, and flushed to the HFiles when necessary. For MOB, the Cells are saved in the MemStore as well, but they're flushed to elsewhere out of HBase in the format of HFiles. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121312#comment-14121312 ] Jonathan Hsieh commented on HBASE-11339: I have created a new version in the jira, hbase-11339, and a new branch with the same name off of master in the repo. We will commit changes under the HBASE-11339 umbrella to this branch. The last commit before this branch has this hash bcfc6d65af. HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: 2.0.0 Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user guide_v2.docx, MOB user guide_v3.docx, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11339) HBase MOB
[ https://issues.apache.org/jira/browse/HBASE-11339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11339: --- Fix Version/s: (was: 2.0.0) hbase-11339 HBase MOB - Key: HBASE-11339 URL: https://issues.apache.org/jira/browse/HBASE-11339 Project: HBase Issue Type: Umbrella Components: regionserver, Scanners Affects Versions: 2.0.0 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBase MOB Design-v2.pdf, HBase MOB Design-v3.pdf, HBase MOB Design-v4.pdf, HBase MOB Design.pdf, MOB user guide.docx, MOB user guide_v2.docx, MOB user guide_v3.docx, hbase-11339-in-dev.patch It's quite useful to save the medium binary data like images, documents into Apache HBase. Unfortunately directly saving the binary MOB(medium object) to HBase leads to a worse performance since the frequent split and compaction. In this design, the MOB data are stored in an more efficient way, which keeps a high write/read performance and guarantees the data consistency in Apache HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11897) Add append and remove peer table-cfs cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-11897: Attachment: HBASE-11897-trunk-v1.diff Patch for the trunk Add append and remove peer table-cfs cmds for replication - Key: HBASE-11897 URL: https://issues.apache.org/jira/browse/HBASE-11897 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Priority: Minor Attachments: HBASE-11897-trunk-v1.diff HBASE-8751 introduces the tables/table-column families config for a replication peer. It's very flexible for practical replication in hbase clusters. But it is easy to make mistakes during add or remove a table/table-column family for a existing peer, especially when the table-cfs is very long, for we need to copy the current table-cfs of the peer first, and then add or remove a table/table-column family to/from the table-cfs, at last set back the table-cfs using the cmd: set_peer_tableCFs. So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs to do the operation of adding and removing a table/table-column family. They are useful operation tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11897) Add append and remove peer table-cfs cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121325#comment-14121325 ] Liu Shaohui commented on HBASE-11897: - Please help to review it at https://reviews.apache.org/r/25339. Thanks Add append and remove peer table-cfs cmds for replication - Key: HBASE-11897 URL: https://issues.apache.org/jira/browse/HBASE-11897 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Priority: Minor Attachments: HBASE-11897-trunk-v1.diff HBASE-8751 introduces the tables/table-column families config for a replication peer. It's very flexible for practical replication in hbase clusters. But it is easy to make mistakes during add or remove a table/table-column family for a existing peer, especially when the table-cfs is very long, for we need to copy the current table-cfs of the peer first, and then add or remove a table/table-column family to/from the table-cfs, at last set back the table-cfs using the cmd: set_peer_tableCFs. So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs to do the operation of adding and removing a table/table-column family. They are useful operation tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11897) Add append and remove peer table-cfs cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-11897: Assignee: Liu Shaohui Affects Version/s: 2.0.0 Status: Patch Available (was: Open) Add append and remove peer table-cfs cmds for replication - Key: HBASE-11897 URL: https://issues.apache.org/jira/browse/HBASE-11897 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-11897-trunk-v1.diff HBASE-8751 introduces the tables/table-column families config for a replication peer. It's very flexible for practical replication in hbase clusters. But it is easy to make mistakes during add or remove a table/table-column family for a existing peer, especially when the table-cfs is very long, for we need to copy the current table-cfs of the peer first, and then add or remove a table/table-column family to/from the table-cfs, at last set back the table-cfs using the cmd: set_peer_tableCFs. So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs to do the operation of adding and removing a table/table-column family. They are useful operation tools. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11886) The creator of the table should have all permissions on the table
[ https://issues.apache.org/jira/browse/HBASE-11886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121334#comment-14121334 ] Devaraj Das commented on HBASE-11886: - Thanks to you too, [~apurtell], for the unit test. The creator of the table should have all permissions on the table - Key: HBASE-11886 URL: https://issues.apache.org/jira/browse/HBASE-11886 Project: HBase Issue Type: Bug Affects Versions: 0.98.3 Reporter: Devaraj Das Assignee: Devaraj Das Priority: Critical Fix For: 0.99.0, 2.0.0, 0.98.6 Attachments: 11886-1.txt, HBASE-11886-0.98.patch, HBASE-11886.patch In our testing of 0.98.4 with security ON, we found that table creator doesn't have RWXCA on the created table. Instead, the user representing the HBase daemon gets all permissions. Due to this the table creator can't write to the table he just created. I am suspecting HBASE-11275 introduced the problem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11869) Support snapshot owner
[ https://issues.apache.org/jira/browse/HBASE-11869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121375#comment-14121375 ] Anoop Sam John commented on HBASE-11869: Overall looks good Put one question and a small nit on RB Support snapshot owner -- Key: HBASE-11869 URL: https://issues.apache.org/jira/browse/HBASE-11869 Project: HBase Issue Type: Improvement Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Fix For: 2.0.0 Attachments: HBASE-11869-trunk-v1.diff, HBASE-11869-trunk-v3.diff In current codebase, the table snapshot operations only can be done by the global admin , not by the table admin. There is a multi-tenant hbase cluster, each table has different snapshot policies, eg: do snapshot per week, or snapshot after the new data are imported. We want to release the snapshot permission to each table admin. According to [~mbertozzi]'s suggestion, we implement the snapshot owner feature. * The user with table admin permission can create snapshot and the owner of this snapshot is this user. * The owner of snapshot can delete and restore the snapshot. * Only the user with global admin permission can clone a snapshot, for this operation creates a new table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashish Singhi updated HBASE-11896:
--
Attachment: HBASE-11896.patch
Attaching a simple patch.
The patch will replace all ':' with empty string if any exists in the path.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11896:
---
Status: Patch Available (was: Open)
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11896:
---
Attachment: 11896-v1.txt
Patch v1 replaces : in tablename with underscore.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121387#comment-14121387
]
Ashish Singhi commented on HBASE-11896:
---
In the patch also removed unused arguments from method signature.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121391#comment-14121391
]
Matteo Bertozzi commented on HBASE-11896:
-
can you add a unit test?
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 1.0.0, 2.0.0
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
[
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121395#comment-14121395
]
Anoop Sam John commented on HBASE-11894:
Region states for the CP handled regions also would have changed. These are not
getting reported? Than handling meta entry by CP and passing that to Master,
the CP handled region state change also can be passed to Master along with that
of the user table regions(?)
MetaEntries from coprocessor hooks in split and merge are not getting added
to hbase:meta after HBASE-11611
---
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in
split and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions +
this.region_a +
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes
the region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(),
region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass
meta entries also so that we can add them to meta along with split or merge
updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
[
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121394#comment-14121394
]
Anoop Sam John commented on HBASE-11894:
Region states for the CP handled regions also would have changed. These are not
getting reported? Than handling meta entry by CP and passing that to Master,
the CP handled region state change also can be passed to Master along with that
of the user table regions(?)
MetaEntries from coprocessor hooks in split and merge are not getting added
to hbase:meta after HBASE-11611
---
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in
split and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions +
this.region_a +
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes
the region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(),
region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass
meta entries also so that we can add them to meta along with split or merge
updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
[
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11894:
---
Comment: was deleted
(was: Region states for the CP handled regions also would have changed. These
are not getting reported? Than handling meta entry by CP and passing that to
Master, the CP handled region state change also can be passed to Master along
with that of the user table regions(?))
MetaEntries from coprocessor hooks in split and merge are not getting added
to hbase:meta after HBASE-11611
---
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in
split and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions +
this.region_a +
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes
the region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(),
region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass
meta entries also so that we can add them to meta along with split or merge
updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11609) LoadIncrementalHFiles fails if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11609:
---
Fix Version/s: (was: 1.0.0)
0.99.0
LoadIncrementalHFiles fails if the namespace is specified
-
Key: HBASE-11609
URL: https://issues.apache.org/jira/browse/HBASE-11609
Project: HBase
Issue Type: Bug
Components: Client
Affects Versions: 1.0.0, 0.98.4, 2.0.0
Reporter: Matteo Bertozzi
Assignee: Matteo Bertozzi
Fix For: 0.99.0, 0.98.5, 2.0.0
Attachments: HBASE-11609-unittest.patch, HBASE-11609-v0.patch,
HBASE-11609-v1.patch
from Jianshi Huang on the user list
trying to bulk load a table in a namespace, like:
$ hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles test/
foo:testtb
we get an exception
{code}
2014-07-29 19:59:53,373 ERROR [main] mapreduce.LoadIncrementalHFiles:
Unexpected execution exception during splitting
java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException:
java.net.URISyntaxException: Relative path in absolute URI:
foo:testtb,1.bottom
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:188)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.groupOrSplitPhase(LoadIncrementalHFiles.java:449)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:304)
at
org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:899)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
{code}
The problem is related to the ':' symbol going to the file path. the simple
fix is to replace the current LoadIncrementalHFiles.getUniqueName()
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anoop Sam John updated HBASE-11896:
---
Fix Version/s: (was: 1.0.0)
0.98.7
0.99.0
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121403#comment-14121403
]
Anoop Sam John commented on HBASE-11896:
Nit : Better to use TableName.getNameAsString() than TableName.toString() -
Even if both return the same String the usage of 1st one looks better.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11897) Add append and remove peer table-cfs cmds for replication
[
https://issues.apache.org/jira/browse/HBASE-11897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121420#comment-14121420
]
Hadoop QA commented on HBASE-11897:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666472/HBASE-11897-trunk-v1.diff
against trunk revision .
ATTACHMENT ID: 12666472
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestPerTableCFReplication
org.apache.hadoop.hbase.client.replication.TestReplicationAdmin
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10712//console
This message is automatically generated.
Add append and remove peer table-cfs cmds for replication
-
Key: HBASE-11897
URL: https://issues.apache.org/jira/browse/HBASE-11897
Project: HBase
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Attachments: HBASE-11897-trunk-v1.diff
HBASE-8751 introduces the tables/table-column families config for a
replication peer. It's very flexible for practical replication in hbase
clusters.
But it is easy to make mistakes during add or remove a table/table-column
family for a existing peer, especially when the table-cfs is very long, for
we need to copy the current table-cfs of the peer first, and then add or
remove a table/table-column family to/from the table-cfs, at last set back
the table-cfs using the cmd: set_peer_tableCFs.
So we implements two new cmds: append_peer_tableCFs and remove_peer_tableCFs
to do the operation of adding and removing a table/table-column family. They
are useful operation tools.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121421#comment-14121421
]
Hadoop QA commented on HBASE-11896:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666484/11896-v1.txt
against trunk revision .
ATTACHMENT ID: 12666484
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestClassFinder
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10713//console
This message is automatically generated.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121445#comment-14121445
]
Ashish Singhi commented on HBASE-11896:
---
I am a beginner here. Can some one please give me reference to write a test for
LoadIncrementalHFlies in secure mode.
I checked TestSecureLoadIncrementalHFiles but here we are setting
*hbase.client.userprovider.class* to
*HadoopSecurityEnabledUserProviderForTesting* which returns false on
isHadoopSecurityEnabled call.
So the buggy code will never get executed.
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121448#comment-14121448
]
Ashish Singhi commented on HBASE-11896:
---
bq.which returns false on isHadoopSecurityEnabled call
Sorry for typo,
I meant on isHBaseSecurityEnabled call
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121455#comment-14121455 ] stack commented on HBASE-11165: --- [~mantonov] bq. ...that's kind of hard to compare the relative complexity without proposed detailed designs for both Hopefully we don't need detailed design. I think a sketch will be sufficient. TODO. [~eclark] bq. Then we should be focusing there on places where we have slowness. 'slowness' is but one of the dimensions that needs addressing. There is also 'size' -- size in HDFS, size of cache -- as well as availability No to federation. I don't think we need to split master. Is the failure you refer to our having a root and not making use of it? Let me post something for folks to skewer (listing tangible benefit).. Hopefully tonight (out for the day). Good stuff Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11894) MetaEntries from coprocessor hooks in split and merge are not getting added to hbase:meta after HBASE-11611
[
https://issues.apache.org/jira/browse/HBASE-11894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121478#comment-14121478
]
rajeshbabu commented on HBASE-11894:
The region state changes need not pass [~anoop.hbase]. They will be properly
handled by core itself. We need to pass split or merge updates of the regions
handled in CP(but it's completely depend on CP implementation).
MetaEntries from coprocessor hooks in split and merge are not getting added
to hbase:meta after HBASE-11611
---
Key: HBASE-11894
URL: https://issues.apache.org/jira/browse/HBASE-11894
Project: HBase
Issue Type: Bug
Components: Region Assignment
Reporter: rajeshbabu
Fix For: 2.0.0
As part of HBASE-9249 HBASE-9489, added new hooks in split and merge which
take meta entries from coprocessor hooks if any. These meta entries helps to
ensure atomicity of split(merge) of regions by server and split(merge) of the
regions handled in coprocessors(This is required in secondary indexing case).
After HBASE-11611 the meta entries are not getting added to meta both in
split and merge.
{code}
@MetaMutationAnnotation
ListMutation metaEntries = new ArrayListMutation();
if (rsCoprocessorHost != null) {
if (rsCoprocessorHost.preMergeCommit(this.region_a, this.region_b,
metaEntries)) {
throw new IOException(Coprocessor bypassing regions +
this.region_a +
+ this.region_b + merge.);
}
try {
for (Mutation p : metaEntries) {
HRegionInfo.parseRegionName(p.getRow());
}
} catch (IOException e) {
LOG.error(Row key of mutation from coprocessor is not parsable as
region name.
+ Mutations from coprocessor should only be for hbase:meta
table., e);
throw e;
}
}
// This is the point of no return. Similar with SplitTransaction.
// IF we reach the PONR then subsequent failures need to crash out this
// regionserver
this.journal.add(JournalEntry.PONR);
// Add merged region and delete region_a and region_b
// as an atomic update. See HBASE-7721. This update to hbase:meta makes
the region
// will determine whether the region is merged or not in case of failures.
// If it is successful, master will roll-forward, if not, master will
// rollback
if (services != null
!services.reportRegionStateTransition(TransitionCode.MERGE_PONR,
mergedRegionInfo, region_a.getRegionInfo(),
region_b.getRegionInfo())) {
// Passed PONR, let SSH clean it up
throw new IOException(Failed to notify master that merge passed PONR:
+ region_a.getRegionInfo().getRegionNameAsString() + and
+ region_b.getRegionInfo().getRegionNameAsString());
}
{code}
I think while reporting region state transition to master we need to pass
meta entries also so that we can add them to meta along with split or merge
updates.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11647) MOB integration testing
[ https://issues.apache.org/jira/browse/HBASE-11647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-11647: --- Fix Version/s: hbase-11339 MOB integration testing --- Key: HBASE-11647 URL: https://issues.apache.org/jira/browse/HBASE-11647 Project: HBase Issue Type: Sub-task Components: Performance, test Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11647-without-sweep-tool-V2.diff, HBASE-11647-without-sweep-tool-V3.diff, HBASE-11647-without-sweep-tool.diff, HBASE-11647.diff The integration testings include the integration function testing and performance testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121545#comment-14121545 ] Elliott Clark commented on HBASE-11165: --- bq.Is the failure you refer to our having a root and not making use of it? No. I was referring to HDFS Federation as a failure. In my mind it's a failure. 99% of HDFS users don't need or want it yet federation slows down all feature developent and increases complexity on just about every operation. In my mind the solutions being discussed here seem to run parallel. 99% of users don't need them but everyone's going to deal with extra lookpus on region location misses and extra complexity while running the system. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9941) The context ClassLoader isn't set while calling into a coprocessor
[
https://issues.apache.org/jira/browse/HBASE-9941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121569#comment-14121569
]
Vladimir Rodionov commented on HBASE-9941:
--
The additional observed overhead is explained here:
https://bugs.openjdk.java.net/browse/JDK-6642881?page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel
Class.getClassLoader() is expensive op because of JNI nature.
CoprocessorHost.Environment
{code}
@Override
public ClassLoader getClassLoader() {
return impl.getClass().getClassLoader();
}
{code}
does not do any attempts to cache class loader instance. I think it is safe to
cache it. Correct me if I am wrong.
The context ClassLoader isn't set while calling into a coprocessor
--
Key: HBASE-9941
URL: https://issues.apache.org/jira/browse/HBASE-9941
Project: HBase
Issue Type: Sub-task
Components: Coprocessors
Affects Versions: 0.96.0
Reporter: Benoit Sigoure
Assignee: Andrew Purtell
Fix For: 0.98.0, 0.99.0
Attachments: 9941.patch, 9941.patch, 9941.patch, 9941.patch,
9941.patch
Whenever one of the methods of a coprocessor is invoked, the context
{{ClassLoader}} isn't set to be the {{CoprocessorClassLoader}}. It's only
set properly when calling the coprocessor's {{start}} method. This means
that if the coprocessor code attempts to load classes using the context
{{ClassLoader}}, it will fail to find the classes it's looking for.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source
[ https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121591#comment-14121591 ] Esteban Gutierrez commented on HBASE-11885: --- [~dimaspivak] I think having the option to automate the download of maven and the Java JDK as a flag or environment variable should be fine. However I don't think the docker image should have specific URLs or Java/JDK versions and the user should provide that, for instance you could use wget and accept additional parameters besides the URL and so if you want to download the Java JDK you need to explicitly pass the oraclelicense cookie in order to download the tarball directly from Oracle. Provide a Dockerfile to easily build and run HBase from source -- Key: HBASE-11885 URL: https://issues.apache.org/jira/browse/HBASE-11885 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Attachments: HBASE-11885.patch [A recent email to dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E] highlighted the difficulty that new users can face in getting HBase compiled from source and running locally. I'd like to provide a Dockerfile that would allow anyone with Docker running on a machine with a reasonably current Linux kernel to do so with ease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
Vladimir Rodionov created HBASE-11898: - Summary: CoprocessorHost.Environment should cache class loader instance Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.98.7 HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source
[ https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121605#comment-14121605 ] Dima Spivak commented on HBASE-11885: - Yeah, I could create a wrapper script to do that, I just wonder how much use it would get. Someone wanting to automatically download Oracle and Maven would still have to go to the sites to get URLs, and as you point out, in the case of JDK, would also need to pass the correct cookie accepting Oracle's license agreement. In fact, my first iteration of the Dockerfile did just that (i.e. had hardcoded URLs and the cookie passed with wget) before I worried that it would quickly become stale. What do you think? Provide a Dockerfile to easily build and run HBase from source -- Key: HBASE-11885 URL: https://issues.apache.org/jira/browse/HBASE-11885 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Attachments: HBASE-11885.patch [A recent email to dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E] highlighted the difficulty that new users can face in getting HBase compiled from source and running locally. I'd like to provide a Dockerfile that would allow anyone with Docker running on a machine with a reasonably current Linux kernel to do so with ease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121607#comment-14121607
]
Vladimir Rodionov commented on HBASE-11898:
---
The additional observed overhead is explained here:
https://bugs.openjdk.java.net/browse/JDK-6642881?page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel
Class.getClassLoader() is expensive op because of JNI nature.
CoprocessorHost.Environment
{code}
@Override
public ClassLoader getClassLoader() {
return impl.getClass().getClassLoader();
}
{code}
does not do any attempts to cache class loader instance.
CoprocessorHost.Environment should cache class loader instance
--
Key: HBASE-11898
URL: https://issues.apache.org/jira/browse/HBASE-11898
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 0.98.7
HBASE-9941 fix causes additional overhead...
It looks like for every invocation it does a getClassLoader call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11882) Row level consistency may not be maintained with bulk load and compaction
[ https://issues.apache.org/jira/browse/HBASE-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121622#comment-14121622 ] Ted Yu commented on HBASE-11882: @Ram: Go ahead. Row level consistency may not be maintained with bulk load and compaction - Key: HBASE-11882 URL: https://issues.apache.org/jira/browse/HBASE-11882 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 2.0.0 Reporter: Jerry He Assignee: Jerry He Priority: Critical Fix For: 0.99.0, 2.0.0 Attachments: HBASE-11882-master-v1.patch, HBASE-11882-master-v2.patch, HBASE-11882-master-v3.patch, TestHRegionServerBulkLoad.java.patch While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I found the root cause is that row level atomicity may not be maintained with bulk load together with compation. TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses multiple threads to do bulk load and scan continuously and do compactions periodically. It verifies row level data is always consistent across column families. After HBASE-11591, we added readpoint checks for bulkloaded data using the seqId at the time of bulk load. Now a scanner will not see the data from a bulk load if the scanner's readpoint is earlier than the bulk load seqId. Previously, the atomic bulk load result is visible immediately to all scanners. The problem is with compaction after bulk load. Compaction does not lock the region and it is done one store (column family) at a time. It also compact away the seqId marker of bulk load. Here is an event sequence where the row level consistency is broken. 1. A scanner is started to scan a region with cf1 and cf2. The readpoint is 10. 2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is 11. Bulk load is guarded by region write lock. So it is atomic. 3. There is a compaction that compacts cf1. It compacts away the seqId marker of the bulk load. 4. The scanner tries to next to row-1001. It gets the bulk load data for cf1 since there is no seqId preventing it. It does not get the bulk load data for cf2 since the scanner's readpoint (10) is less than the bulk load seqId (11). Now the row level consistency is broken in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11879) Change TableInputFormatBase to take interface arguments
[
https://issues.apache.org/jira/browse/HBASE-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121638#comment-14121638
]
Nick Dimiduk commented on HBASE-11879:
--
+1 for option 2. Be sure to define/document the semantics of the connection
lifecycle -- presumably the caller owns it.
Change TableInputFormatBase to take interface arguments
---
Key: HBASE-11879
URL: https://issues.apache.org/jira/browse/HBASE-11879
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
As part of the ongoing interface abstraction work, I'm now investigating
{{TableInputFormatBase}}, which has two methods that break encapsulation:
{code}
protected HTable getHTable();
protected void setHTable(HTable table);
{code}
While these are protected methods, the base @InterfaceAudience.Public is
abstract, meaning that it supports extension by user code.
I propose deprecating these two methods and replacing them with these four,
once the Table interface is merged:
{code}
protected Table getTable();
protected void setTable(Table table);
protected RegionLocator getRegionLocator();
protected void setRegionLocator(RegionLocator regionLocator);
{code}
Since users will frequently call {{setTable}} and {{setRegionLocator}}
together, it probably also makes sense to add the following convenience
method:
{code}
protected void setTableAndRegionLocator(Table table, RegionLocator
regionLocator);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11885) Provide a Dockerfile to easily build and run HBase from source
[ https://issues.apache.org/jira/browse/HBASE-11885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121652#comment-14121652 ] Esteban Gutierrez commented on HBASE-11885: --- It will become stale if you leave valid URLs there or you could probably leave them commented out as an example. If you could pass the URLs and other wget arguments if present, that would simply the extra step of having to download and then copy the files. Also the option to assume that those dependencies have already been provided might be good to. If possible try to avoid globs, I know this is done within the container but blindly expanding globs sometimes ir problematic. Provide a Dockerfile to easily build and run HBase from source -- Key: HBASE-11885 URL: https://issues.apache.org/jira/browse/HBASE-11885 Project: HBase Issue Type: New Feature Reporter: Dima Spivak Assignee: Dima Spivak Attachments: HBASE-11885.patch [A recent email to dev@|http://mail-archives.apache.org/mod_mbox/hbase-dev/201408.mbox/%3CCAAef%2BM4q%3Da8Dqxe_EHSFTueY%2BXxz%2BtTe%2BJKsWWbXjhB_Pz7oSA%40mail.gmail.com%3E] highlighted the difficulty that new users can face in getting HBase compiled from source and running locally. I'd like to provide a Dockerfile that would allow anyone with Docker running on a machine with a reasonably current Linux kernel to do so with ease. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11891) Introduce HBaseInterfaceAudience level to denote class names that appear in configs.
[ https://issues.apache.org/jira/browse/HBASE-11891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121730#comment-14121730 ] Andrew Purtell commented on HBASE-11891: +1 Introduce HBaseInterfaceAudience level to denote class names that appear in configs. Key: HBASE-11891 URL: https://issues.apache.org/jira/browse/HBASE-11891 Project: HBase Issue Type: Improvement Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11891.patch We have several classes that are private use as far as APIs are concerned, but the class name appear in user provided config files. We should add an HBaseInterfaceAudience level CONFIG and use it to denote these classes as LimitedPrivate. HBase contributors can then use this audience marking to see when changes to a FQCN will require a release note to help upgrading users correct their configuration files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11898: --- Fix Version/s: 2.0.0 0.99.0 CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Attachment: HBASE_11898.patch 0.98-trunk patch attached. CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10841) Scan,Get,Put,Delete,etc setters should consistently return this
[
https://issues.apache.org/jira/browse/HBASE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-10841:
--
Attachment: hbase-10841_v1.patch
reattach.
Scan,Get,Put,Delete,etc setters should consistently return this
---
Key: HBASE-10841
URL: https://issues.apache.org/jira/browse/HBASE-10841
Project: HBase
Issue Type: Sub-task
Components: Client, Usability
Affects Versions: 0.99.0
Reporter: Nick Dimiduk
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.99.0, 2.0.0
Attachments: hbase-10841_v1.patch, hbase-10841_v1.patch
While addressing review comments on HBASE-10818, I noticed that our {{Scan}}
class is inconsistent with it's setter methods. Some of them return {{this}},
other's don't. They should be consistent. I suggest making them all return
{{this}}, to support chained invocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Status: Patch Available (was: Open) CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10841) Scan,Get,Put,Delete,etc setters should consistently return this
[
https://issues.apache.org/jira/browse/HBASE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121772#comment-14121772
]
Hadoop QA commented on HBASE-10841:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666538/hbase-10841_v1.patch
against trunk revision .
ATTACHMENT ID: 12666538
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10714//console
This message is automatically generated.
Scan,Get,Put,Delete,etc setters should consistently return this
---
Key: HBASE-10841
URL: https://issues.apache.org/jira/browse/HBASE-10841
Project: HBase
Issue Type: Sub-task
Components: Client, Usability
Affects Versions: 0.99.0
Reporter: Nick Dimiduk
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.99.0, 2.0.0
Attachments: hbase-10841_v1.patch, hbase-10841_v1.patch
While addressing review comments on HBASE-10818, I noticed that our {{Scan}}
class is inconsistent with it's setter methods. Some of them return {{this}},
other's don't. They should be consistent. I suggest making them all return
{{this}}, to support chained invocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121811#comment-14121811 ] Ted Yu commented on HBASE-11898: +1, if tests pass CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11879) Change TableInputFormatBase to take interface arguments
[
https://issues.apache.org/jira/browse/HBASE-11879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121887#comment-14121887
]
Carter commented on HBASE-11879:
#2 it is. Agreed the caller should own the connection. We'll have to create
the Connection interface first, then work can proceed on this issue.
Change TableInputFormatBase to take interface arguments
---
Key: HBASE-11879
URL: https://issues.apache.org/jira/browse/HBASE-11879
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
As part of the ongoing interface abstraction work, I'm now investigating
{{TableInputFormatBase}}, which has two methods that break encapsulation:
{code}
protected HTable getHTable();
protected void setHTable(HTable table);
{code}
While these are protected methods, the base @InterfaceAudience.Public is
abstract, meaning that it supports extension by user code.
I propose deprecating these two methods and replacing them with these four,
once the Table interface is merged:
{code}
protected Table getTable();
protected void setTable(Table table);
protected RegionLocator getRegionLocator();
protected void setRegionLocator(RegionLocator regionLocator);
{code}
Since users will frequently call {{setTable}} and {{setRegionLocator}}
together, it probably also makes sense to add the following convenience
method:
{code}
protected void setTableAndRegionLocator(Table table, RegionLocator
regionLocator);
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11858) Audit regionserver classes that are missing InterfaceAudience
[ https://issues.apache.org/jira/browse/HBASE-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121902#comment-14121902 ] Enis Soztutar commented on HBASE-11858: --- Did you guys see HBASE-10462. Maybe we should adopt the test there for hbase-server as well. Audit regionserver classes that are missing InterfaceAudience - Key: HBASE-11858 URL: https://issues.apache.org/jira/browse/HBASE-11858 Project: HBase Issue Type: Task Components: regionserver Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11858.patch There are ~20 classes in regionserver that are missing InterfaceAudience markers. Most are probably private, but HBASE-10378 has already hit atleast one that should be LimitedPrivate(COPROC), so it's worth checking and cleaning things up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys
[ https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121903#comment-14121903 ] rajeshbabu edited comment on HBASE-10576 at 9/4/14 8:07 PM: Here is the WIP patch introducing new state 'shadow' to region so that AM/balancer can skip the assignment and at the same time the writes and reads can be served by the region. To make the region a shadow region using transaction approach. [~jeffreyz] is this approach fine for you? If it's ok, I will add more tests,APIs and utils tomorrow and upload new patch. Ping [~jxiang] [~stack] Thanks. was (Author: rajesh23): Here is the WIP patch introducing new state 'shadow' to region so that AM/balancer can skip the assignment and at the same time the writes and reads can be served by the region. To make the region a shadow region using transaction approach. [~jeffreyz] is this approach fine for you? If it's ok, I will add more tests,APIs and utils if any required. Ping [~jxiang] [~stack] Thanks. Custom load balancer to co-locate the regions of two tables which are having same split keys Key: HBASE-10576 URL: https://issues.apache.org/jira/browse/HBASE-10576 Project: HBase Issue Type: Sub-task Components: Balancer Reporter: rajeshbabu Assignee: rajeshbabu Attachments: HBASE-10536_v2.patch, HBASE-10576.patch, HBASE-10576_shadow_regions_wip.patch To support local indexing both user table and index table should have same split keys. This issue to provide custom balancer to colocate the regions of two tables which are having same split keys. This helps in Phoenix as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11858) Audit regionserver classes that are missing InterfaceAudience
[ https://issues.apache.org/jira/browse/HBASE-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121904#comment-14121904 ] Enis Soztutar commented on HBASE-11858: --- Let me do a quick review. I think we'll like to have this in branch-1 and possibly 0.98 as well. Audit regionserver classes that are missing InterfaceAudience - Key: HBASE-11858 URL: https://issues.apache.org/jira/browse/HBASE-11858 Project: HBase Issue Type: Task Components: regionserver Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11858.patch There are ~20 classes in regionserver that are missing InterfaceAudience markers. Most are probably private, but HBASE-10378 has already hit atleast one that should be LimitedPrivate(COPROC), so it's worth checking and cleaning things up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys
[ https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-10576: --- Attachment: HBASE-10576_shadow_regions_wip.patch Here is the WIP patch introducing new state 'shadow' to region so that AM/balancer can skip the assignment and at the same time the writes and reads can be served by the region. To make the region a shadow region using transaction approach. [~jeffreyz] is this approach fine for you? If it's ok, I will add more tests,APIs and utils if any required. Ping [~jxiang] [~stack] Thanks. Custom load balancer to co-locate the regions of two tables which are having same split keys Key: HBASE-10576 URL: https://issues.apache.org/jira/browse/HBASE-10576 Project: HBase Issue Type: Sub-task Components: Balancer Reporter: rajeshbabu Assignee: rajeshbabu Attachments: HBASE-10536_v2.patch, HBASE-10576.patch, HBASE-10576_shadow_regions_wip.patch To support local indexing both user table and index table should have same split keys. This issue to provide custom balancer to colocate the regions of two tables which are having same split keys. This helps in Phoenix as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11858) Audit regionserver classes that are missing InterfaceAudience
[ https://issues.apache.org/jira/browse/HBASE-11858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121965#comment-14121965 ] Andrew Purtell commented on HBASE-11858: Sure, +1 for 0.98 Audit regionserver classes that are missing InterfaceAudience - Key: HBASE-11858 URL: https://issues.apache.org/jira/browse/HBASE-11858 Project: HBase Issue Type: Task Components: regionserver Reporter: Sean Busbey Assignee: Sean Busbey Priority: Minor Attachments: HBASE-11858.patch There are ~20 classes in regionserver that are missing InterfaceAudience markers. Most are probably private, but HBASE-10378 has already hit atleast one that should be LimitedPrivate(COPROC), so it's worth checking and cleaning things up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121977#comment-14121977
]
Hadoop QA commented on HBASE-11898:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12666537/HBASE_11898.patch
against trunk revision .
ATTACHMENT ID: 12666537
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10715//console
This message is automatically generated.
CoprocessorHost.Environment should cache class loader instance
--
Key: HBASE-11898
URL: https://issues.apache.org/jira/browse/HBASE-11898
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: HBASE_11898.patch
HBASE-9941 fix causes additional overhead...
It looks like for every invocation it does a getClassLoader call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121987#comment-14121987 ] Vladimir Rodionov commented on HBASE-11898: --- The failed test is missing in 0.98. The patch is for 0.98. My bad. I will submit patch for the trunk (master). CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-8674) JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
[
https://issues.apache.org/jira/browse/HBASE-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-8674:
-
Resolution: Fixed
Fix Version/s: 2.0.0
Status: Resolved (was: Patch Available)
Committed to master. We can backport to branch-1 together with the HBASE-4955
fix.
JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
Key: HBASE-8674
URL: https://issues.apache.org/jira/browse/HBASE-8674
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.98.0, 0.94.8, 0.95.1
Reporter: Andrew Purtell
Assignee: Gary Helmling
Fix For: 2.0.0
Attachments: HBASE-8674.patch
people.apache.org cannot currently host personal or transient Maven repos.
{noformat}
$ curl --connect-timeout 60 -v
http://people.apache.org/~garyh/mvn/org/apache/maven/plugins/maven-remote-resources-plugin/1.4/maven-remote-resources-plugin-1.4.pom
* About to connect() to people.apache.org port 80 (#0)
* Trying 140.211.11.9...
* Connection timed out after 60064 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 60064 milliseconds
{noformat}
All builds are at the moment broken if the HBase custom junit or surefire
jars are not already in cache. Even if this is a temporary condition, we
should find a new home for these artifacts, upgrade to versions that include
our submitted changes (if any), or fall back to release versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11805) KeyValue to Cell Convert in WALEdit APIs
[
https://issues.apache.org/jira/browse/HBASE-11805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122039#comment-14122039
]
Andrew Purtell commented on HBASE-11805:
Looks good, except this:
{code}
+@InterfaceAudience.LimitedPrivate(HBaseInterfaceAudience.COPROC)
public class WALEdit implements Writable, HeapSize {
{code}
This might conflict with one of the JIRAs adding these annotations.
Leave this hunk out and +1
KeyValue to Cell Convert in WALEdit APIs
Key: HBASE-11805
URL: https://issues.apache.org/jira/browse/HBASE-11805
Project: HBase
Issue Type: Improvement
Components: wal
Reporter: Anoop Sam John
Assignee: Anoop Sam John
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: HBASE-11805.patch, HBASE-11805_0.98.patch,
HBASE-11805_V2.patch
In almost all other main interface class/APIs we have changed KeyValue to
Cell. But missing in WALEdit. This is public marked for Replication (Well it
should be for CP also)
These 2 APIs deal with KVs
add(KeyValue kv)
ArrayListKeyValue getKeyValues()
Suggest deprecate them and add for 0.98
add(Cell kv)
ListCell getCells()
And just replace from 1.0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11896) LoadIncrementalHFiles fails in secure mode if the namespace is specified
[
https://issues.apache.org/jira/browse/HBASE-11896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122044#comment-14122044
]
Andrew Purtell commented on HBASE-11896:
+1
LoadIncrementalHFiles fails in secure mode if the namespace is specified
Key: HBASE-11896
URL: https://issues.apache.org/jira/browse/HBASE-11896
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.3
Reporter: Ashish Singhi
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: 11896-v1.txt, HBASE-11896.patch
completebulkload tool is failing in secure mode when namespace is specified.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI:
hdfs__ns:a1__eb0l9esfa6ta4msuul40vk1kgts2hn0goo953geplsnjv1qbeq1a4sacvp07qcn6
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.init(Path.java:172)
at org.apache.hadoop.fs.Path.init(Path.java:94)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:303)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.createStagingDir(SecureBulkLoadEndpoint.java:296)
at
org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.prepareBulkLoad(SecureBulkLoadEndpoint.java:160)
at
org.apache.hadoop.hbase.protobuf.generated.SecureBulkLoadProtos$SecureBulkLoadService.callMethod(SecureBulkLoadProtos.java:4626)
at
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5696)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3332)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3314)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29501)
{noformat}
Here I feel we should replace the ':' in table name
{code}
bulkToken = new SecureBulkLoadClient(table).prepareBulkLoad(table.getName())
{code}
Because at the later part of code we are creating staging directory using
table name.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122045#comment-14122045 ] Lars Hofhansl commented on HBASE-11898: --- Looks good... +1 CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11679) Replace HTable with HTableInterface where backwards-compatible
[
https://issues.apache.org/jira/browse/HBASE-11679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122060#comment-14122060
]
Carter commented on HBASE-11679:
Okey doke. It's going to be easier to just redo the patch. IntelliJ makes
this refactor a breeze, so I'm also refactoring for {{RegionLocator}} and the
{{Admin}} interface.
After running the tests and a manual review of the diff, I'll upload the new
patch.
Replace HTable with HTableInterface where backwards-compatible
--
Key: HBASE-11679
URL: https://issues.apache.org/jira/browse/HBASE-11679
Project: HBase
Issue Type: Improvement
Reporter: Carter
Assignee: Carter
Attachments: HBASE_11679.patch, HBASE_11679.patch,
HBASE_11679_v3.patch
This is a refactor to move more of the code towards using interfaces for
proper encapsulation of logic.
The amount of code touched is large, but it should be fairly easy to review.
It changes variable declarations from HTable to HTableInterface where the
following holds:
# The declaration being updated won't break assignment
# The declaration change does not break the compile (eg trying to access
non-interface methods)
The two main situations are to change something like this:
{code}
HTable h = new HTable(c, tn);
{code}
to
{code}
HTableInterface h = new HTable(c, tn);
{code}
and this:
{code}
public void doSomething(HTable h) { ... }
{code}
to this:
{code}
public void doSomething(HTableInterface h) { ... }
{code}
This gets most of the obvious cases out of the way and prepares for more
complicated interface refactors in the future. In method signatures, I
changed parameters, but did _not_ change any public or protected method
return values, since that would violate criteria #1 above and break
compatibility.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-8674) JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
[
https://issues.apache.org/jira/browse/HBASE-8674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122113#comment-14122113
]
Hudson commented on HBASE-8674:
---
FAILURE: Integrated in HBase-TRUNK #5466 (See
[https://builds.apache.org/job/HBase-TRUNK/5466/])
HBASE-8674 Remove temporary repo for forked builds of JUnit and Surefire
(garyh: rev 7a699b9041394f7323bb4b763e89a6faba34e9d3)
* pom.xml
JUnit and Surefire TRUNK-HBASE-2 plugins need a new home
Key: HBASE-8674
URL: https://issues.apache.org/jira/browse/HBASE-8674
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.98.0, 0.94.8, 0.95.1
Reporter: Andrew Purtell
Assignee: Gary Helmling
Fix For: 2.0.0
Attachments: HBASE-8674.patch
people.apache.org cannot currently host personal or transient Maven repos.
{noformat}
$ curl --connect-timeout 60 -v
http://people.apache.org/~garyh/mvn/org/apache/maven/plugins/maven-remote-resources-plugin/1.4/maven-remote-resources-plugin-1.4.pom
* About to connect() to people.apache.org port 80 (#0)
* Trying 140.211.11.9...
* Connection timed out after 60064 milliseconds
* Closing connection 0
curl: (28) Connection timed out after 60064 milliseconds
{noformat}
All builds are at the moment broken if the HBase custom junit or surefire
jars are not already in cache. Even if this is a temporary condition, we
should find a new home for these artifacts, upgrade to versions that include
our submitted changes (if any), or fall back to release versions.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Attachment: (was: HBASE_11898.patch) CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Status: Open (was: Patch Available) CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Attachment: HBASE_11898.patch patch for 'master'. CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-11898: -- Status: Patch Available (was: Open) TestRegionReplicaReplicationEndpoint passed in my local env. I think this test is not stable. CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[ https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11898: --- Hadoop Flags: Reviewed CoprocessorHost.Environment should cache class loader instance -- Key: HBASE-11898 URL: https://issues.apache.org/jira/browse/HBASE-11898 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 0.98.6 Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 0.99.0, 2.0.0, 0.98.7 Attachments: HBASE_11898.patch HBASE-9941 fix causes additional overhead... It looks like for every invocation it does a getClassLoader call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10576) Custom load balancer to co-locate the regions of two tables which are having same split keys
[
https://issues.apache.org/jira/browse/HBASE-10576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122141#comment-14122141
]
Ted Yu commented on HBASE-10576:
{code}
+if (this.offLine == o.offLine) {
+ if(this.shadow == o.shadow) return 0;
+ if (this.shadow == true) return -1;
{code}
Why is check of shadow tied to offline status ?
{code}
+ public static void shadowRegion(final HConnection hConnection, HRegionInfo
region, ServerName sn)
+ throws IOException {
{code}
Can you add javadoc for the above method ?
{code}
+ byte[] tableRow = Bytes.toBytes(region.getRegionNameAsString() +
HConstants.DELIMITER);
{code}
What's the purpose for adding trailing delimiter ?
Reviewboard would be handy for reviewers.
Custom load balancer to co-locate the regions of two tables which are having
same split keys
Key: HBASE-10576
URL: https://issues.apache.org/jira/browse/HBASE-10576
Project: HBase
Issue Type: Sub-task
Components: Balancer
Reporter: rajeshbabu
Assignee: rajeshbabu
Attachments: HBASE-10536_v2.patch, HBASE-10576.patch,
HBASE-10576_shadow_regions_wip.patch
To support local indexing both user table and index table should have same
split keys. This issue to provide custom balancer to colocate the regions of
two tables which are having same split keys.
This helps in Phoenix as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11899) Speedup region placement assignment plan by threading region servers updating
Alex Goltman created HBASE-11899: Summary: Speedup region placement assignment plan by threading region servers updating Key: HBASE-11899 URL: https://issues.apache.org/jira/browse/HBASE-11899 Project: HBase Issue Type: New Feature Components: Region Assignment Affects Versions: 0.89-fb Reporter: Alex Goltman Priority: Minor Fix For: 0.89-fb -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11898) CoprocessorHost.Environment should cache class loader instance
[
https://issues.apache.org/jira/browse/HBASE-11898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122218#comment-14122218
]
Hadoop QA commented on HBASE-11898:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1204/HBASE_11898.patch
against trunk revision .
ATTACHMENT ID: 1204
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.coprocessor.TestMasterObserver.testRegionTransitionOperations(TestMasterObserver.java:1435)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10716//console
This message is automatically generated.
CoprocessorHost.Environment should cache class loader instance
--
Key: HBASE-11898
URL: https://issues.apache.org/jira/browse/HBASE-11898
Project: HBase
Issue Type: Bug
Components: Coprocessors
Affects Versions: 0.98.6
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: HBASE_11898.patch
HBASE-9941 fix causes additional overhead...
It looks like for every invocation it does a getClassLoader call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-10841) Scan,Get,Put,Delete,etc setters should consistently return this
[
https://issues.apache.org/jira/browse/HBASE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-10841:
--
Attachment: hbase-10841_v2.patch
It seems v1 contained unrelated changes. Lets try v2.
Scan,Get,Put,Delete,etc setters should consistently return this
---
Key: HBASE-10841
URL: https://issues.apache.org/jira/browse/HBASE-10841
Project: HBase
Issue Type: Sub-task
Components: Client, Usability
Affects Versions: 0.99.0
Reporter: Nick Dimiduk
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.99.0, 2.0.0
Attachments: hbase-10841_v1.patch, hbase-10841_v1.patch,
hbase-10841_v2.patch
While addressing review comments on HBASE-10818, I noticed that our {{Scan}}
class is inconsistent with it's setter methods. Some of them return {{this}},
other's don't. They should be consistent. I suggest making them all return
{{this}}, to support chained invocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-11900) Optimization for incremental load reducer
Yi Deng created HBASE-11900: --- Summary: Optimization for incremental load reducer Key: HBASE-11900 URL: https://issues.apache.org/jira/browse/HBASE-11900 Project: HBase Issue Type: Improvement Components: HFile Affects Versions: 0.98.6 Reporter: Yi Deng Priority: Minor In current implementation, the key of reducer configured by HFileOutputFormat.configureIncrementalLoad, is row. So, the reducer has to do an in-memory sort before writing key values to the disk. When we meet with some rows with a huge number of comlumns/versions, there could be OOM. A better way is: Use the KeyValue as the key, value can be a NullWritable. Partioner partitions the KeyValue only by it's row part. Set a sort comparator that sort KeyValue with KeyValue.COMPARATOR -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122279#comment-14122279 ] Nick Dimiduk commented on HBASE-10403: -- Agreed. [~stack]: anything left you'd like to see done? Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Improvement Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 2.0.0 Attachments: HBASE-10403.0.patch The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps BucketCache into meaning a CombinedCache instance with these implementations for L1 and L2. - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10841) Scan,Get,Put,Delete,etc setters should consistently return this
[
https://issues.apache.org/jira/browse/HBASE-10841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122289#comment-14122289
]
Hadoop QA commented on HBASE-10841:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/1228/hbase-10841_v2.patch
against trunk revision .
ATTACHMENT ID: 1228
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestShell
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.blur.mapreduce.lib.BlurOutputFormatMiniClusterTest.testBlurOutputFormat(BlurOutputFormatMiniClusterTest.java:170)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/10717//console
This message is automatically generated.
Scan,Get,Put,Delete,etc setters should consistently return this
---
Key: HBASE-10841
URL: https://issues.apache.org/jira/browse/HBASE-10841
Project: HBase
Issue Type: Sub-task
Components: Client, Usability
Affects Versions: 0.99.0
Reporter: Nick Dimiduk
Assignee: Enis Soztutar
Priority: Minor
Fix For: 0.99.0, 2.0.0
Attachments: hbase-10841_v1.patch, hbase-10841_v1.patch,
hbase-10841_v2.patch
While addressing review comments on HBASE-10818, I noticed that our {{Scan}}
class is inconsistent with it's setter methods. Some of them return {{this}},
other's don't. They should be consistent. I suggest making them all return
{{this}}, to support chained invocation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-9531) a command line (hbase shell) interface to retreive the replication metrics and show replication lag
[
https://issues.apache.org/jira/browse/HBASE-9531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jerry He updated HBASE-9531:
Attachment: HBASE-9531-master-v4.patch
a command line (hbase shell) interface to retreive the replication metrics
and show replication lag
---
Key: HBASE-9531
URL: https://issues.apache.org/jira/browse/HBASE-9531
Project: HBase
Issue Type: New Feature
Components: Replication
Affects Versions: 0.99.0
Reporter: Demai Ni
Assignee: Demai Ni
Fix For: 0.99.0, 2.0.0, 0.98.7
Attachments: HBASE-9531-master-v1.patch, HBASE-9531-master-v1.patch,
HBASE-9531-master-v1.patch, HBASE-9531-master-v2.patch,
HBASE-9531-master-v3.patch, HBASE-9531-master-v4.patch,
HBASE-9531-trunk-v0.patch, HBASE-9531-trunk-v0.patch
This jira is to provide a command line (hbase shell) interface to retreive
the replication metrics info such as:ageOfLastShippedOp,
timeStampsOfLastShippedOp, sizeOfLogQueue ageOfLastAppliedOp, and
timeStampsOfLastAppliedOp. And also to provide a point of time info of the
lag of replication(source only)
Understand that hbase is using Hadoop
metrics(http://hbase.apache.org/metrics.html), which is a common way to
monitor metric info. This Jira is to serve as a light-weight client
interface, comparing to a completed(certainly better, but heavier)GUI
monitoring package. I made the code works on 0.94.9 now, and like to use this
jira to get opinions about whether the feature is valuable to other
users/workshop. If so, I will build a trunk patch.
All inputs are greatly appreciated. Thank you!
The overall design is to reuse the existing logic which supports hbase shell
command 'status', and invent a new module, called ReplicationLoad. In
HRegionServer.buildServerLoad() , use the local replication service objects
to get their loads which could be wrapped in a ReplicationLoad object and
then simply pass it to the ServerLoad. In ReplicationSourceMetrics and
ReplicationSinkMetrics, a few getters and setters will be created, and ask
Replication to build a ReplicationLoad. (many thanks to Jean-Daniel for
his kindly suggestions through dev email list)
the replication lag will be calculated for source only, and use this formula:
{code:title=Replication lag|borderStyle=solid}
if sizeOfLogQueue != 0 then max(ageOfLastShippedOp, (current time -
timeStampsOfLastShippedOp)) //err on the large side
else if (current time - timeStampsOfLastShippedOp) 2*
ageOfLastShippedOp then lag = ageOfLastShippedOp // last shipped happen
recently
else lag = 0 // last shipped may happens last night, so NO real lag
although ageOfLastShippedOp is non-zero
{code}
External will look something like:
{code:title=status 'replication'|borderStyle=solid}
hbase(main):001:0 status 'replication'
version 0.94.9
3 live servers
hdtest017.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
SINK :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04
14:48:48 PDT 2013
hdtest018.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
SINK :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04
14:50:59 PDT 2013
hdtest015.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
SINK :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04
14:48:48 PDT 2013
hbase(main):002:0 status 'replication','source'
version 0.94.9
3 live servers
hdtest017.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=14, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:49:48 PDT 2013
hdtest018.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
hdtest015.svl.ibm.com:
SOURCE:PeerID=1, ageOfLastShippedOp=0, sizeOfLogQueue=0,
timeStampsOfLastShippedOp=Wed Sep 04 14:48:48 PDT 2013
hbase(main):003:0 status 'replication','sink'
version 0.94.9
3 live servers
hdtest017.svl.ibm.com:
SINK :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04
14:48:48 PDT 2013
hdtest018.svl.ibm.com:
SINK :AgeOfLastAppliedOp=14, TimeStampsOfLastAppliedOp=Wed Sep 04
14:50:59 PDT 2013
hdtest015.svl.ibm.com:
SINK :AgeOfLastAppliedOp=0, TimeStampsOfLastAppliedOp=Wed Sep 04
14:48:48 PDT 2013
hbase(main):003:0 status 'replication','lag'
version 0.94.9
3 live servers
hdtest017.svl.ibm.com: lag = 0
hdtest018.svl.ibm.com: lag = 14
[jira] [Created] (HBASE-11901) Improve the value size of the reference cell in mob column
Jingcheng Du created HBASE-11901: Summary: Improve the value size of the reference cell in mob column Key: HBASE-11901 URL: https://issues.apache.org/jira/browse/HBASE-11901 Project: HBase Issue Type: Improvement Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Now in the value of a reference cell in a mob column, it's realMobValueLength(Use a long type currently) + mobFileName. Actually the value length in cell is a int type, it's not necessary to use a long here. In the improvement, we'll use the int type instead long in a reference mob value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10403) Simplify offheap cache configuration
[ https://issues.apache.org/jira/browse/HBASE-10403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122319#comment-14122319 ] stack commented on HBASE-10403: --- I think this issue should stay open for 2.0. We've done enough for 1.0. Simplify offheap cache configuration Key: HBASE-10403 URL: https://issues.apache.org/jira/browse/HBASE-10403 Project: HBase Issue Type: Improvement Components: io Reporter: Nick Dimiduk Assignee: Nick Dimiduk Priority: Minor Fix For: 2.0.0 Attachments: HBASE-10403.0.patch The BucketCache (HBASE-7404) is a very nice piece of functionality which is hidden behind complex configuration. Enabling it currently requires manual calculation of L1 cache. It'd be nice to make this easier to use and conform better with the existing heap management tools we already have. Turning it on currently requires explicitly setting LruBlockCache (L1) instance size and IOEngine (L2) size, making sure that L1 size isn't too big vs global memstore and total heap. This is further confused by hbase.bucketcache.size accepting a percentage of total heap OR explicit size in MB. Enabling SlabCache is slightly easier in that it just accepts whatever LruBlockCache is provided. Turning on BucketCache using off-heap mode could look like: hbase-env.sh: Set HBASE_REGIONSERVER_OPTS: -Xmx5000m -XX:MaxDirectMemorySize=15000m hbase-site.xml: - hbase.regionserver.global.memstore.size = 0.7 - hbase.regionserver.onheap.blockcache.size = 0.1 - hbase.regionserver.blockcache.impl = BucketCache - hbase.bucketcache.ioengine = offheap The result being a CombinedCache instance with 500m LruBlockCache + 15000m ByteBufferIOEngine running in direct mode. This example does a couple things (mostly for the admin): - knows NOT to enable SlabCache - s/hfile.block.cache.size/hbase.regionserver.onheap.blockcache.size/ - maintains the validity of HBaseConfiguration's existing check that global MemStore + LruBlockCache == 0.8 - maps BucketCache into meaning a CombinedCache instance with these implementations for L1 and L2. - Figures out appropriate values for hbase.bucketcache.size and hbase.bucketcache.percentage.in.combinedcache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-11902) RegionServer was blocked while aborting
Victor Xu created HBASE-11902: - Summary: RegionServer was blocked while aborting Key: HBASE-11902 URL: https://issues.apache.org/jira/browse/HBASE-11902 Project: HBase Issue Type: Bug Components: regionserver, wal Affects Versions: 0.98.4 Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7 Reporter: Victor Xu Generally, regionserver automatically aborts when isHealth() returns false. But it sometimes got blocked while aborting. I saved the jstack and logs, and found out that it was caused by datanodes failures. The regionserver60020 thread was blocked while closing WAL. This issue doesn't happen so frequently, but if it happens, it always leads to huge amount of requests failure. The only way to do is KILL -9. I think it's a bug, but I haven't found a decent solution. Does anyone have the same problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11902) RegionServer was blocked while aborting
[ https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Xu updated HBASE-11902: -- Attachment: hbase-hadoop-regionserver-hadoop461.cm6.log Upload the hbase log file. You can find the details by searching STOPPED: One or more threads are no longer alive RegionServer was blocked while aborting --- Key: HBASE-11902 URL: https://issues.apache.org/jira/browse/HBASE-11902 Project: HBase Issue Type: Bug Components: regionserver, wal Affects Versions: 0.98.4 Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7 Reporter: Victor Xu Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log Generally, regionserver automatically aborts when isHealth() returns false. But it sometimes got blocked while aborting. I saved the jstack and logs, and found out that it was caused by datanodes failures. The regionserver60020 thread was blocked while closing WAL. This issue doesn't happen so frequently, but if it happens, it always leads to huge amount of requests failure. The only way to do is KILL -9. I think it's a bug, but I haven't found a decent solution. Does anyone have the same problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11901) Improve the value size of the reference cell in mob column
[ https://issues.apache.org/jira/browse/HBASE-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11901: - Issue Type: Sub-task (was: Improvement) Parent: HBASE-11339 Improve the value size of the reference cell in mob column -- Key: HBASE-11901 URL: https://issues.apache.org/jira/browse/HBASE-11901 Project: HBase Issue Type: Sub-task Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Now in the value of a reference cell in a mob column, it's realMobValueLength(Use a long type currently) + mobFileName. Actually the value length in cell is a int type, it's not necessary to use a long here. In the improvement, we'll use the int type instead long in a reference mob value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11902) RegionServer was blocked while aborting
[ https://issues.apache.org/jira/browse/HBASE-11902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Victor Xu updated HBASE-11902: -- Attachment: jstack_hadoop461.cm6.log Upload the jstack output. Search for 'regionserver60020' for more details. RegionServer was blocked while aborting --- Key: HBASE-11902 URL: https://issues.apache.org/jira/browse/HBASE-11902 Project: HBase Issue Type: Bug Components: regionserver, wal Affects Versions: 0.98.4 Environment: hbase-0.98.4, hadoop-2.3.0-cdh5.1, jdk1.7 Reporter: Victor Xu Attachments: hbase-hadoop-regionserver-hadoop461.cm6.log, jstack_hadoop461.cm6.log Generally, regionserver automatically aborts when isHealth() returns false. But it sometimes got blocked while aborting. I saved the jstack and logs, and found out that it was caused by datanodes failures. The regionserver60020 thread was blocked while closing WAL. This issue doesn't happen so frequently, but if it happens, it always leads to huge amount of requests failure. The only way to do is KILL -9. I think it's a bug, but I haven't found a decent solution. Does anyone have the same problem? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11901) Improve the value size of the reference cell in mob column
[ https://issues.apache.org/jira/browse/HBASE-11901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11901: - Attachment: HBASE-11901.diff Upload the patch to improvement the value size of a reference cell in a mob-enabled column. Improve the value size of the reference cell in mob column -- Key: HBASE-11901 URL: https://issues.apache.org/jira/browse/HBASE-11901 Project: HBase Issue Type: Sub-task Affects Versions: hbase-11339 Reporter: Jingcheng Du Assignee: Jingcheng Du Fix For: hbase-11339 Attachments: HBASE-11901.diff Now in the value of a reference cell in a mob column, it's realMobValueLength(Use a long type currently) + mobFileName. Actually the value length in cell is a int type, it's not necessary to use a long here. In the improvement, we'll use the int type instead long in a reference mob value. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11882) Row level consistency may not be maintained with bulk load and compaction
[ https://issues.apache.org/jira/browse/HBASE-11882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-11882: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to master and branch-1. Thanks for the patch Jerry He and for the reviews Anoop. Row level consistency may not be maintained with bulk load and compaction - Key: HBASE-11882 URL: https://issues.apache.org/jira/browse/HBASE-11882 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.99.0, 2.0.0 Reporter: Jerry He Assignee: Jerry He Priority: Critical Fix For: 0.99.0, 2.0.0 Attachments: HBASE-11882-master-v1.patch, HBASE-11882-master-v2.patch, HBASE-11882-master-v3.patch, TestHRegionServerBulkLoad.java.patch While looking into the TestHRegionServerBulkLoad failure for HBASE-11772, I found the root cause is that row level atomicity may not be maintained with bulk load together with compation. TestHRegionServerBulkLoad is used to test bulk load atomicity. The test uses multiple threads to do bulk load and scan continuously and do compactions periodically. It verifies row level data is always consistent across column families. After HBASE-11591, we added readpoint checks for bulkloaded data using the seqId at the time of bulk load. Now a scanner will not see the data from a bulk load if the scanner's readpoint is earlier than the bulk load seqId. Previously, the atomic bulk load result is visible immediately to all scanners. The problem is with compaction after bulk load. Compaction does not lock the region and it is done one store (column family) at a time. It also compact away the seqId marker of bulk load. Here is an event sequence where the row level consistency is broken. 1. A scanner is started to scan a region with cf1 and cf2. The readpoint is 10. 2. There is a bulk load that loads into cf1 and cf2. The bulk load seqId is 11. Bulk load is guarded by region write lock. So it is atomic. 3. There is a compaction that compacts cf1. It compacts away the seqId marker of the bulk load. 4. The scanner tries to next to row-1001. It gets the bulk load data for cf1 since there is no seqId preventing it. It does not get the bulk load data for cf2 since the scanner's readpoint (10) is less than the bulk load seqId (11). Now the row level consistency is broken in this case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11165) Scaling so cluster can host 1M regions and beyond (50M regions?)
[ https://issues.apache.org/jira/browse/HBASE-11165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122350#comment-14122350 ] stack commented on HBASE-11165: --- bq. ...In my mind it's a failure. OK. Lets avoid going that route. bq. ...99% of users don't need them... True. We could just hang out in the realm of the 99% in an orbit just above webscale. Meantime the excluded 1% are our brothers and sisters who need to go bigger. Lets club together and try and figure a way where we can go 'big' w/o going complicated. Scaling so cluster can host 1M regions and beyond (50M regions?) Key: HBASE-11165 URL: https://issues.apache.org/jira/browse/HBASE-11165 Project: HBase Issue Type: Brainstorming Reporter: stack Attachments: HBASE-11165.zip, Region Scalability test.pdf, zk_less_assignment_comparison_2.pdf This discussion issue comes out of Co-locate Meta And Master HBASE-10569 and comments on the doc posted there. A user -- our Francis Liu -- needs to be able to scale a cluster to do 1M regions maybe even 50M later. This issue is about discussing how we will do that (or if not 50M on a cluster, how otherwise we can attain same end). More detail to follow. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
