[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704417#comment-14704417 ] Hudson commented on HBASE-14148: SUCCESS: Integrated in HBase-1.2-IT #101 (See [https://builds.apache.org/job/HBase-1.2-IT/101/]) HBASE-14148 Default HBase web pages to be non-framable. (busbey: rev c182e9967677322c1a5633a04eddfcfdc05fbdbc) * hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestHttpServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/ClickjackingPreventionFilter.java Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Improvement Reporter: Apekshit Sharma Assignee: Gabor Liptak Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: HBASE-14148-cleanroom.1.patch, HBASE-14148-cleanroom.2.patch, HBASE-14148-cleanroom.3.patch, HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qianxi Zhang updated HBASE-14267:
-
Affects Version/s: (was: 1.1.0.1)
Status: Patch Available (was: Open)
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qianxi Zhang updated HBASE-14267:
-
Status: Open (was: Patch Available)
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Attachments: HBASE_14267_trunk_v1.patch
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbidden create table using 'hbase' namespace
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Status: Patch Available (was: Open)
we should forbidden create table using 'hbase' namespace
Key: HBASE-14265
URL: https://issues.apache.org/jira/browse/HBASE-14265
Project: HBase
Issue Type: Bug
Reporter: Heng Chen
Attachments: HBASE-14265.patch
Now, there is no limit for users who can create table under 'hbase'
NameSpace. I think it has some risk.
Because we use {{TableName.systemTable}} to decide whether this table is
System or not.
But as code, {{TableName.systemTable}} will be true, if NS equals hbase'
{code}
if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
this.namespaceAsString =
NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
this.systemTable = true;
}
{code}
And we treat system table and normal table differently.
For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704523#comment-14704523
]
Heng Chen commented on HBASE-14267:
---
IMO if you need to update the row content, you'd better clone it outside after
call {{Result.getRow()}}
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Attachments: HBASE_14267_trunk_v1.patch
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14259) Backport Namespace quota support to 98 branch
[
https://issues.apache.org/jira/browse/HBASE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705223#comment-14705223
]
Andrew Purtell edited comment on HBASE-14259 at 8/20/15 4:55 PM:
-
bq. Also, please point me to the compatibility checker tool that I need to run
against the patch.
See dev-support/check_compatibility.sh.
This is what I did to check. YMMV. There are maybe easier ways to use it but I
like to set up the comparison manually for fine tuning details:
{noformat}
$ git checkout -b HBASE-14259-0.98 0.98
$ curl
https://issues.apache.org/jira/secure/attachment/12751416/HBASE-14259_v3_0.98.patch
| patch -p1
$ git add hbase-server
$ git commit -mHBASE-14259 Backport Namespace quota support to 98 branch
(Vandana Ayyalasomayajula)
$ mkdir -p dev-support/target/compatibility/1
$ mkdir -p dev-support/target/compatibility/2
$ cp -a .git dev-support/target/compatibility/1
$ cp -a .git dev-support/target/compatibility/2
$ pushd dev-support/target/compatibility/1
$ git checkout -f 0.98
$ mvn -DskipTests install
$ cd ../2
$ git checkout -f
$ mvn -DskipTests install
$ popd
$ bash dev-support/check_compatibility.sh -n 0.98.14-SNAPSHOT
0.98.14-SNAPSHOT-14259
[...]
creating compatibility report ...
result: COMPATIBLE
total Binary compatibility problems: 0, warnings: 0
total Source compatibility problems: 0, warnings: 0
{noformat}
was (Author: apurtell):
bq. Also, please point me to the compatibility checker tool that I need to run
against the patch.
See dev-support/check_compatibility.sh.
This is what I did to check. YMMV. There are maybe easier ways to use it but I
like to set up the comparison manually for fine tuning details:
{noformat}
$ git checkout -b HBASE-14259-0.98 0.98
$ curl
https://issues.apache.org/jira/secure/attachment/12751416/HBASE-14259_v3_0.98.patch
| patch -p1
$ git add hbase-server
$ git commit -mHBASE-14259 Backport Namespace quota support to 98 branch
(Vandana Ayyalasomayajula)
$ mkdir -p dev-support/target/compatibility/1
$ mkdir -p dev-support/target/compatibility/2
$ cp -a .git dev-support/target/compatibility/1
$ cp -a .git dev-support/target/compatibility/2
$ pushd dev-support/target/compatibility/1
$ git co -f 0.98
$ mvn -DskipTests install
$ cd ../2
$ git co -f
$ mvn -DskipTests install
$ popd
$ bash dev-support/check_compatibility.sh -n 0.98.14-SNAPSHOT
0.98.14-SNAPSHOT-14259
[...]
creating compatibility report ...
result: COMPATIBLE
total Binary compatibility problems: 0, warnings: 0
total Source compatibility problems: 0, warnings: 0
{noformat}
Backport Namespace quota support to 98 branch
--
Key: HBASE-14259
URL: https://issues.apache.org/jira/browse/HBASE-14259
Project: HBase
Issue Type: New Feature
Affects Versions: 0.98.14
Reporter: Vandana Ayyalasomayajula
Assignee: Vandana Ayyalasomayajula
Priority: Minor
Attachments: HBASE-14259_v1_0.98.patch, HBASE-14259_v2_0.98.patch,
HBASE-14259_v3_0.98.patch
Namespace quota support (HBASE-8410) has been backported to branch-1
(HBASE-13438). This jira would backport the same to 98 branch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14263) ExploringCompactionPolicy logic around file selection is broken
[
https://issues.apache.org/jira/browse/HBASE-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-14263:
--
Status: Patch Available (was: Open)
ExploringCompactionPolicy logic around file selection is broken
---
Key: HBASE-14263
URL: https://issues.apache.org/jira/browse/HBASE-14263
Project: HBase
Issue Type: Bug
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 2.0.0
Attachments: HBASE-14263.patch
It seems that logic around selection of store file candidates is broken:
{code}
// Compute the total size of files that will
// have to be read if this set of files is compacted.
long size = getTotalStoreSize(potentialMatchFiles);
// Store the smallest set of files. This stored set of files will be
used
// if it looks like the algorithm is stuck.
if (mightBeStuck size smallestSize) {
smallest = potentialMatchFiles;
smallestSize = size;
}
if (size comConf.getMaxCompactSize()) {
continue;
}
++opts;
if (size = comConf.getMinCompactSize()
!filesInRatio(potentialMatchFiles, currentRatio)) {
continue;
}
{code}
This is from applyCompactionPolicy method. As you can see, both min
compaction size and max compaction size are applied to a *selection* of files
and not to individual files. It mostly works as expected only because nobody
seems using non-default hbase.hstore.compaction.max.size, which is
Long.MAX_VALUE and it is not that easy to figure out what is going
on on an opposite side (why small files do not get included?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11368) Multi-column family BulkLoad fails if compactions go on too long
[ https://issues.apache.org/jira/browse/HBASE-11368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705298#comment-14705298 ] Esteban Gutierrez commented on HBASE-11368: --- Hey [~tianq] are you still working on this? Also I agree with [~enis] regarding ref-counting might be an alternative but I couldn't find the JIRA for that, any pointer [~lhofhansl]? Multi-column family BulkLoad fails if compactions go on too long Key: HBASE-11368 URL: https://issues.apache.org/jira/browse/HBASE-11368 Project: HBase Issue Type: Bug Reporter: stack Assignee: Qiang Tian Attachments: hbase-11368-0.98.5.patch, hbase11368-master.patch, key_stacktrace_hbase10882.TXT, performance_improvement_verification_98.5.patch Compactions take a read lock. If a multi-column family region, before bulk loading, we want to take a write lock on the region. If the compaction takes too long, the bulk load fails. Various recipes include: + Making smaller regions (lame) + [~victorunique] suggests major compacting just before bulk loading over in HBASE-10882 as a work around. Does the compaction need a read lock for that long? Does the bulk load need a full write lock when multiple column families? Can we fail more gracefully at least? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14246) branch 0.98 and earlier has incorrect LICENSE/NOTICE for several dependencies
[
https://issues.apache.org/jira/browse/HBASE-14246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705300#comment-14705300
]
Hadoop QA commented on HBASE-14246:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751407/HBASE-14246-0.98.1.patch
against 0.98 branch at commit dc9c2efcc9a7d0f00d738b1b7dbd59a208c5a7a9.
ATTACHMENT ID: 12751407
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
21 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential
org.apache.hadoop.hbase.util.TestHBaseFsckEncryption
org.apache.hadoop.hbase.util.TestMergeTool
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15193//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15193//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15193//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15193//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15193//console
This message is automatically generated.
branch 0.98 and earlier has incorrect LICENSE/NOTICE for several dependencies
-
Key: HBASE-14246
URL: https://issues.apache.org/jira/browse/HBASE-14246
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 0.98.14
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Blocker
Fix For: 0.98.14
Attachments: HBASE-14246-0.98.1.patch, HBASE-14246.1.patch
I'll add details later, but several dependencies have incomplete information
on the 0.98 and 0.94 branches. If you look at generated LICENSE/NOTICE files
for unsubstituted variables (they start with $) it's a combination of
ambiguous license name, and lack of copyright notice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13996) Add write sniffing in canary
[
https://issues.apache.org/jira/browse/HBASE-13996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705311#comment-14705311
]
Hadoop QA commented on HBASE-13996:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751248/HBASE-13996-v003.diff
against master branch at commit e917787d9782961976c6489fb4a1dba19984b0ca.
ATTACHMENT ID: 12751248
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1857 checkstyle errors (more than the master's current 1853 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+description = The hbase:canary table is used to sniff the
write availbility of each regionserver.;
+System.err.println( -writeTableThe table used for write sniffing.
Default is hbase:canary);
+By default, the canary tool only check the read operations, it's hard to find
the problem in the write path.
+To enable the write sniffing, you can run canary with the `-writeSniffing`
option. When the write sniffing is enabled,
+the canary tool will create a hbase table and make sure the regions of the
table distributed on all region servers.
+In each sniffing period, the canary will try to put data to these regions to
check the write availability of each region server.
+The default name of the write table is `hbase:canary` and can be specified by
the option `-writeTable`.
+$ ${HBASE_HOME}/bin/hbase org.apache.hadoop.hbase.tool.Canary -writeSniffing
-writeTable namespace:write-canary
+The default value size of each put is 10 bytes and you can set it by the
config key: `hbase.canary.write.value.size`.
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.filter.TestScanRowPrefix
org.apache.hadoop.hbase.TestGlobalMemStoreSize
org.apache.hadoop.hbase.filter.TestMultiRowRangeFilter
org.apache.hadoop.hbase.TestZooKeeper
org.apache.hadoop.hbase.TestServerSideScanMetricsFromClientSide
org.apache.hadoop.hbase.filter.TestFuzzyRowAndColumnRangeFilter
org.apache.hadoop.hbase.filter.TestFilterWrapper
org.apache.hadoop.hbase.ipc.TestDelayedRpc
org.apache.hadoop.hbase.procedure.TestZKProcedure
org.apache.hadoop.hbase.TestHColumnDescriptorDefaultVersions
org.apache.hadoop.hbase.TestNamespace
org.apache.hadoop.hbase.TestJMXListener
org.apache.hadoop.hbase.fs.TestBlockReorder
org.apache.hadoop.hbase.TestAcidGuarantees
org.apache.hadoop.hbase.filter.TestFilterWithScanLimits
org.apache.hadoop.hbase.TestMetaTableAccessor
org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd
org.apache.hadoop.hbase.TestIOFencing
org.apache.hadoop.hbase.TestPartialResultsFromClientSide
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15194//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15194//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15194//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15194//console
This message is automatically generated.
Add write sniffing in canary
[jira] [Commented] (HBASE-14259) Backport Namespace quota support to 98 branch
[ https://issues.apache.org/jira/browse/HBASE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705317#comment-14705317 ] Andrew Purtell commented on HBASE-14259: Use of the quota manager needs to be made optional and default off. Backport Namespace quota support to 98 branch -- Key: HBASE-14259 URL: https://issues.apache.org/jira/browse/HBASE-14259 Project: HBase Issue Type: New Feature Affects Versions: 0.98.14 Reporter: Vandana Ayyalasomayajula Assignee: Vandana Ayyalasomayajula Priority: Minor Attachments: HBASE-14259_v1_0.98.patch, HBASE-14259_v2_0.98.patch, HBASE-14259_v3_0.98.patch Namespace quota support (HBASE-8410) has been backported to branch-1 (HBASE-13438). This jira would backport the same to 98 branch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12769) Replication fails to delete all corresponding zk nodes when peer is removed
[ https://issues.apache.org/jira/browse/HBASE-12769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705336#comment-14705336 ] Esteban Gutierrez commented on HBASE-12769: --- [~cuijianwei] do you have time to rebase the patch for trunk? I think as a workaround to clean up the replication queues this should be ok. Also, since most of the work from HBASE-12439 is already in we can probably move forward in HBASE-10504. Replication fails to delete all corresponding zk nodes when peer is removed --- Key: HBASE-12769 URL: https://issues.apache.org/jira/browse/HBASE-12769 Project: HBase Issue Type: Improvement Components: Replication Affects Versions: 0.99.2 Reporter: cuijianwei Priority: Minor Attachments: HBASE-12769-trunk-v0.patch When removing a peer, the client side will delete peerId under peersZNode node; then alive region servers will be notified and delete corresponding hlog queues under its rsZNode of replication. However, if there are failed servers whose hlog queues have not been transferred by alive servers(this likely happens if setting a big value to replication.sleep.before.failover and lots of region servers restarted), these hlog queues won't be deleted after the peer is removed. I think remove_peer should guarantee all corresponding zk nodes have been removed after it completes; otherwise, if we create a new peer with the same peerId with the removed one, there might be unexpected data to be replicated. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qianxi Zhang updated HBASE-14267:
-
Attachment: HBASE_14267_trunk_v1.patch
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Attachments: HBASE_14267_trunk_v1.patch
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Fix Version/s: 2.0.0
Status: Patch Available (was: Open)
Separate the old WALs into different regionserver directories
-
Key: HBASE-14247
URL: https://issues.apache.org/jira/browse/HBASE-14247
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-14247-v001.diff
Currently all old WALs of regionservers are achieved into the single
directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
replications, the number of files under oldWALs may reach the
max-directory-items limit of HDFS, which will make the hbase cluster crashed.
{quote}
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
limit=1048576 items=1048576
{quote}
A simple solution is to separate the old WALs into different directories
according to the server name of the WAL.
Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu Shaohui updated HBASE-14247:
Attachment: HBASE-14247-v001.diff
First patch to trigger large tests
Separate the old WALs into different regionserver directories
-
Key: HBASE-14247
URL: https://issues.apache.org/jira/browse/HBASE-14247
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-14247-v001.diff
Currently all old WALs of regionservers are achieved into the single
directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
replications, the number of files under oldWALs may reach the
max-directory-items limit of HDFS, which will make the hbase cluster crashed.
{quote}
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
limit=1048576 items=1048576
{quote}
A simple solution is to separate the old WALs into different directories
according to the server name of the WAL.
Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704572#comment-14704572
]
Qianxi Zhang commented on HBASE-14267:
--
IMO, this problem is serious and covert. User should know the data can not be
modified.
In our use case, this bug is covert. When lease expired, it will occur,
otherwise it will not.
There is three method to solve this problem:
1. not allowed to modify the row in Result.getRow();
2. return the clone data;
3. API doc for that to let user know the data should be modified.
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Attachments: HBASE_14267_trunk_v1.patch
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14257) Periodic flusher only handles hbase:meta, not other system tables
[
https://issues.apache.org/jira/browse/HBASE-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-14257:
---
Assignee: Abhishek Singh Chouhan
Status: Patch Available (was: Open)
Patch to check if the region is of a systemTable and correspondingly use lower
flush interval for all system table flushes and not just meta.
Periodic flusher only handles hbase:meta, not other system tables
-
Key: HBASE-14257
URL: https://issues.apache.org/jira/browse/HBASE-14257
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 1.2.0
Reporter: Lars George
Assignee: Abhishek Singh Chouhan
Labels: beginner
Attachments: HBASE-14257.patch
In {{HRegion.shouldFlush}} we have
{code}
long modifiedFlushCheckInterval = flushCheckInterval;
if (getRegionInfo().isMetaRegion()
getRegionInfo().getReplicaId() == HRegionInfo.DEFAULT_REPLICA_ID) {
modifiedFlushCheckInterval = META_CACHE_FLUSH_INTERVAL;
}
{code}
That method is called by the {{PeriodicMemstoreFlusher}} thread, and prefers
the {{hbase:meta}} only for faster flushing. It should be doing the same for
other system tables. I suggest to use {{HRI.isSystemTable()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-14227: -- Attachment: HBASE-14227_v3.patch Thanks for your review. update patch as your suggestions! [~anoop.hbase] changes: * add comments for CompactType enum and param 'compactType' Fold special cased MOB APIs into existing APIs -- Key: HBASE-14227 URL: https://issues.apache.org/jira/browse/HBASE-14227 Project: HBase Issue Type: Task Components: mob Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Heng Chen Priority: Blocker Fix For: 2.0.0 Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, HBASE-14227_v2.patch, HBASE-14227_v3.patch There are a number of APIs that came in with MOB that are not new actions for HBase, simply new actions for a MOB implementation: - compactMob - compactMobs - majorCompactMob - majorCompactMobs - getMobCompactionState And in HBaseAdmin: - validateMobColumnFamily Remove these special cases from the Admin API where possible by folding them into existing APIs. We definitely don't need one method for a singleton and another for collections. Ideally we will not have any APIs named *Mob when finished, whether MOBs are in use on a table or not should be largely an internal detail. Exposing as schema option would be fine, this conforms to existing practice for other features. Marking critical because I think removing the *Mob special cased APIs should be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14249) shaded jar modules create spurious source and test jars with incorrect LICENSE/NOTICE info
[
https://issues.apache.org/jira/browse/HBASE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14249:
Status: Patch Available (was: In Progress)
shaded jar modules create spurious source and test jars with incorrect
LICENSE/NOTICE info
--
Key: HBASE-14249
URL: https://issues.apache.org/jira/browse/HBASE-14249
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14249.4.patch, HBASE-14249.patch,
HBASE-14249.patch, HBASE-14249.patch, effective-pom.xml
the shaded jar modules don't need to create a source or test jar (because the
jars contain nothing other than META-INF)
currently we create
* the test jars are missing LICENSE
* source jars have LICENSE/NOTICE files that claim all the bundled works in
the normal jar.
{code}
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar//META-INF
hbase-shaded-server-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-server-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-server-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar//META-INF
hbase-shaded-client-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-client-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-client-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar//META-INF
hbase-shaded-client-1.1.2-tests.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar//META-INF
hbase-shaded-server-1.1.2-tests.jar//META-INF/NOTICE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14266) RegionServers have a lock contention of Configuration.getProps
Toshihiro Suzuki created HBASE-14266:
Summary: RegionServers have a lock contention of
Configuration.getProps
Key: HBASE-14266
URL: https://issues.apache.org/jira/browse/HBASE-14266
Project: HBase
Issue Type: Improvement
Components: regionserver
Environment: hbase-0.98.6-cdh5.3.1
Reporter: Toshihiro Suzuki
Here's an extract from thread dump of the RegionServer of my cluster:
{code}
...
Thread 267 (RW.default.readRpcServer.handler=184,queue=15,port=60020):
State: BLOCKED
Blocked count: 204028
Waited count: 9702639
Blocked on org.apache.hadoop.conf.Configuration@5a5e3da
Blocked by 250 (RW.default.readRpcServer.handler=167,queue=18,port=60020)
Stack:
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2250)
org.apache.hadoop.conf.Configuration.get(Configuration.java:861)
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:880)
org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1281)
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138)
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:157)
org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1804)
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1794)
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3852)
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1952)
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1938)
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1915)
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4872)
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4847)
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2918)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29921)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96)
...
{code}
There are such many threads in the thread dump.
I think that RegionServers have a lock contention which causes performance
issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704391#comment-14704391 ] Jingcheng Du commented on HBASE-14227: -- The comments about the CompactType seems not enough, the words Currently, there are only two compact types: is not a good choice in comments. We need to tell what it is and why we have this, for instance differentiate the compaction between store files and mob files. Do you have suggestion on this? [~anoopsamjohn], thanks. Fold special cased MOB APIs into existing APIs -- Key: HBASE-14227 URL: https://issues.apache.org/jira/browse/HBASE-14227 Project: HBase Issue Type: Task Components: mob Affects Versions: 2.0.0 Reporter: Andrew Purtell Assignee: Heng Chen Priority: Blocker Fix For: 2.0.0 Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, HBASE-14227_v2.patch, HBASE-14227_v3.patch There are a number of APIs that came in with MOB that are not new actions for HBase, simply new actions for a MOB implementation: - compactMob - compactMobs - majorCompactMob - majorCompactMobs - getMobCompactionState And in HBaseAdmin: - validateMobColumnFamily Remove these special cases from the Admin API where possible by folding them into existing APIs. We definitely don't need one method for a singleton and another for collections. Ideally we will not have any APIs named *Mob when finished, whether MOBs are in use on a table or not should be largely an internal detail. Exposing as schema option would be fine, this conforms to existing practice for other features. Marking critical because I think removing the *Mob special cased APIs should be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704494#comment-14704494
]
Qianxi Zhang commented on HBASE-14267:
--
Hi, [~te...@apache.org] and [~stack]
How do you think about it?
I think we should the Result should be not modified. May be we can return the
cloneValue(row).
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Affects Versions: 1.1.0.1
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14210) Create test for cell level ACLs involving user group
[ https://issues.apache.org/jira/browse/HBASE-14210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704580#comment-14704580 ] Hudson commented on HBASE-14210: FAILURE: Integrated in HBase-1.3-IT #105 (See [https://builds.apache.org/job/HBase-1.3-IT/105/]) HBASE-14210 Create test for cell level ACLs involving user group (apurtell: rev 22d39c0712fdaf7affda732a8214e24385eebb98) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLs.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java Create test for cell level ACLs involving user group Key: HBASE-14210 URL: https://issues.apache.org/jira/browse/HBASE-14210 Project: HBase Issue Type: Test Reporter: Ted Yu Assignee: Ashish Singhi Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.3.0, 1.1.3 Attachments: HBASE-14210-0.98.patch, HBASE-14210-branch-1.patch, HBASE-14210-v1.patch, HBASE-14210-v2.patch, HBASE-14210-v3.patch, HBASE-14210.patch Currently we have TestCellACLs and TestCellACLWithMultipleVersions which exercise cell level ACLs for users. However, test for cell level ACLs involving user group is missing. This issue is to add such test(s) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14243) Incorrect NOTICE file in hbase-it test-jar
[ https://issues.apache.org/jira/browse/HBASE-14243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704581#comment-14704581 ] Hudson commented on HBASE-14243: FAILURE: Integrated in HBase-1.3-IT #105 (See [https://builds.apache.org/job/HBase-1.3-IT/105/]) HBASE-14243 Correct NOTICE files. (busbey: rev f622297aa56bee1c3738d72c69edc36bfdf493d8) * hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm * hbase-it/pom.xml Incorrect NOTICE file in hbase-it test-jar -- Key: HBASE-14243 URL: https://issues.apache.org/jira/browse/HBASE-14243 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 Attachments: HBASE-14243.1.patch The hbase-it module makes use of the test resources from the hbase-server module directly. this results in a NOTICE file with placeholders instead of project attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14264) Update RAT plugin version in 0.98 branch
[
https://issues.apache.org/jira/browse/HBASE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704367#comment-14704367
]
Hadoop QA commented on HBASE-14264:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751384/HBASE-14264-0.98.patch
against 0.98 branch at commit dc9c2efcc9a7d0f00d738b1b7dbd59a208c5a7a9.
ATTACHMENT ID: 12751384
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15178//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15178//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15178//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15178//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15178//console
This message is automatically generated.
Update RAT plugin version in 0.98 branch
Key: HBASE-14264
URL: https://issues.apache.org/jira/browse/HBASE-14264
Project: HBase
Issue Type: Task
Components: build
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 0.98.14
Attachments: HBASE-14264-0.98.patch
Update the RAT plugin version in 0.98 branch to 0.11.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14247) Separate the old WALs into different regionserver directories
[
https://issues.apache.org/jira/browse/HBASE-14247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704537#comment-14704537
]
Liu Shaohui commented on HBASE-14247:
-
Please help to review at https://reviews.apache.org/r/37641/
Separate the old WALs into different regionserver directories
-
Key: HBASE-14247
URL: https://issues.apache.org/jira/browse/HBASE-14247
Project: HBase
Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui
Priority: Minor
Fix For: 2.0.0
Attachments: HBASE-14247-v001.diff
Currently all old WALs of regionservers are achieved into the single
directory of oldWALs. In big clusters, because of long TTL of WAL or disabled
replications, the number of files under oldWALs may reach the
max-directory-items limit of HDFS, which will make the hbase cluster crashed.
{quote}
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException):
The directory item limit of /hbase/lgprc-xiaomi/.oldlogs is exceeded:
limit=1048576 items=1048576
{quote}
A simple solution is to separate the old WALs into different directories
according to the server name of the WAL.
Suggestions are welcomed~ Thanks
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14225) Exclude **/target/** from RAT checks
[
https://issues.apache.org/jira/browse/HBASE-14225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704361#comment-14704361
]
Hadoop QA commented on HBASE-14225:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12751387/HBASE-14225-0.98.patch
against 0.98 branch at commit dc9c2efcc9a7d0f00d738b1b7dbd59a208c5a7a9.
ATTACHMENT ID: 12751387
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation, build,
or dev-support patch that doesn't require tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
22 warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15177//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15177//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15177//artifact/patchprocess/checkstyle-aggregate.html
Javadoc warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15177//artifact/patchprocess/patchJavadocWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15177//console
This message is automatically generated.
Exclude **/target/** from RAT checks
Key: HBASE-14225
URL: https://issues.apache.org/jira/browse/HBASE-14225
Project: HBase
Issue Type: Bug
Components: build
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Fix For: 0.98.14
Attachments: HBASE-14225-0.98.patch
We need to exclude **/target/** from RAT checks in order to build releases
now.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14257) Periodic flusher only handles hbase:meta, not other system tables
[
https://issues.apache.org/jira/browse/HBASE-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhishek Singh Chouhan updated HBASE-14257:
---
Attachment: HBASE-14257.patch
Periodic flusher only handles hbase:meta, not other system tables
-
Key: HBASE-14257
URL: https://issues.apache.org/jira/browse/HBASE-14257
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 2.0.0, 1.2.0
Reporter: Lars George
Labels: beginner
Attachments: HBASE-14257.patch
In {{HRegion.shouldFlush}} we have
{code}
long modifiedFlushCheckInterval = flushCheckInterval;
if (getRegionInfo().isMetaRegion()
getRegionInfo().getReplicaId() == HRegionInfo.DEFAULT_REPLICA_ID) {
modifiedFlushCheckInterval = META_CACHE_FLUSH_INTERVAL;
}
{code}
That method is called by the {{PeriodicMemstoreFlusher}} thread, and prefers
the {{hbase:meta}} only for faster flushing. It should be doing the same for
other system tables. I suggest to use {{HRI.isSystemTable()}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14266) RegionServers have a lock contention of Configuration.getProps
[
https://issues.apache.org/jira/browse/HBASE-14266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Toshihiro Suzuki updated HBASE-14266:
-
Attachment: thread_dump.txt
I attached the thread dump file.
RegionServers have a lock contention of Configuration.getProps
--
Key: HBASE-14266
URL: https://issues.apache.org/jira/browse/HBASE-14266
Project: HBase
Issue Type: Improvement
Components: regionserver
Environment: hbase-0.98.6-cdh5.3.1
Reporter: Toshihiro Suzuki
Attachments: thread_dump.txt
Here's an extract from thread dump of the RegionServer of my cluster:
{code}
...
Thread 267 (RW.default.readRpcServer.handler=184,queue=15,port=60020):
State: BLOCKED
Blocked count: 204028
Waited count: 9702639
Blocked on org.apache.hadoop.conf.Configuration@5a5e3da
Blocked by 250 (RW.default.readRpcServer.handler=167,queue=18,port=60020)
Stack:
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2250)
org.apache.hadoop.conf.Configuration.get(Configuration.java:861)
org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:880)
org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1281)
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:138)
org.apache.hadoop.hbase.regionserver.StoreScanner.init(StoreScanner.java:157)
org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:1804)
org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:1794)
org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.init(HRegion.java:3852)
org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1952)
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1938)
org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1915)
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4872)
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:4847)
org.apache.hadoop.hbase.regionserver.HRegionServer.get(HRegionServer.java:2918)
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29921)
org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:116)
org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:96)
...
{code}
There are such many threads in the thread dump.
I think that RegionServers have a lock contention which causes performance
issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14265) we should forbidden create table using 'hbase' namespace
[
https://issues.apache.org/jira/browse/HBASE-14265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14265:
--
Attachment: HBASE-14265.patch
prepare a patch.
we should forbidden create table using 'hbase' namespace
Key: HBASE-14265
URL: https://issues.apache.org/jira/browse/HBASE-14265
Project: HBase
Issue Type: Bug
Reporter: Heng Chen
Attachments: HBASE-14265.patch
Now, there is no limit for users who can create table under 'hbase'
NameSpace. I think it has some risk.
Because we use {{TableName.systemTable}} to decide whether this table is
System or not.
But as code, {{TableName.systemTable}} will be true, if NS equals hbase'
{code}
if (Bytes.equals(NamespaceDescriptor.SYSTEM_NAMESPACE_NAME, namespace)) {
this.namespace = NamespaceDescriptor.SYSTEM_NAMESPACE_NAME;
this.namespaceAsString =
NamespaceDescriptor.SYSTEM_NAMESPACE_NAME_STR;
this.systemTable = true;
}
{code}
And we treat system table and normal table differently.
For example, https://issues.apache.org/jira/browse/HBASE-14257 will flush
fast if table belong to system table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
[
https://issues.apache.org/jira/browse/HBASE-14267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qianxi Zhang updated HBASE-14267:
-
Status: Patch Available (was: Open)
In Mapreduce on HBase scenario, restart in TableInputFormat will result in
getting wrong data.
--
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Attachments: HBASE_14267_trunk_v1.patch
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region
server expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704379#comment-14704379 ] Hudson commented on HBASE-14148: FAILURE: Integrated in HBase-TRUNK #6739 (See [https://builds.apache.org/job/HBase-TRUNK/6739/]) HBASE-14148 Default HBase web pages to be non-framable. (busbey: rev dc9c2efcc9a7d0f00d738b1b7dbd59a208c5a7a9) * hbase-server/src/main/java/org/apache/hadoop/hbase/http/ClickjackingPreventionFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestHttpServer.java Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Improvement Reporter: Apekshit Sharma Assignee: Gabor Liptak Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: HBASE-14148-cleanroom.1.patch, HBASE-14148-cleanroom.2.patch, HBASE-14148-cleanroom.3.patch, HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12890) Provide a way to throttle the number of regions moved by the balancer
[ https://issues.apache.org/jira/browse/HBASE-12890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704383#comment-14704383 ] Anoop Sam John commented on HBASE-12890: This has to be a balancer implementation detail. If the balancer in use is not having support, will be best to extend it and use the new one. Provide a way to throttle the number of regions moved by the balancer - Key: HBASE-12890 URL: https://issues.apache.org/jira/browse/HBASE-12890 Project: HBase Issue Type: Improvement Affects Versions: 0.98.10 Reporter: churro morales Assignee: churro morales Fix For: 2.0.0, 1.3.0, 0.98.15 Attachments: HBASE-12890.patch We have a very large cluster and we frequently add remove quite a few regionservers from our cluster. Whenever we do this the balancer moves thousands of regions at once. Instead we provide a configuration parameter: hbase.balancer.max.regions. This limits the number of regions that are balanced per iteration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14249) shaded jar modules create spurious source and test jars with incorrect LICENSE/NOTICE info
[
https://issues.apache.org/jira/browse/HBASE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14249:
Attachment: HBASE-14249.4.patch
-04
* relies on user properties to skip test and source jars
* tells the shade plugin not to create a shaded test jar
the shade plugin was forcing the creation of a test jar even when we skipped
the module's test-jar.
tested by running {{mvn -Prelease -Papache-release -DskipTests clean package
}}. using [~ndimiduk]'s invocation now.
shaded jar modules create spurious source and test jars with incorrect
LICENSE/NOTICE info
--
Key: HBASE-14249
URL: https://issues.apache.org/jira/browse/HBASE-14249
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14249.4.patch, HBASE-14249.patch,
HBASE-14249.patch, HBASE-14249.patch, effective-pom.xml
the shaded jar modules don't need to create a source or test jar (because the
jars contain nothing other than META-INF)
currently we create
* the test jars are missing LICENSE
* source jars have LICENSE/NOTICE files that claim all the bundled works in
the normal jar.
{code}
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar//META-INF
hbase-shaded-server-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-server-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-server-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar//META-INF
hbase-shaded-client-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-client-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-client-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar//META-INF
hbase-shaded-client-1.1.2-tests.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar//META-INF
hbase-shaded-server-1.1.2-tests.jar//META-INF/NOTICE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14243) Incorrect NOTICE file in hbase-it test-jar
[ https://issues.apache.org/jira/browse/HBASE-14243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704407#comment-14704407 ] Hudson commented on HBASE-14243: FAILURE: Integrated in HBase-0.98 #1089 (See [https://builds.apache.org/job/HBase-0.98/1089/]) HBASE-14243 Correct NOTICE files. (busbey: rev 20dc2913423cc96bd4cd211bb7cb3e9f57f4e6c2) * hbase-it/pom.xml * hbase-resource-bundle/src/main/resources/META-INF/NOTICE.vm Incorrect NOTICE file in hbase-it test-jar -- Key: HBASE-14243 URL: https://issues.apache.org/jira/browse/HBASE-14243 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2, 1.3.0 Attachments: HBASE-14243.1.patch The hbase-it module makes use of the test resources from the hbase-server module directly. this results in a NOTICE file with placeholders instead of project attributes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14267) In Mapreduce on HBase scenario, restart in TableInputFormat will result in getting wrong data.
Qianxi Zhang created HBASE-14267:
Summary: In Mapreduce on HBase scenario, restart in
TableInputFormat will result in getting wrong data.
Key: HBASE-14267
URL: https://issues.apache.org/jira/browse/HBASE-14267
Project: HBase
Issue Type: Bug
Components: Client, mapreduce
Affects Versions: 1.1.0.1
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
When I run a mapreduce job on HBase, I will modify the row got from
Result.getRow(), for example, reverse the row. Since my program is very
complicated to handle data, it takes long time, and the lease int Region server
expired.
Result#195
{code}
public byte [] getRow() {
if (this.row == null) {
this.row = (this.cells == null || this.cells.length == 0) ?
null :
CellUtil.cloneRow(this.cells[0]);
}
return this.row;
}
{code}
TableInputFormat will restart the scan from last row, but the row has been
modified, so it will read wrong data.
TableRecordReaderImpl#218
{code}
} catch (IOException e) {
// do not retry if the exception tells us not to do so
if (e instanceof DoNotRetryIOException) {
throw e;
}
// try to handle all other IOExceptions by restarting
// the scanner, if the second call fails, it will be rethrown
LOG.info(recovered from + StringUtils.stringifyException(e));
if (lastSuccessfulRow == null) {
LOG.warn(We are restarting the first next() invocation, +
if your mapper has restarted a few other times like this +
then you should consider killing this job and investigate +
why it's taking so long.);
}
if (lastSuccessfulRow == null) {
restart(scan.getStartRow());
} else {
restart(lastSuccessfulRow);
scanner.next();// skip presumed already mapped row
}
value = scanner.next();
if (value != null value.isStale()) numStale++;
numRestarts++;
}
if (value != null value.size() 0) {
key.set(value.getRow());
lastSuccessfulRow = key.get();
return true;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14148) Web UI Framable Page
[ https://issues.apache.org/jira/browse/HBASE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704582#comment-14704582 ] Hudson commented on HBASE-14148: FAILURE: Integrated in HBase-1.3-IT #105 (See [https://builds.apache.org/job/HBase-1.3-IT/105/]) HBASE-14148 Default HBase web pages to be non-framable. (busbey: rev d4c99fc633ad459f765770602b968dac80744611) * hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestHttpServer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/ClickjackingPreventionFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/http/HttpServer.java Web UI Framable Page Key: HBASE-14148 URL: https://issues.apache.org/jira/browse/HBASE-14148 Project: HBase Issue Type: Improvement Reporter: Apekshit Sharma Assignee: Gabor Liptak Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: HBASE-14148-cleanroom.1.patch, HBASE-14148-cleanroom.2.patch, HBASE-14148-cleanroom.3.patch, HBASE-14148-master.patch, HBASE-14148-v2-master.patch, HBASE-14148-v3-master.patch The web UIs do not include the X-Frame-Options header to prevent the pages from being framed from another site. Reference: https://www.owasp.org/index.php/Clickjacking https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet https://developer.mozilla.org/en-US/docs/Web/HTTP/X-Frame-Options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14241) Fix deadlock during cluster shutdown due to concurrent connection close
[
https://issues.apache.org/jira/browse/HBASE-14241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704584#comment-14704584
]
Hudson commented on HBASE-14241:
FAILURE: Integrated in HBase-1.3-IT #105 (See
[https://builds.apache.org/job/HBase-1.3-IT/105/])
HBASE-14241 Fix deadlock during cluster shutdown due to concurrent connection
close (tedyu: rev 558aec84e835db6e3e7097dc472a816b62a72728)
* hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcClientImpl.java
Fix deadlock during cluster shutdown due to concurrent connection close
---
Key: HBASE-14241
URL: https://issues.apache.org/jira/browse/HBASE-14241
Project: HBase
Issue Type: Bug
Affects Versions: 1.0.2
Reporter: Andrew Purtell
Assignee: Ted Yu
Priority: Critical
Fix For: 2.0.0, 1.0.2, 1.2.0, 1.1.2, 1.3.0
Attachments: 14241-v2.txt, 14241-v3.txt, 14241-v4.txt, 14241-v5.txt,
deadlock.txt.gz
Caught while testing branch-1.0, shutting down TestMasterMetricsWrapper.
Found one Java-level deadlock:
=
MASTER_META_SERVER_OPERATIONS-ip-10-32-130-237:55342-0:
waiting to lock monitor 0x7f2a040051c8 (object 0x0007e36108a8, a
org.apache.hadoop.hbase.util.PoolMap),
which is held by M:0;ip-10-32-130-237:55342
M:0;ip-10-32-130-237:55342:
waiting to lock monitor 0x7f2a04005118 (object 0x0007e3610b00, a
org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection),
which is held by MASTER_META_SERVER_OPERATIONS-ip-10-32-130-237:55342-0
Full stack dump and deadlock debug output attached.
Root cause:
In RpcClientImpl#close(), we obtain lock on connections first:
{code}
synchronized (connections) {
for (Connection conn : connections.values()) {
{code}
Then markClosed() tries to obtain lock on connection object:
{code}
if (!conn.isAlive()) {
conn.markClosed(new InterruptedIOException(RpcClient is closing));
conn.close();
{code}
Another thread, MetaServerShutdownHandler, calls
RpcClientImpl$Connection#setupIOstreams() where :
{code}
markClosed(e);
close();
{code}
Lock on connection object is obtained first, then lock on connections is
attempted, leading to deadlock:
{code}
synchronized (connections) {
connections.removeValue(remoteId, this);
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14165) The initial size of RWQueueRpcExecutor.queues should be (numWriteQueues + numReadQueues + numScanQueues)
[
https://issues.apache.org/jira/browse/HBASE-14165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704583#comment-14704583
]
Hudson commented on HBASE-14165:
FAILURE: Integrated in HBase-1.3-IT #105 (See
[https://builds.apache.org/job/HBase-1.3-IT/105/])
HBASE-14165 The initial size of RWQueueRpcExecutor.queues should be
(numWriteQueues + numReadQueues + numScanQueues) (Cui Jianwei) (tedyu: rev
b79326f09be29b56142ef25a299fe35036b2dddb)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RWQueueRpcExecutor.java
The initial size of RWQueueRpcExecutor.queues should be (numWriteQueues +
numReadQueues + numScanQueues)
-
Key: HBASE-14165
URL: https://issues.apache.org/jira/browse/HBASE-14165
Project: HBase
Issue Type: Improvement
Components: rpc
Affects Versions: 0.99.2
Reporter: cuijianwei
Assignee: cuijianwei
Priority: Minor
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14165-trunk.patch
The RWQueueRpcExecutor.queues will be initialized as:
{code}
queues = new ArrayListBlockingQueueCallRunner(writeHandlersCount +
readHandlersCount);
{code}
It seems this could be improved as:
{code}
queues = new ArrayListBlockingQueueCallRunner(numWriteQueues +
numReadQueues + numScanQueues);
{code}
Suggestions are welcomed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14269) FuzzyRowFilter omits certain rows when multiple fuzzy key exist
[ https://issues.apache.org/jira/browse/HBASE-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706185#comment-14706185 ] hongbin ma commented on HBASE-14269: It does not have to be priority queue, The key is to re-sort the queue after a new row is inserted. FuzzyRowFilter omits certain rows when multiple fuzzy key exist --- Key: HBASE-14269 URL: https://issues.apache.org/jira/browse/HBASE-14269 Project: HBase Issue Type: Bug Components: Filters Reporter: hongbin ma Assignee: hongbin ma Attachments: HBASE-14269-v1.patch, HBASE-14269.patch https://issues.apache.org/jira/browse/HBASE-13761 introduced a RowTracker in FuzzyRowFilter to avoid performing getNextForFuzzyRule() for each fuzzy key on each getNextCellHint() by maintaining a list of possible row matches for each fuzzy key. The implementation assumes that the prepared rows will be matched one by one, so it removes the first row in the list as soon as it is used. However, this approach may lead to omitting rows in some cases: Consider a case where we have two fuzzy keys: 1?1 2?2 and the data is like: 000 111 112 121 122 211 212 when the first row 000 fails to match, RowTracker will update possible row matches with cell 000 and fuzzy keys 1?1,2?2. This will populate RowTracker with 101 and 202. Then 101 is popped out of RowTracker, hint the scanner to go to row 101. The scanner will get 111 and find it is a match, and continued to find that 112 is not a match, getNextCellHint will be called again. Then comes the bug: Row 101 has been removed out of RowTracker, so RowTracker will jump to 202. As you see row 121 will be omitted, but it is actually a match for fuzzy key 1?1. I will illustrate the bug by adding a new test case in TestFuzzyRowFilterEndToEnd. Also I will provide the bug fix in my patch. The idea of the new solution is to maintain a priority queue for all the possible match rows for each fuzzy key, and whenever getNextCellHint is called, the elements in the queue that are smaller than the parameter currentCell will be updated(and re-insert into the queue). The head of queue will always be the Next cell hint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14278) Fix NPE that is showing up since HBASE-14274 went in
stack created HBASE-14278:
-
Summary: Fix NPE that is showing up since HBASE-14274 went in
Key: HBASE-14278
URL: https://issues.apache.org/jira/browse/HBASE-14278
Project: HBase
Issue Type: Sub-task
Reporter: stack
Saw this in TestDistributedLogSplitting after HBASE-14274 was applied.
{code}
119113 2015-08-20 15:31:10,704 WARN [HBase-Metrics2-1]
impl.MetricsConfig(124): Cannot locate configuration: tried
hadoop-metrics2-hbase.properties,hadoop-metrics2.properties
119114 2015-08-20 15:31:10,710 ERROR [HBase-Metrics2-1]
lib.MethodMetric$2(118): Error invoking method getBlocksTotal
119115 java.lang.reflect.InvocationTargetException
119116 › at sun.reflect.GeneratedMethodAccessor72.invoke(Unknown Source)
119117 › at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
119118 › at java.lang.reflect.Method.invoke(Method.java:606)
119119 › at
org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
119120 › at
org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
119121 › at
org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
119122 › at
org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
119123 › at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
119124 › at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
119125 › at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
119126 › at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
119127 › at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
119128 › at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
119129 › at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
119130 › at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:221)
119131 › at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:96)
119132 › at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:245)
119133 › at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:229)
119134 › at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
119135 › at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
119136 › at java.lang.reflect.Method.invoke(Method.java:606)
119137 › at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:290)
119138 › at com.sun.proxy.$Proxy13.postStart(Unknown Source)
119139 › at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:185)
119140 › at
org.apache.hadoop.metrics2.impl.JmxCacheBuster$JmxCacheBusterRunnable.run(JmxCacheBuster.java:81)
119141 › at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
119142 › at java.util.concurrent.FutureTask.run(FutureTask.java:262)
119143 › at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
119144 › at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
119145 › at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
119146 › at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
119147 › at java.lang.Thread.run(Thread.java:744)
119148 Caused by: java.lang.NullPointerException
119149 › at
org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
119150 › at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3158)
119151 › at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:5652)
119152 › ... 32 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706213#comment-14706213
]
Andrew Purtell commented on HBASE-14260:
Yes please, all branches.
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14277) TestRegionServerHostname.testRegionServerHostname may fail at host with a case sensitive name
[ https://issues.apache.org/jira/browse/HBASE-14277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-14277: Attachment: HBASE-14277-v001.diff Very simple patch to fix the test. [~mbertozzi] [~tedyu] Could you help to review it? TestRegionServerHostname.testRegionServerHostname may fail at host with a case sensitive name - Key: HBASE-14277 URL: https://issues.apache.org/jira/browse/HBASE-14277 Project: HBase Issue Type: Test Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor Attachments: HBASE-14277-v001.diff After HBASE-13995, hostname will be converted to lower case in ServerName. It may cause the test: TestRegionServerHostname.testRegionServerHostname failed at host with a case sensitive name. Just fix it in test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14274) Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs MetricsRegionAggregateSourceImpl
[
https://issues.apache.org/jira/browse/HBASE-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14274:
--
Fix Version/s: 1.3.0
Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs
MetricsRegionAggregateSourceImpl
---
Key: HBASE-14274
URL: https://issues.apache.org/jira/browse/HBASE-14274
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Assignee: Elliott Clark
Fix For: 2.0.0, 1.3.0
Attachments: 23612.stack, HBASE-14274-v1.patch, HBASE-14274.patch
Looking into parent issue, got a hang locally of TestDistributedLogReplay.
We have region closes here:
{code}
RS_CLOSE_META-localhost:59610-0 prio=5 tid=0x7ff65c03f800 nid=0x54347
waiting on condition [0x00011f7ac000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00075636d8c0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:78)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:120)
at
org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:41)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1500)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1344)
- locked 0x0007ff878190 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:102)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
They are trying to MetricsRegionAggregateSourceImpl.deregister. They want to
get a write lock on this classes local ReentrantReadWriteLock while holding
MetricsRegionSourceImpl's readWriteLock write lock.
Then, elsewhere the JmxCacheBuster is running trying to get metrics with
above locks held in reverse:
{code}
HBase-Metrics2-1 daemon prio=5 tid=0x7ff65e14b000 nid=0x59a03 waiting
on condition [0x000140ea5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007cade1480 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.snapshot(MetricsRegionSourceImpl.java:193)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.getMetrics(MetricsRegionAggregateSourceImpl.java:115)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
at
[jira] [Commented] (HBASE-14274) Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs MetricsRegionAggregateSourceImpl
[
https://issues.apache.org/jira/browse/HBASE-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706210#comment-14706210
]
stack commented on HBASE-14274:
---
[~busbey] Looks like this is needed for 1.2 too. I applied to branch-1.
Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs
MetricsRegionAggregateSourceImpl
---
Key: HBASE-14274
URL: https://issues.apache.org/jira/browse/HBASE-14274
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Assignee: Elliott Clark
Fix For: 2.0.0, 1.3.0
Attachments: 23612.stack, HBASE-14274-v1.patch, HBASE-14274.patch
Looking into parent issue, got a hang locally of TestDistributedLogReplay.
We have region closes here:
{code}
RS_CLOSE_META-localhost:59610-0 prio=5 tid=0x7ff65c03f800 nid=0x54347
waiting on condition [0x00011f7ac000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00075636d8c0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:78)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:120)
at
org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:41)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1500)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1344)
- locked 0x0007ff878190 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:102)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
They are trying to MetricsRegionAggregateSourceImpl.deregister. They want to
get a write lock on this classes local ReentrantReadWriteLock while holding
MetricsRegionSourceImpl's readWriteLock write lock.
Then, elsewhere the JmxCacheBuster is running trying to get metrics with
above locks held in reverse:
{code}
HBase-Metrics2-1 daemon prio=5 tid=0x7ff65e14b000 nid=0x59a03 waiting
on condition [0x000140ea5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007cade1480 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.snapshot(MetricsRegionSourceImpl.java:193)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.getMetrics(MetricsRegionAggregateSourceImpl.java:115)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at
[jira] [Commented] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706211#comment-14706211
]
stack commented on HBASE-14260:
---
I'd say yes to all branches. Get our javadoc builds back!
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11490) In HBase Shell set_peer_tableCFs, if the tableCFS is null, it will be wrong
[
https://issues.apache.org/jira/browse/HBASE-11490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706272#comment-14706272
]
Qianxi Zhang commented on HBASE-11490:
--
Thanks [~ashish singhi] and [~anoop.hbase].
I will modify the patch and add a test case.
In HBase Shell set_peer_tableCFs, if the tableCFS is null, it will be wrong
---
Key: HBASE-11490
URL: https://issues.apache.org/jira/browse/HBASE-11490
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.99.0
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Priority: Minor
Attachments: HBASE_11490_trunk_v1.patch
In HBase Shell set_peer_tableCFs, If the tableCFS is null, it will throw NPE
# set all tables to be replicable for a peer
hbase set_peer_tableCFs '1',
hbase set_peer_tableCFs '1'
ReplicationAdmin#199
{code}
public void setPeerTableCFs(String id, String tableCFs) throws
ReplicationException {
this.replicationPeers.setPeerTableCFsConfig(id, tableCFs);
}
{code}
ReplicationPeersZKImpl#177
{code}
public void setPeerTableCFsConfig(String id, String tableCFsStr) throws
ReplicationException {
try {
if (!peerExists(id)) {
throw new IllegalArgumentException(Cannot set peer tableCFs because
id= + id
+ does not exist.);
}
String tableCFsZKNode = getTableCFsNode(id);
byte[] tableCFs = Bytes.toBytes(tableCFsStr);
if (ZKUtil.checkExists(this.zookeeper, tableCFsZKNode) != -1) {
ZKUtil.setData(this.zookeeper, tableCFsZKNode, tableCFs);
} else {
ZKUtil.createAndWatch(this.zookeeper, tableCFsZKNode, tableCFs);
}
LOG.info(Peer tableCFs with id= + id + is now + tableCFsStr);
} catch (KeeperException e) {
throw new ReplicationException(Unable to change tableCFs of the peer
with id= + id, e);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Attachment: (was: HBASE-13158_V6.patch) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.3.0 Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, HBASE-13158_V6.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Attachment: HBASE-13158_V6.patch When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.3.0 Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, HBASE-13158_V6.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13158: --- Attachment: (was: HBASE-13158_V6.patch) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also --- Key: HBASE-13158 URL: https://issues.apache.org/jira/browse/HBASE-13158 Project: HBase Issue Type: Improvement Reporter: Anoop Sam John Assignee: Anoop Sam John Fix For: 2.0.0, 1.3.0 Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, HBASE-13158_V6.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11490) In HBase Shell set_peer_tableCFs, if the tableCFS is null, it will be wrong
[
https://issues.apache.org/jira/browse/HBASE-11490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706226#comment-14706226
]
Anoop Sam John commented on HBASE-11490:
patch looks ok.
nit :
tableCFs = new byte[0];
Instead use HConstants.EMPTY_BYTE_ARRAY.
Will be better to add an FT test case as Ashish suggested.
In HBase Shell set_peer_tableCFs, if the tableCFS is null, it will be wrong
---
Key: HBASE-11490
URL: https://issues.apache.org/jira/browse/HBASE-11490
Project: HBase
Issue Type: Bug
Components: Replication
Affects Versions: 0.99.0
Reporter: Qianxi Zhang
Assignee: Qianxi Zhang
Priority: Minor
Attachments: HBASE_11490_trunk_v1.patch
In HBase Shell set_peer_tableCFs, If the tableCFS is null, it will throw NPE
# set all tables to be replicable for a peer
hbase set_peer_tableCFs '1',
hbase set_peer_tableCFs '1'
ReplicationAdmin#199
{code}
public void setPeerTableCFs(String id, String tableCFs) throws
ReplicationException {
this.replicationPeers.setPeerTableCFsConfig(id, tableCFs);
}
{code}
ReplicationPeersZKImpl#177
{code}
public void setPeerTableCFsConfig(String id, String tableCFsStr) throws
ReplicationException {
try {
if (!peerExists(id)) {
throw new IllegalArgumentException(Cannot set peer tableCFs because
id= + id
+ does not exist.);
}
String tableCFsZKNode = getTableCFsNode(id);
byte[] tableCFs = Bytes.toBytes(tableCFsStr);
if (ZKUtil.checkExists(this.zookeeper, tableCFsZKNode) != -1) {
ZKUtil.setData(this.zookeeper, tableCFsZKNode, tableCFs);
} else {
ZKUtil.createAndWatch(this.zookeeper, tableCFsZKNode, tableCFs);
}
LOG.info(Peer tableCFs with id= + id + is now + tableCFsStr);
} catch (KeeperException e) {
throw new ReplicationException(Unable to change tableCFs of the peer
with id= + id, e);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14260) building javadocs takes a long time
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705965#comment-14705965
]
Nick Dimiduk commented on HBASE-14260:
--
Didn't someone recently mention omitting generated classes from javadoc
generation? Seems like that would accelerate hbase-protocol module
significantly.
building javadocs takes a long time
---
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14260) building javadocs takes a long time
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705980#comment-14705980
]
stack commented on HBASE-14260:
---
We should skip generating javadoc for hbase-protocol... The classes are
generated (except
hbase-protocol/src/main/java//com/google/protobuf/HBaseZeroCopyByteString.java
which is a hack we'd like to hide anyways). These generated classes have no
javadoc to show.
building javadocs takes a long time
---
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706012#comment-14706012 ] Elliott Clark commented on HBASE-14239: --- This is zk-less only. I haven't tried with meta co-located yet. Branch-1.2 AM can get stuck when meta moves --- Key: HBASE-14239 URL: https://issues.apache.org/jira/browse/HBASE-14239 Project: HBase Issue Type: Sub-task Components: Region Assignment Affects Versions: 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.2.0 Attachments: HBASE-14239-v1.patch, HBASE-14239.patch When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: HBASE-14230.patch
retry
replace reflection in FSHlog with
HdfsDataOutputStream#getCurrentBlockReplication()
---
Key: HBASE-14230
URL: https://issues.apache.org/jira/browse/HBASE-14230
Project: HBase
Issue Type: Improvement
Components: wal
Reporter: Heng Chen
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14230.patch
As comment TODO said, we use
{{HdfsDataOutputStream#getCurrentBlockReplication}} and
{{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14230) replace reflection in FSHlog with HdfsDataOutputStream#getCurrentBlockReplication()
[
https://issues.apache.org/jira/browse/HBASE-14230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Heng Chen updated HBASE-14230:
--
Attachment: (was: HBASE-14230.patch)
replace reflection in FSHlog with
HdfsDataOutputStream#getCurrentBlockReplication()
---
Key: HBASE-14230
URL: https://issues.apache.org/jira/browse/HBASE-14230
Project: HBase
Issue Type: Improvement
Components: wal
Reporter: Heng Chen
Assignee: Heng Chen
Priority: Minor
Attachments: HBASE-14230.patch
As comment TODO said, we use
{{HdfsDataOutputStream#getCurrentBlockReplication}} and
{{DFSOutputStream.getPipeLine}} to replace reflection in FSHlog
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14272) Enforce major compaction on stores with KEEP_DELETED_CELLS=true
[ https://issues.apache.org/jira/browse/HBASE-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14272: -- Attachment: HBASE-14272.patch Patch for master. Enforce major compaction on stores with KEEP_DELETED_CELLS=true --- Key: HBASE-14272 URL: https://issues.apache.org/jira/browse/HBASE-14272 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Attachments: HBASE-14272.patch Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (*hbase.hstore.time.to.purge.deletes*) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14272) Enforce major compaction on stores with KEEP_DELETED_CELLS=true
[ https://issues.apache.org/jira/browse/HBASE-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14272: -- Status: Patch Available (was: Open) Enforce major compaction on stores with KEEP_DELETED_CELLS=true --- Key: HBASE-14272 URL: https://issues.apache.org/jira/browse/HBASE-14272 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Attachments: HBASE-14272.patch Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (*hbase.hstore.time.to.purge.deletes*) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14268) Improve KeyLocker
Hiroshi Ikeda created HBASE-14268:
-
Summary: Improve KeyLocker
Key: HBASE-14268
URL: https://issues.apache.org/jira/browse/HBASE-14268
Project: HBase
Issue Type: Improvement
Components: util
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
synchronized block, which doesn't make sense. Moreover, logic inside the
synchronized block is not trivial so that it makes less performance in heavy
multi-threaded environment.
2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already locked,
but it doesn't follow the contract of {{ReentrantLock}} because you are not
allowed to freely invoke lock/unlock methods under that contract. That
introduces a potential risk; Whenever you see a variable of the type
{{RentrantLock}}, you should pay attention to what the included instance is
coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hiroshi Ikeda updated HBASE-14268:
--
Attachment: KeyLockerPerformance.java
HBASE-14268.patch
Added a patch, and a simple application to check performance.
In my environment the revised KeyLocker is about 10 times faster, aside from
whether the performance test practically makes sense.
Improve KeyLocker
-
Key: HBASE-14268
URL: https://issues.apache.org/jira/browse/HBASE-14268
Project: HBase
Issue Type: Improvement
Components: util
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
Attachments: HBASE-14268.patch, KeyLockerPerformance.java
1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
synchronized block, which doesn't make sense. Moreover, logic inside the
synchronized block is not trivial so that it makes less performance in heavy
multi-threaded environment.
2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
locked, but it doesn't follow the contract of {{ReentrantLock}} because you
are not allowed to freely invoke lock/unlock methods under that contract.
That introduces a potential risk; Whenever you see a variable of the type
{{RentrantLock}}, you should pay attention to what the included instance is
coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14249) shaded jar modules create spurious source and test jars with incorrect LICENSE/NOTICE info
[
https://issues.apache.org/jira/browse/HBASE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14704691#comment-14704691
]
Sean Busbey commented on HBASE-14249:
-
second command, still no source nor test jars for shaded modules.
shaded jar modules create spurious source and test jars with incorrect
LICENSE/NOTICE info
--
Key: HBASE-14249
URL: https://issues.apache.org/jira/browse/HBASE-14249
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14249.4.patch, HBASE-14249.patch,
HBASE-14249.patch, HBASE-14249.patch, effective-pom.xml
the shaded jar modules don't need to create a source or test jar (because the
jars contain nothing other than META-INF)
currently we create
* the test jars are missing LICENSE
* source jars have LICENSE/NOTICE files that claim all the bundled works in
the normal jar.
{code}
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar//META-INF
hbase-shaded-server-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-server-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-server-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar//META-INF
hbase-shaded-client-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-client-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-client-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar//META-INF
hbase-shaded-client-1.1.2-tests.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar//META-INF
hbase-shaded-server-1.1.2-tests.jar//META-INF/NOTICE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14274) Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs MetricsRegionAggregateSourceImpl
[
https://issues.apache.org/jira/browse/HBASE-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706009#comment-14706009
]
Elliott Clark commented on HBASE-14274:
---
Ok so I can't get that log message to go away without lots of interesting work
other places. so lets get this in and we can work on the log message more in a
follow on jira.
Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs
MetricsRegionAggregateSourceImpl
---
Key: HBASE-14274
URL: https://issues.apache.org/jira/browse/HBASE-14274
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Attachments: 23612.stack, HBASE-14274-v1.patch, HBASE-14274.patch
Looking into parent issue, got a hang locally of TestDistributedLogReplay.
We have region closes here:
{code}
RS_CLOSE_META-localhost:59610-0 prio=5 tid=0x7ff65c03f800 nid=0x54347
waiting on condition [0x00011f7ac000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00075636d8c0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:78)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:120)
at
org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:41)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1500)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1344)
- locked 0x0007ff878190 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:102)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
They are trying to MetricsRegionAggregateSourceImpl.deregister. They want to
get a write lock on this classes local ReentrantReadWriteLock while holding
MetricsRegionSourceImpl's readWriteLock write lock.
Then, elsewhere the JmxCacheBuster is running trying to get metrics with
above locks held in reverse:
{code}
HBase-Metrics2-1 daemon prio=5 tid=0x7ff65e14b000 nid=0x59a03 waiting
on condition [0x000140ea5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007cade1480 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.snapshot(MetricsRegionSourceImpl.java:193)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.getMetrics(MetricsRegionAggregateSourceImpl.java:115)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:319)
at
[jira] [Commented] (HBASE-14251) javadoc jars use LICENSE/NOTICE from primary artifact
[ https://issues.apache.org/jira/browse/HBASE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706069#comment-14706069 ] Hudson commented on HBASE-14251: FAILURE: Integrated in HBase-1.2 #124 (See [https://builds.apache.org/job/HBase-1.2/124/]) HBASE-14251 ensure javadoc jars have LICENSE/NOTICE that match content. (busbey: rev 056a9fb7e91dd6b7ec0a0354b652ee17396e4849) * hbase-common/pom.xml * hbase-thrift/pom.xml * hbase-server/src/main/javadoc/META-INF/NOTICE * hbase-server/src/main/javadoc/META-INF/LICENSE * hbase-thrift/src/main/javadoc/META-INF/LICENSE * hbase-server/pom.xml * hbase-thrift/src/main/javadoc/META-INF/NOTICE * hbase-common/src/main/javadoc/META-INF/NOTICE javadoc jars use LICENSE/NOTICE from primary artifact - Key: HBASE-14251 URL: https://issues.apache.org/jira/browse/HBASE-14251 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Attachments: HBASE-14251.1.patch Our generated javadoc jars have the same LICENSE/NOTICE files as our primary artifacts but do not include a copy of hte full source. the following modules end up with incorrect artifacts: * hbase-server * hbase-common (maybe? depends on the are-apis-copyrightable court case) * hbase-thrift -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706110#comment-14706110 ] Ted Yu commented on HBASE-13408: Reading InMemoryMemstoreCompactionScansEvaluationResults.pdf, can you explain what 0.98-inmem represents ? Have you updated the patch on review board ? Thanks HBase In-Memory Memstore Compaction --- Key: HBASE-13408 URL: https://issues.apache.org/jira/browse/HBASE-13408 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: HBASE-13408-trunk-v01.patch, HBASE-13408-trunk-v02.patch, HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, InMemoryMemstoreCompactionEvaluationResults.pdf, InMemoryMemstoreCompactionScansEvaluationResults.pdf A store unit holds a column family in a region, where the memstore is its in-memory component. The memstore absorbs all updates to the store; from time to time these updates are flushed to a file on disk, where they are compacted. Unlike disk components, the memstore is not compacted until it is written to the filesystem and optionally to block-cache. This may result in underutilization of the memory due to duplicate entries per row, for example, when hot data is continuously updated. Generally, the faster the data is accumulated in memory, more flushes are triggered, the data sinks to disk more frequently, slowing down retrieval of data, even if very recent. In high-churn workloads, compacting the memstore can help maintain the data in memory, and thereby speed up data retrieval. We suggest a new compacted memstore with the following principles: 1.The data is kept in memory for as long as possible 2.Memstore data is either compacted or in process of being compacted 3.Allow a panic mode, which may interrupt an in-progress compaction and force a flush of part of the memstore. We suggest applying this optimization only to in-memory column families. A design document is attached. This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14260:
Summary: don't build javadocs for hbase-protocol module (was: building
javadocs takes a long time)
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14260:
Attachment: HBASE-14260.1.patch
-01
* skip javadoc in hbase-protocol module
after:
{code}
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 13.101 s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 2.232 s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.681 s]
[INFO] Apache HBase - Annotations . SUCCESS [ 5.983 s]
[INFO] Apache HBase - Protocol SUCCESS [ 38.162 s]
[INFO] Apache HBase - Common .. SUCCESS [01:15 min]
[INFO] Apache HBase - Procedure ... SUCCESS [ 22.196 s]
[INFO] Apache HBase - Client .. SUCCESS [ 58.287 s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 9.558 s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 11.001 s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 11.428 s]
[INFO] Apache HBase - Server .. SUCCESS [01:29 min]
[INFO] Apache HBase - Testing Util SUCCESS [ 8.219 s]
[INFO] Apache HBase - Thrift .. SUCCESS [ 54.796 s]
[INFO] Apache HBase - Shell ... SUCCESS [ 5.193 s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 5.263 s]
[INFO] Apache HBase - Examples SUCCESS [ 10.641 s]
[INFO] Apache HBase - Rest SUCCESS [ 19.626 s]
[INFO] Apache HBase - Assembly SUCCESS [ 14.284 s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.315 s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 20.721 s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 34.187 s]
[INFO] Apache HBase - Spark ... SUCCESS [02:07 min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 10:42 min
[INFO] Finished at: 2015-08-20T21:25:25-05:00
[INFO] Final Memory: 496M/1617M
[INFO]
{code}
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded ..
[jira] [Updated] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14260:
Status: Patch Available (was: In Progress)
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.1.0, 1.0.0, 2.0.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706155#comment-14706155
]
Sean Busbey commented on HBASE-14260:
-
want this in 0.98?
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13127) Add timeouts on all tests so less zombie sightings
[ https://issues.apache.org/jira/browse/HBASE-13127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705921#comment-14705921 ] stack commented on HBASE-13127: --- Thanks [~ndimiduk] Thats nice. Lets get a run in though hadoopqa is broke with unrelated OOME'ing.. I'll get it in when get a clean run. Add timeouts on all tests so less zombie sightings -- Key: HBASE-13127 URL: https://issues.apache.org/jira/browse/HBASE-13127 Project: HBase Issue Type: Improvement Components: test Reporter: stack Assignee: stack Attachments: 13127.alternate.txt, 13127.alternate.txt, 13127.alternate.txt, 13127.alternate.txt, 13127.alternate.v2.txt, 13127.alternate.v3.txt, 13127.alternate.v3.txt, 13127.alternate.v3.txt, 13127.alternate.v3.txt, 13127.alternate.v4.txt, 13127.alternate.v5.txt, 13127.alternate.v6.txt, 13127.alternate.v6.txt, 13127.alternate.v7.txt, 13127.alternate.v7.txt, 13127.alternate.v7.txt, 13127.alternate.v7.txt, 13127.alternate.v8.txt, 13127.txt, 13127v2.txt [~Apache9] and [~octo47] have been working hard at trying to get our builds passing again. They are almost there. TRUNK just failed with a zombie TestMasterObserver. Help the lads out by adding timeouts on all tests so less zombie incidence... will help identify the frequent failing issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14249) shaded jar modules create spurious source and test jars with incorrect LICENSE/NOTICE info
[
https://issues.apache.org/jira/browse/HBASE-14249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705951#comment-14705951
]
Hudson commented on HBASE-14249:
ABORTED: Integrated in HBase-1.2 #123 (See
[https://builds.apache.org/job/HBase-1.2/123/])
HBASE-14249 shaded jar modules create spurious source and test jars with
incorrect LICENSE/NOTICE info (busbey: rev
63d91af1dece4da9090af4c9c339d9403522498c)
* hbase-shaded/pom.xml
shaded jar modules create spurious source and test jars with incorrect
LICENSE/NOTICE info
--
Key: HBASE-14249
URL: https://issues.apache.org/jira/browse/HBASE-14249
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0
Attachments: HBASE-14249.4.patch, HBASE-14249.patch,
HBASE-14249.patch, HBASE-14249.patch, effective-pom.xml
the shaded jar modules don't need to create a source or test jar (because the
jars contain nothing other than META-INF)
currently we create
* the test jars are missing LICENSE
* source jars have LICENSE/NOTICE files that claim all the bundled works in
the normal jar.
{code}
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar/
hbase-shaded-server-1.1.2-sources.jar//META-INF
hbase-shaded-server-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-server-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-server-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar/
hbase-shaded-client-1.1.2-sources.jar//META-INF
hbase-shaded-client-1.1.2-sources.jar//META-INF/LICENSE
hbase-shaded-client-1.1.2-sources.jar//META-INF/MANIFEST.MF
hbase-shaded-client-1.1.2-sources.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar/
hbase-shaded-client-1.1.2-tests.jar//META-INF
hbase-shaded-client-1.1.2-tests.jar//META-INF/NOTICE
hbase-1.1.2-rc0 busbey$ find hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar/
hbase-shaded-server-1.1.2-tests.jar//META-INF
hbase-shaded-server-1.1.2-tests.jar//META-INF/NOTICE
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706053#comment-14706053 ] Elliott Clark edited comment on HBASE-14239 at 8/21/15 1:22 AM: [~elserj] so the issue comes in that with enough machines on a cluster the number of report region in transition actions can over whelm the entire priority threads. That means that the meta movement can be finalized. However if meta is in transition none of the reportRegionStateTransition calls can complete until meta is online. Before HBASE-13351 report region in transition for meta went to the priority thread all other report region in transition calls went to the default thread pool ( write pool if RW thread pool was enabled ). HBASE-13351 made it so all calls to reportRegionStateTransition went to the same pool. was (Author: eclark): [~elserj] so the issue comes in that with enough machines on a cluster the number of report region in transition actions can over whelm the entire priority threads. That means that non of the meta movements can be finalized. However if meta is in transition none of the reportRegionStateTransition calls can complete until meta is online. Before HBASE-13351 report region in transition for meta went to the priority thread all other report region in transition calls went to the default thread pool ( write pool if RW thread pool was enabled ). HBASE-13351 made it so all calls to reportRegionStateTransition went to the same pool. Branch-1.2 AM can get stuck when meta moves --- Key: HBASE-14239 URL: https://issues.apache.org/jira/browse/HBASE-14239 Project: HBase Issue Type: Sub-task Components: Region Assignment Affects Versions: 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.2.0 Attachments: HBASE-14239-v1.patch, HBASE-14239.patch When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14251) javadoc jars use LICENSE/NOTICE from primary artifact
[ https://issues.apache.org/jira/browse/HBASE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706077#comment-14706077 ] Hudson commented on HBASE-14251: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1044 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1044/]) HBASE-14251 ensure javadoc jars have LICENSE/NOTICE that match content. (busbey: rev 1fc9a6e0b1138295c508b288fce6e0a616505695) * hbase-common/src/main/javadoc/META-INF/NOTICE * hbase-thrift/src/main/javadoc/META-INF/NOTICE * hbase-server/src/main/javadoc/META-INF/NOTICE * hbase-thrift/pom.xml * hbase-server/src/main/javadoc/META-INF/LICENSE * hbase-server/pom.xml * hbase-common/pom.xml * hbase-thrift/src/main/javadoc/META-INF/LICENSE javadoc jars use LICENSE/NOTICE from primary artifact - Key: HBASE-14251 URL: https://issues.apache.org/jira/browse/HBASE-14251 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Attachments: HBASE-14251.1.patch Our generated javadoc jars have the same LICENSE/NOTICE files as our primary artifacts but do not include a copy of hte full source. the following modules end up with incorrect artifacts: * hbase-server * hbase-common (maybe? depends on the are-apis-copyrightable court case) * hbase-thrift -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14260) building javadocs takes a long time
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706106#comment-14706106
]
Andrew Purtell commented on HBASE-14260:
Javadoc on methods for the generated protobuf classes is for the most part the
associated line from the proto file. Other/internal methods do not have
javadoc. Type docs are like Protobuf type foo. This is not worth generating
docs for.
building javadocs takes a long time
---
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hiroshi Ikeda updated HBASE-14268:
--
Attachment: HBASE-14268-V2.patch
Added a revised patch.
Changed to invoke {{purge}} before {{getLock}}, because it was possible to try
to remove the same stale reference twice.
Also added comment inside the method {{purge}} about its lightweight.
Improve KeyLocker
-
Key: HBASE-14268
URL: https://issues.apache.org/jira/browse/HBASE-14268
Project: HBase
Issue Type: Improvement
Components: util
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
Attachments: HBASE-14268-V2.patch, HBASE-14268.patch,
KeyLockerPerformance.java
1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
synchronized block, which doesn't make sense. Moreover, logic inside the
synchronized block is not trivial so that it makes less performance in heavy
multi-threaded environment.
2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
locked, but it doesn't follow the contract of {{ReentrantLock}} because you
are not allowed to freely invoke lock/unlock methods under that contract.
That introduces a potential risk; Whenever you see a variable of the type
{{RentrantLock}}, you should pay attention to what the included instance is
coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14258) Make region_mover.rb script case insensitive in regard to hostname
[ https://issues.apache.org/jira/browse/HBASE-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705961#comment-14705961 ] Devaraj Das commented on HBASE-14258: - +1 Make region_mover.rb script case insensitive in regard to hostname -- Key: HBASE-14258 URL: https://issues.apache.org/jira/browse/HBASE-14258 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Priority: Minor Fix For: 2.0.0 Attachments: HBASE-14258.patch The script is case sensitive and fails when case of a host name being unloaded does not match with a case of a region server name returned by HBase API. This doc clarifies IETF rules on case insensitivities in DNS: https://www.ietf.org/rfc/rfc4343.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[
https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706017#comment-14706017
]
Heng Chen commented on HBASE-14227:
---
How about this
{code}
+ /**
+ * {@code NORMAL} means do store files compaction;
+ * {@code MOB} means do mob files compaction.
+ * It is different, MOB compaction only take care of MOB files.
+ * Generally, MOB CF is larger than normal CF, a compaction on the MOB CF
will be more IO oriented than the normal CF.
+ * And the impact of the number of MOB files is not significant in read,
usually they are deleted when expired.
+ * So sometimes MOB compactions are not necessary. More MOB detail see
HBASE-11339
+ * */
+ public enum CompactType {
+
+NORMAL(0),
+MOB (1);
+
+CompactType(int value) {}
+ }
{code}
Fold special cased MOB APIs into existing APIs
--
Key: HBASE-14227
URL: https://issues.apache.org/jira/browse/HBASE-14227
Project: HBase
Issue Type: Task
Components: mob
Affects Versions: 2.0.0
Reporter: Andrew Purtell
Assignee: Heng Chen
Priority: Blocker
Fix For: 2.0.0
Attachments: HBASE-14227.patch, HBASE-14227_v1.patch,
HBASE-14227_v2.patch, HBASE-14227_v3.patch
There are a number of APIs that came in with MOB that are not new actions for
HBase, simply new actions for a MOB implementation:
- compactMob
- compactMobs
- majorCompactMob
- majorCompactMobs
- getMobCompactionState
And in HBaseAdmin:
- validateMobColumnFamily
Remove these special cases from the Admin API where possible by folding them
into existing APIs.
We definitely don't need one method for a singleton and another for
collections.
Ideally we will not have any APIs named *Mob when finished, whether MOBs are
in use on a table or not should be largely an internal detail. Exposing as
schema option would be fine, this conforms to existing practice for other
features.
Marking critical because I think removing the *Mob special cased APIs should
be a precondition for release of this feature either in 2.0 or as a backport.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14260) building javadocs takes a long time
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey reassigned HBASE-14260:
---
Assignee: Sean Busbey
building javadocs takes a long time
---
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706114#comment-14706114
]
Hiroshi Ikeda commented on HBASE-14268:
---
With the Oracle (Sun) implementation of {{ReferenceQueue}}, the method
{{purge}} seems lightweight while there is no stale reference, because it just
checks a volatile instance variable in {{ReferenceQueue}}.
I could create an independent thread to remove stale references, but the
{{KeyLocker}} itself would be never garbage-collected until someone stops the
thread.
Improve KeyLocker
-
Key: HBASE-14268
URL: https://issues.apache.org/jira/browse/HBASE-14268
Project: HBase
Issue Type: Improvement
Components: util
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
Attachments: HBASE-14268.patch, KeyLockerPerformance.java
1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
synchronized block, which doesn't make sense. Moreover, logic inside the
synchronized block is not trivial so that it makes less performance in heavy
multi-threaded environment.
2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
locked, but it doesn't follow the contract of {{ReentrantLock}} because you
are not allowed to freely invoke lock/unlock methods under that contract.
That introduces a potential risk; Whenever you see a variable of the type
{{RentrantLock}}, you should pay attention to what the included instance is
coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HBASE-14260) building javadocs takes a long time
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HBASE-14260 started by Sean Busbey.
---
building javadocs takes a long time
---
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706134#comment-14706134
]
Andrew Purtell commented on HBASE-14260:
+1
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14277) TestRegionServerHostname.testRegionServerHostname may fail at host with a case sensitive name
Liu Shaohui created HBASE-14277: --- Summary: TestRegionServerHostname.testRegionServerHostname may fail at host with a case sensitive name Key: HBASE-14277 URL: https://issues.apache.org/jira/browse/HBASE-14277 Project: HBase Issue Type: Test Reporter: Liu Shaohui Assignee: Liu Shaohui Priority: Minor After HBASE-13995, hostname will be converted to lower case in ServerName. It may cause the test: TestRegionServerHostname.testRegionServerHostname failed at host with a case sensitive name. Just fix it in test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14268) Improve KeyLocker
[
https://issues.apache.org/jira/browse/HBASE-14268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706162#comment-14706162
]
Ted Yu commented on HBASE-14268:
Alright.
Let's get a green QA run when master build becomes stable.
Improve KeyLocker
-
Key: HBASE-14268
URL: https://issues.apache.org/jira/browse/HBASE-14268
Project: HBase
Issue Type: Improvement
Components: util
Reporter: Hiroshi Ikeda
Assignee: Hiroshi Ikeda
Priority: Minor
Attachments: HBASE-14268.patch, KeyLockerPerformance.java
1. In the implementation of {{KeyLocker}} it uses atomic variables inside a
synchronized block, which doesn't make sense. Moreover, logic inside the
synchronized block is not trivial so that it makes less performance in heavy
multi-threaded environment.
2. {{KeyLocker}} gives an instance of {{RentrantLock}} which is already
locked, but it doesn't follow the contract of {{ReentrantLock}} because you
are not allowed to freely invoke lock/unlock methods under that contract.
That introduces a potential risk; Whenever you see a variable of the type
{{RentrantLock}}, you should pay attention to what the included instance is
coming from.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14274) Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs MetricsRegionAggregateSourceImpl
[
https://issues.apache.org/jira/browse/HBASE-14274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705929#comment-14705929
]
Elliott Clark commented on HBASE-14274:
---
For that one it seems like the cache buster ran before the nn was all set up. I
don't think that it's caused by the patch, more likely caused the fact that
HBase has to stop and start the metrics system.
# That shouldn't be an issue in real life.
# Let me get something up so that the logs are cleaner.
Deadlock in region metrics on shutdown: MetricsRegionSourceImpl vs
MetricsRegionAggregateSourceImpl
---
Key: HBASE-14274
URL: https://issues.apache.org/jira/browse/HBASE-14274
Project: HBase
Issue Type: Sub-task
Components: test
Reporter: stack
Attachments: 23612.stack, HBASE-14274-v1.patch, HBASE-14274.patch
Looking into parent issue, got a hang locally of TestDistributedLogReplay.
We have region closes here:
{code}
RS_CLOSE_META-localhost:59610-0 prio=5 tid=0x7ff65c03f800 nid=0x54347
waiting on condition [0x00011f7ac000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00075636d8c0 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:945)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:78)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:120)
at
org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:41)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1500)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1344)
- locked 0x0007ff878190 (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:102)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:103)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
{code}
They are trying to MetricsRegionAggregateSourceImpl.deregister. They want to
get a write lock on this classes local ReentrantReadWriteLock while holding
MetricsRegionSourceImpl's readWriteLock write lock.
Then, elsewhere the JmxCacheBuster is running trying to get metrics with
above locks held in reverse:
{code}
HBase-Metrics2-1 daemon prio=5 tid=0x7ff65e14b000 nid=0x59a03 waiting
on condition [0x000140ea5000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x0007cade1480 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:964)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1282)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:731)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.snapshot(MetricsRegionSourceImpl.java:193)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.getMetrics(MetricsRegionAggregateSourceImpl.java:115)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getNewMBeanClassName(DefaultMBeanServerInterceptor.java:333)
at
[jira] [Commented] (HBASE-14255) Simplify Cell creation post 1.0
[
https://issues.apache.org/jira/browse/HBASE-14255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705949#comment-14705949
]
Nick Dimiduk commented on HBASE-14255:
--
I think you want
[CellUtil|http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/CellUtil.html].
Simplify Cell creation post 1.0
---
Key: HBASE-14255
URL: https://issues.apache.org/jira/browse/HBASE-14255
Project: HBase
Issue Type: Improvement
Components: Client
Affects Versions: 1.0.0, 2.0.0
Reporter: Lars George
Priority: Critical
After the switch to the new Cell based client API, and making KeyValue
private (but especially as soon as DBB backed Cells land) it is rather
difficult to create a {{Cell}} instance. I am using this now:
{code}
@Override
public void postGetOp(ObserverContextRegionCoprocessorEnvironment e,
Get get, ListCell results) throws IOException {
Put put = new Put(get.getRow());
put.addColumn(get.getRow(), FIXED_COLUMN, Bytes.toBytes(counter.get()));
CellScanner scanner = put.cellScanner();
scanner.advance();
Cell cell = scanner.current();
LOG.debug(Adding fake cell: + cell);
results.add(cell);
}
{code}
That is, I have to create a {{Put}} instance to add a cell and then retrieve
its instance. The {{KeyValue}} methods are private now and should not be
used. Create a CellBuilder helper?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14251) javadoc jars use LICENSE/NOTICE from primary artifact
[ https://issues.apache.org/jira/browse/HBASE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706030#comment-14706030 ] Hudson commented on HBASE-14251: ABORTED: Integrated in HBase-1.3 #122 (See [https://builds.apache.org/job/HBase-1.3/122/]) HBASE-14251 ensure javadoc jars have LICENSE/NOTICE that match content. (busbey: rev 71fbd4ac63122f0c9de562bc84778fb3b824dbd9) * hbase-thrift/src/main/javadoc/META-INF/NOTICE * hbase-server/src/main/javadoc/META-INF/NOTICE * hbase-thrift/src/main/javadoc/META-INF/LICENSE * hbase-common/pom.xml * hbase-common/src/main/javadoc/META-INF/NOTICE * hbase-thrift/pom.xml * hbase-server/src/main/javadoc/META-INF/LICENSE * hbase-server/pom.xml javadoc jars use LICENSE/NOTICE from primary artifact - Key: HBASE-14251 URL: https://issues.apache.org/jira/browse/HBASE-14251 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Attachments: HBASE-14251.1.patch Our generated javadoc jars have the same LICENSE/NOTICE files as our primary artifacts but do not include a copy of hte full source. the following modules end up with incorrect artifacts: * hbase-server * hbase-common (maybe? depends on the are-apis-copyrightable court case) * hbase-thrift -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706062#comment-14706062 ] Josh Elser commented on HBASE-14239: Thanks for the explanation, [~eclark]. I don't think I realized the RITs could prevent meta's location from being finalized. A revert of that definitely seems appropriate then. Branch-1.2 AM can get stuck when meta moves --- Key: HBASE-14239 URL: https://issues.apache.org/jira/browse/HBASE-14239 Project: HBase Issue Type: Sub-task Components: Region Assignment Affects Versions: 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.2.0 Attachments: HBASE-14239-v1.patch, HBASE-14239.patch When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14260) don't build javadocs for hbase-protocol module
[
https://issues.apache.org/jira/browse/HBASE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706157#comment-14706157
]
Sean Busbey commented on HBASE-14260:
-
I guess the question also holds for the branch-1s. 1.1? 1.0? [~enis]
[~ndimiduk]?
don't build javadocs for hbase-protocol module
--
Key: HBASE-14260
URL: https://issues.apache.org/jira/browse/HBASE-14260
Project: HBase
Issue Type: Improvement
Affects Versions: 1.0.0, 2.0.0, 1.1.0, 1.2.0
Reporter: Sean Busbey
Assignee: Sean Busbey
Attachments: HBASE-14260.1.patch
I'm not sure I have all the affected versions, but it seems that something is
amiss in making our javadocs:
{code}
mvn -Papache-release -Prelease -DskipTests clean package
... SNIP ...
[INFO]
[INFO] Reactor Summary:
[INFO]
[INFO] Apache HBase ... SUCCESS [ 11.149
s]
[INFO] Apache HBase - Checkstyle .. SUCCESS [ 1.249
s]
[INFO] Apache HBase - Resource Bundle . SUCCESS [ 0.539
s]
[INFO] Apache HBase - Annotations . SUCCESS [ 4.438
s]
[INFO] Apache HBase - Protocol SUCCESS [10:15
min]
[INFO] Apache HBase - Common .. SUCCESS [ 48.465
s]
[INFO] Apache HBase - Procedure ... SUCCESS [ 14.375
s]
[INFO] Apache HBase - Client .. SUCCESS [ 45.187
s]
[INFO] Apache HBase - Hadoop Compatibility SUCCESS [ 6.998
s]
[INFO] Apache HBase - Hadoop Two Compatibility SUCCESS [ 14.891
s]
[INFO] Apache HBase - Prefix Tree . SUCCESS [ 14.214
s]
[INFO] Apache HBase - Server .. SUCCESS [02:01
min]
[INFO] Apache HBase - Testing Util SUCCESS [ 12.779
s]
[INFO] Apache HBase - Thrift .. SUCCESS [01:15
min]
[INFO] Apache HBase - Shell ... SUCCESS [ 6.649
s]
[INFO] Apache HBase - Integration Tests ... SUCCESS [ 6.429
s]
[INFO] Apache HBase - Examples SUCCESS [ 13.200
s]
[INFO] Apache HBase - Rest SUCCESS [ 27.831
s]
[INFO] Apache HBase - Assembly SUCCESS [ 19.400
s]
[INFO] Apache HBase - Shaded .. SUCCESS [ 0.419
s]
[INFO] Apache HBase - Shaded - Client . SUCCESS [ 23.707
s]
[INFO] Apache HBase - Shaded - Server . SUCCESS [ 43.654
s]
[INFO] Apache HBase - Spark ... SUCCESS [02:22
min]
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 21:13 min
[INFO] Finished at: 2015-08-19T15:48:00-05:00
[INFO] Final Memory: 181M/1513M
[INFO]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14250) branch-1.1 hbase-server test-jar has incorrect LICENSE
[
https://issues.apache.org/jira/browse/HBASE-14250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705979#comment-14705979
]
Nick Dimiduk commented on HBASE-14250:
--
{noformat}
diff --git a/dev-support/make_rc.sh b/dev-support/make_rc.sh
index 0d0319b..0f0d4a4 100755
--- a/dev-support/make_rc.sh
+++ b/dev-support/make_rc.sh
@@ -54,15 +54,27 @@ function tgz_mover {
}
function deploy {
- MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests -Prelease
- MAVEN_OPTS=${mvnopts} ${mvn} install -DskipTests post-site assembly:single
-Prelease
+ MAVEN_OPTS=${mvnopts} ${mvn} -X clean install -DskipTests -Prelease 21
clean_install.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE clean_install.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
clean_install.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE clean_install.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
clean_install.log
+ MAVEN_OPTS=${mvnopts} ${mvn} -X install -DskipTests post-site
assembly:single -Prelease 21 install_assemble.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE install_assemble.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
install_assemble.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE install_assemble.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
install_assemble.log
tgz_mover
- MAVEN_OPTS=${mvnopts} ${mvn} deploy -DskipTests -Papache-release -Prelease
+ MAVEN_OPTS=${mvnopts} ${mvn} install -DskipTests -Papache-release
-Prelease install_apache-release.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE install_apache-release.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
install_apache-release.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE install_apache-release.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
install_apache-release.log
}
# Build src tarball
-MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests assembly:single
-Dassembly.file=${HBASE_HOME}/hbase-assembly/src/main/assembly/src.xml
-Prelease
-tgz_mover
+#MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests assembly:single
-Dassembly.file=${HBASE_HOME}/hbase-assembly/src/main/assembly/src.xml
-Prelease
+#tgz_mover
# Now do the two builds, one for hadoop1, then hadoop2
deploy
{noformat}
{noformat}
$ grep -A5 unzip clean_install.log
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
Apache HBase - Server
Copyright 2007-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
--
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
{noformat}
{noformat}
$ grep -A5 unzip install_a
install_apache-release.log install_assemble.log
hw11397:hbase ndimiduk$ grep -A5 unzip install_assemble.log
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
Apache HBase - Server
Copyright 2007-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
--
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
{noformat}
{noformat}
$ grep -A5 unzip install_apache-release.log
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
Apache HBase - Server
Copyright 2007-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
--
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
{noformat}
branch-1.1 hbase-server test-jar has incorrect LICENSE
--
Key: HBASE-14250
URL: https://issues.apache.org/jira/browse/HBASE-14250
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 1.1.2
Reporter: Sean Busbey
Assignee: Sean Busbey
Priority: Blocker
test-jar LICENSE file for hbase-server claims jquery and the orca logo are
[jira] [Updated] (HBASE-14272) Enforce major compaction on stores with KEEP_DELETED_CELLS=true
[ https://issues.apache.org/jira/browse/HBASE-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14272: -- Description: Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (*hbase.hstore.time.to.purge.deletes*) was: Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (hbase.hstore.time.to.purge.deletes) Enforce major compaction on stores with KEEP_DELETED_CELLS=true --- Key: HBASE-14272 URL: https://issues.apache.org/jira/browse/HBASE-14272 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (*hbase.hstore.time.to.purge.deletes*) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14272) Enforce major compaction on stores with KEEP_DELETED_CELLS=true
[ https://issues.apache.org/jira/browse/HBASE-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14272: -- Description: Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (hbase.hstore.time.to.purge.deletes) was: Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again when locality is below threshold defined by *hbase.hstore.min.locality.to.skip.major.compact*. If file has locality greater than this threshold it will never be major compacted. This does not work for files with with locality 1.0 and KEEP_DELETED_CELLS=true Enforce major compaction on stores with KEEP_DELETED_CELLS=true --- Key: HBASE-14272 URL: https://issues.apache.org/jira/browse/HBASE-14272 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again - when locality is below threshold, defined by *hbase.hstore.min.locality.to.skip.major.compact* or TTL expired some cells. If file has locality greater than this threshold it will never be major compacted until Store's TTL kicks in. For CF with KEEP_DELETED_CELLS on, compaction must be enabled always (even for single file), regardless of locality, when deleted cells are expired (hbase.hstore.time.to.purge.deletes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14239) Branch-1.2 AM can get stuck when meta moves
[ https://issues.apache.org/jira/browse/HBASE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706053#comment-14706053 ] Elliott Clark commented on HBASE-14239: --- [~elserj] so the issue comes in that with enough machines on a cluster the number of report region in transition actions can over whelm the entire priority threads. That means that non of the meta movements can be finalized. However if meta is in transition none of the reportRegionStateTransition calls can complete until meta is online. Before HBASE-13351 report region in transition for meta went to the priority thread all other report region in transition calls went to the default thread pool ( write pool if RW thread pool was enabled ). HBASE-13351 made it so all calls to reportRegionStateTransition went to the same pool. Branch-1.2 AM can get stuck when meta moves --- Key: HBASE-14239 URL: https://issues.apache.org/jira/browse/HBASE-14239 Project: HBase Issue Type: Sub-task Components: Region Assignment Affects Versions: 1.2.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 1.2.0 Attachments: HBASE-14239-v1.patch, HBASE-14239.patch When regions are moving master can get totally stuck trying to talk to meta. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706125#comment-14706125 ] Ted Yu commented on HBASE-13408: There were two rejected hunks for HStore.java FYI HBase In-Memory Memstore Compaction --- Key: HBASE-13408 URL: https://issues.apache.org/jira/browse/HBASE-13408 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: HBASE-13408-trunk-v01.patch, HBASE-13408-trunk-v02.patch, HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, InMemoryMemstoreCompactionEvaluationResults.pdf, InMemoryMemstoreCompactionScansEvaluationResults.pdf A store unit holds a column family in a region, where the memstore is its in-memory component. The memstore absorbs all updates to the store; from time to time these updates are flushed to a file on disk, where they are compacted. Unlike disk components, the memstore is not compacted until it is written to the filesystem and optionally to block-cache. This may result in underutilization of the memory due to duplicate entries per row, for example, when hot data is continuously updated. Generally, the faster the data is accumulated in memory, more flushes are triggered, the data sinks to disk more frequently, slowing down retrieval of data, even if very recent. In high-churn workloads, compacting the memstore can help maintain the data in memory, and thereby speed up data retrieval. We suggest a new compacted memstore with the following principles: 1.The data is kept in memory for as long as possible 2.Memstore data is either compacted or in process of being compacted 3.Allow a panic mode, which may interrupt an in-progress compaction and force a flush of part of the memstore. We suggest applying this optimization only to in-memory column families. A design document is attached. This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14250) branch-1.1 hbase-server test-jar has incorrect LICENSE
[
https://issues.apache.org/jira/browse/HBASE-14250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705979#comment-14705979
]
Nick Dimiduk edited comment on HBASE-14250 at 8/20/15 11:36 PM:
{noformat}
diff --git a/dev-support/make_rc.sh b/dev-support/make_rc.sh
index 0d0319b..0f0d4a4 100755
--- a/dev-support/make_rc.sh
+++ b/dev-support/make_rc.sh
@@ -54,15 +54,27 @@ function tgz_mover {
}
function deploy {
- MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests -Prelease
- MAVEN_OPTS=${mvnopts} ${mvn} install -DskipTests post-site assembly:single
-Prelease
+ MAVEN_OPTS=${mvnopts} ${mvn} -X clean install -DskipTests -Prelease 21
clean_install.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE clean_install.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
clean_install.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE clean_install.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
clean_install.log
+ MAVEN_OPTS=${mvnopts} ${mvn} -X install -DskipTests post-site
assembly:single -Prelease 21 install_assemble.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE install_assemble.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
install_assemble.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE install_assemble.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
install_assemble.log
tgz_mover
- MAVEN_OPTS=${mvnopts} ${mvn} deploy -DskipTests -Papache-release -Prelease
+ MAVEN_OPTS=${mvnopts} ${mvn} install -DskipTests -Papache-release
-Prelease install_apache-release.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/NOTICE install_apache-release.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
install_apache-release.log
+ echo unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar
META-INF/LICENSE install_apache-release.log
+ unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
install_apache-release.log
}
# Build src tarball
-MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests assembly:single
-Dassembly.file=${HBASE_HOME}/hbase-assembly/src/main/assembly/src.xml
-Prelease
-tgz_mover
+#MAVEN_OPTS=${mvnopts} ${mvn} clean install -DskipTests assembly:single
-Dassembly.file=${HBASE_HOME}/hbase-assembly/src/main/assembly/src.xml
-Prelease
+#tgz_mover
# Now do the two builds, one for hadoop1, then hadoop2
deploy
{noformat}
{noformat}
$ grep -A5 unzip clean_install.log
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
Apache HBase - Server
Copyright 2007-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
--
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
$ tail clean_install.log
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an AS IS BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
{noformat}
{noformat}
$ grep -A5 unzip install_a
install_apache-release.log install_assemble.log
hw11397:hbase ndimiduk$ grep -A5 unzip install_assemble.log
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/NOTICE
Apache HBase - Server
Copyright 2007-2015 The Apache Software Foundation
This product includes software developed at
The Apache Software Foundation (http://www.apache.org/).
--
unzip -p hbase-server/target/hbase-server-1.1.2-tests.jar META-INF/LICENSE
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
$ tail install_assemble.log
Work is licensed under the CCPL, Creative Commons does not authorize
the use by either party of the trademark Creative Commons or any
related trademark or logo of Creative Commons without the prior
written consent of Creative Commons. Any permitted use will be in
compliance with Creative Commons' then-current trademark usage
guidelines, as may be published on its website or otherwise made
available upon request from time to time.
[jira] [Commented] (HBASE-14251) javadoc jars use LICENSE/NOTICE from primary artifact
[ https://issues.apache.org/jira/browse/HBASE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706080#comment-14706080 ] Hudson commented on HBASE-14251: SUCCESS: Integrated in HBase-0.98 #1091 (See [https://builds.apache.org/job/HBase-0.98/1091/]) HBASE-14251 ensure javadoc jars have LICENSE/NOTICE that match content. (busbey: rev 1fc9a6e0b1138295c508b288fce6e0a616505695) * hbase-thrift/src/main/javadoc/META-INF/NOTICE * hbase-thrift/src/main/javadoc/META-INF/LICENSE * hbase-server/src/main/javadoc/META-INF/LICENSE * hbase-thrift/pom.xml * hbase-server/pom.xml * hbase-common/pom.xml * hbase-server/src/main/javadoc/META-INF/NOTICE * hbase-common/src/main/javadoc/META-INF/NOTICE javadoc jars use LICENSE/NOTICE from primary artifact - Key: HBASE-14251 URL: https://issues.apache.org/jira/browse/HBASE-14251 Project: HBase Issue Type: Bug Components: build Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0, 0.98.15, 1.0.3 Attachments: HBASE-14251.1.patch Our generated javadoc jars have the same LICENSE/NOTICE files as our primary artifacts but do not include a copy of hte full source. the following modules end up with incorrect artifacts: * hbase-server * hbase-common (maybe? depends on the are-apis-copyrightable court case) * hbase-thrift -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14264) Update RAT plugin version in 0.98 branch
[ https://issues.apache.org/jira/browse/HBASE-14264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706081#comment-14706081 ] Hudson commented on HBASE-14264: SUCCESS: Integrated in HBase-0.98 #1091 (See [https://builds.apache.org/job/HBase-0.98/1091/]) HBASE-14264 Update RAT plugin version in 0.98 branch (apurtell: rev 0ed23036270fba1ef9982a72761d9a5beac42c3a) * pom.xml Update RAT plugin version in 0.98 branch Key: HBASE-14264 URL: https://issues.apache.org/jira/browse/HBASE-14264 Project: HBase Issue Type: Task Components: build Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Fix For: 0.98.14 Attachments: HBASE-14264-0.98.patch Update the RAT plugin version in 0.98 branch to 0.11. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14225) Exclude **/target/** from RAT checks
[ https://issues.apache.org/jira/browse/HBASE-14225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706079#comment-14706079 ] Hudson commented on HBASE-14225: SUCCESS: Integrated in HBase-0.98 #1091 (See [https://builds.apache.org/job/HBase-0.98/1091/]) HBASE-14225 Exclude **/target/** from RAT checks (apurtell: rev d96563c3acc548e1400513126d94e1cca2bc6e86) * pom.xml Exclude **/target/** from RAT checks Key: HBASE-14225 URL: https://issues.apache.org/jira/browse/HBASE-14225 Project: HBase Issue Type: Bug Components: build Reporter: Andrew Purtell Assignee: Andrew Purtell Fix For: 0.98.14 Attachments: HBASE-14225-0.98.patch We need to exclude **/target/** from RAT checks in order to build releases now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14269) FuzzyRowFilter omits certain rows when multiple fuzzy key exist
[ https://issues.apache.org/jira/browse/HBASE-14269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706179#comment-14706179 ] Vladimir Rodionov commented on HBASE-14269: --- I did not have time to check the patch yet, but 'priority queue' scares me a bit. Will check the the code later. FuzzyRowFilter omits certain rows when multiple fuzzy key exist --- Key: HBASE-14269 URL: https://issues.apache.org/jira/browse/HBASE-14269 Project: HBase Issue Type: Bug Components: Filters Reporter: hongbin ma Assignee: hongbin ma Attachments: HBASE-14269-v1.patch, HBASE-14269.patch https://issues.apache.org/jira/browse/HBASE-13761 introduced a RowTracker in FuzzyRowFilter to avoid performing getNextForFuzzyRule() for each fuzzy key on each getNextCellHint() by maintaining a list of possible row matches for each fuzzy key. The implementation assumes that the prepared rows will be matched one by one, so it removes the first row in the list as soon as it is used. However, this approach may lead to omitting rows in some cases: Consider a case where we have two fuzzy keys: 1?1 2?2 and the data is like: 000 111 112 121 122 211 212 when the first row 000 fails to match, RowTracker will update possible row matches with cell 000 and fuzzy keys 1?1,2?2. This will populate RowTracker with 101 and 202. Then 101 is popped out of RowTracker, hint the scanner to go to row 101. The scanner will get 111 and find it is a match, and continued to find that 112 is not a match, getNextCellHint will be called again. Then comes the bug: Row 101 has been removed out of RowTracker, so RowTracker will jump to 202. As you see row 121 will be omitted, but it is actually a match for fuzzy key 1?1. I will illustrate the bug by adding a new test case in TestFuzzyRowFilterEndToEnd. Also I will provide the bug fix in my patch. The idea of the new solution is to maintain a priority queue for all the possible match rows for each fuzzy key, and whenever getNextCellHint is called, the elements in the queue that are smaller than the parameter currentCell will be updated(and re-insert into the queue). The head of queue will always be the Next cell hint. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14226) Merge group admin APIs into Admin
[ https://issues.apache.org/jira/browse/HBASE-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705345#comment-14705345 ] Francis Liu commented on HBASE-14226: - Great will put up an addendum for both 0.98 and trunk in a bit. Merge group admin APIs into Admin - Key: HBASE-14226 URL: https://issues.apache.org/jira/browse/HBASE-14226 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Assignee: Francis Liu Labels: hbase-6721 The initial implementation strategy proposed for group based assignment was coprocessor based, but we went a different route. Support for the group admin APIs will be universal with it in core, so consider merging these APIs into Admin. When doing so, consider where existing APIs can be overloaded to hold down the number of methods in Admin. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14272) Enforce major compaction on stores with KEEP_DELETED_CELLS=true
Vladimir Rodionov created HBASE-14272: - Summary: Enforce major compaction on stores with KEEP_DELETED_CELLS=true Key: HBASE-14272 URL: https://issues.apache.org/jira/browse/HBASE-14272 Project: HBase Issue Type: Bug Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 Currently, if store has one (major compacted) file, the only case when major compaction will be triggered for this file again when locality is below threshold defined by *hbase.hstore.min.locality.to.skip.major.compact*. If file has locality greater than this threshold it will never be major compacted. This does not work for files with with locality 1.0 and KEEP_DELETED_CELLS=true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14263) ExploringCompactionPolicy logic around file selection is broken
[
https://issues.apache.org/jira/browse/HBASE-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14705379#comment-14705379
]
stack commented on HBASE-14263:
---
Patch looks reasonable mighty [~vrodionov] Any chance of our getting a test in
here so can guard against regression? Thanks.
ExploringCompactionPolicy logic around file selection is broken
---
Key: HBASE-14263
URL: https://issues.apache.org/jira/browse/HBASE-14263
Project: HBase
Issue Type: Bug
Reporter: Vladimir Rodionov
Assignee: Vladimir Rodionov
Fix For: 2.0.0
Attachments: HBASE-14263.patch
It seems that logic around selection of store file candidates is broken:
{code}
// Compute the total size of files that will
// have to be read if this set of files is compacted.
long size = getTotalStoreSize(potentialMatchFiles);
// Store the smallest set of files. This stored set of files will be
used
// if it looks like the algorithm is stuck.
if (mightBeStuck size smallestSize) {
smallest = potentialMatchFiles;
smallestSize = size;
}
if (size comConf.getMaxCompactSize()) {
continue;
}
++opts;
if (size = comConf.getMinCompactSize()
!filesInRatio(potentialMatchFiles, currentRatio)) {
continue;
}
{code}
This is from applyCompactionPolicy method. As you can see, both min
compaction size and max compaction size are applied to a *selection* of files
and not to individual files. It mostly works as expected only because nobody
seems using non-default hbase.hstore.compaction.max.size, which is
Long.MAX_VALUE and it is not that easy to figure out what is going
on on an opposite side (why small files do not get included?)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
