[jira] [Commented] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723050#comment-14723050
]
Anoop Sam John commented on HBASE-14307:
Looks good. Minor comments
@return true if succeeded reading the extra bytes, or reading extra bytes
737* was not required
Why say reading extra not required? When is that? Seems we return true when
necessary+ extra read. false when only necessary read and even necessary also
not read, we throw IOE. Mind correcting doc?
return bytesRead != necessaryLen && bytesRemaining <= 0;
Return can be like bytesRemaining ==0 only?
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Attachments: HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14279) Race condition in ConcurrentIndex
[
https://issues.apache.org/jira/browse/HBASE-14279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723055#comment-14723055
]
Heng Chen commented on HBASE-14279:
---
{quote}
There is a similar class, KeyLoccker, whose key type is a generic one, and you
can use it without copying its implementation logic. The implementation of
IdLocker and KeyLocker is not good and should be re-written, and I believe it
is better to avoid to copy their implementation.
{quote}
yeah, you are right. {{KeyLocker}} is a better choice.
> Race condition in ConcurrentIndex
> -
>
> Key: HBASE-14279
> URL: https://issues.apache.org/jira/browse/HBASE-14279
> Project: HBase
> Issue Type: Bug
>Reporter: Hiroshi Ikeda
>Assignee: Heng Chen
>Priority: Minor
> Attachments: HBASE-14279.patch, HBASE-14279_v2.patch
>
>
> {{ConcurrentIndex.put}} and {{remove}} are in race condition. It is possible
> to remove a non-empty set, and to add a value to a removed set. Also
> {{ConcurrentIndex.values}} is vague in sense that the returned set sometimes
> trace the current state and sometimes doesn't.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hiroshi Ikeda updated HBASE-14331:
--
Attachment: BlockingQueuesPerformanceTestApp-output.pdf
BlockingQueuesPerformanceTestApp-output.txt
Added a result of the performance test in my environment, which has 8 cpu cores.
It compares (1){{LinkedBlockingQueue}} in the standard Java library,
(2){{BoundedPriorityBlockingQueue}} in HBase, and the newly implemented
classes, (3){{SemaphoreBasedLinkedBlockingQueue}} and
(4){{SemaphoreBasedPriorityBlockingQueue}}. I just changed the code of the test
application to add (2){{BoundedPriorityBlockingQueue}}.
The tasks are just executing {{Thread.yield}} and there should be excessively
context switches, but the result says, the reading threads in the existing
queues, (1){{LinkedBlockingQueue}} and (2){{BoundPriorityBlockingQueue}}, are
rarely waken up. That means many reading threads are blocked in the queues.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14342) Recursive call in RegionMergeTransactionImpl.getJournal()
Lars George created HBASE-14342:
---
Summary: Recursive call in RegionMergeTransactionImpl.getJournal()
Key: HBASE-14342
URL: https://issues.apache.org/jira/browse/HBASE-14342
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.1.1
Reporter: Lars George
Fix For: 1.2.0, 1.1.2
HBASE-12975 in its branch-1 patch
(https://issues.apache.org/jira/secure/attachment/12708578/HBASE-12975-branch-1.patch)
introduced a recursive call for {{getJournal()}}. Needs to return just the
{{journal}} variable like master patch does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[
https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724710#comment-14724710
]
Hadoop QA commented on HBASE-14261:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753421/HBASE-14261-branch-1_v3.patch
against branch-1 branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753421
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 30 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+throw new IOException("Hadoop home configuration parameter i.e.
'hbase.it.clustermanager.hadoop.home' " +
+throw new IOException("Zookeeper home configuration parameter i.e.
'hbase.it.clustermanager.zookeeper.home' " +
+ private Pair exec(String hostname, ServiceType service,
String... cmd) throws IOException {
+String ret = execWithRetries(hostname, service,
getCommandProvider(service).isRunningCommand(service))
+ public static final String KILL_ZK_NODE_TIMEOUT_KEY =
"hbase.chaosmonkey.action.killzknodetimeout";
+ public static final String START_ZK_NODE_TIMEOUT_KEY =
"hbase.chaosmonkey.action.startzknodetimeout";
+ public static final String KILL_DATANODE_TIMEOUT_KEY =
"hbase.chaosmonkey.action.killdatanodetimeout";
+ public static final String START_DATANODE_TIMEOUT_KEY =
"hbase.chaosmonkey.action.startdatanodetimeout";
+killZkNodeTimeout = cluster.getConf().getLong(KILL_ZK_NODE_TIMEOUT_KEY,
KILL_ZK_NODE_TIMEOUT_DEFAULT);
+startZkNodeTimeout = cluster.getConf().getLong(START_ZK_NODE_TIMEOUT_KEY,
START_ZK_NODE_TIMEOUT_DEFAULT);
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.security.access.TestCellACLs.testCoveringCheck(TestCellACLs.java:405)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15372//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15372//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15372//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15372//console
This message is automatically generated.
> Enhance Chaos Monkey framework by adding zookeeper and datanode fault
> injections.
> -
>
> Key: HBASE-14261
> URL: https://issues.apache.org/jira/browse/HBASE-14261
> Project: HBase
> Issue Type: Improvement
>Reporter: Srikanth Srungarapu
>Assignee: Srikanth Srungarapu
> Attachments: HBASE-14261-branch-1.patch,
> HBASE-14261-branch-1_v3.patch, HBASE-14261.branch-1_v2.patch
>
>
> One of the shortcomings of existing ChaosMonkey framework is lack of fault
> injections for hbase dependencies like zookeeper, hdfs etc. This patch
> attempts to solve this problem partially by adding datanode and zk node fault
> injections.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14314) Metrics for block cache should take region replicas into account
[
https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724706#comment-14724706
]
Ted Yu commented on HBASE-14314:
{code}
Printing Failing tests
Failing test : org.apache.hadoop.hbase.TestIOFencing
Failing test : org.apache.hadoop.hbase.master.TestDistributedLogSplitting
{code}
The above two failed in other QA runs too.
TestIOFencing is being tracked by HBASE-14327
> Metrics for block cache should take region replicas into account
>
>
> Key: HBASE-14314
> URL: https://issues.apache.org/jira/browse/HBASE-14314
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14314-v1.txt, 14314-v2.txt, 14314-v3.txt
>
>
> Currently metrics for block cache are aggregates in the sense that they don't
> distinguish primary from secondary / tertiary replicas.
> This JIRA separates the block cache metrics for primary region replica from
> the aggregate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14261) Enhance Chaos Monkey framework by adding zookeeper and datanode fault injections.
[
https://issues.apache.org/jira/browse/HBASE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724588#comment-14724588
]
Enis Soztutar commented on HBASE-14261:
---
LGTM, depending on hadoopqa. Did you test the patch?
Not sure whether {{hadoop}} or {{hdfs}} would be a better default. At least it
is configurable.
{code}
return conf.get("hbase.it.clustermanager.hadoop.hdfs.user", "hadoop");
{code}
> Enhance Chaos Monkey framework by adding zookeeper and datanode fault
> injections.
> -
>
> Key: HBASE-14261
> URL: https://issues.apache.org/jira/browse/HBASE-14261
> Project: HBase
> Issue Type: Improvement
>Reporter: Srikanth Srungarapu
>Assignee: Srikanth Srungarapu
> Attachments: HBASE-14261-branch-1.patch,
> HBASE-14261-branch-1_v3.patch, HBASE-14261.branch-1_v2.patch
>
>
> One of the shortcomings of existing ChaosMonkey framework is lack of fault
> injections for hbase dependencies like zookeeper, hdfs etc. This patch
> attempts to solve this problem partially by adding datanode and zk node fault
> injections.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11236) Last flushed sequence id is ignored by ServerManager
[
https://issues.apache.org/jira/browse/HBASE-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724638#comment-14724638
]
Duo Zhang commented on HBASE-11236:
---
I think this could happen without HBASE-13811 where [~stack] fix a issue that
we may report a wrong last flushed sequence id if a flush is aborted.
[~eclark] Any more informations? What's happened to the region before this
log(reassignment or flush?)? Maybe there are other issues.
Thanks.
> Last flushed sequence id is ignored by ServerManager
>
>
> Key: HBASE-11236
> URL: https://issues.apache.org/jira/browse/HBASE-11236
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>
> I got lots of error messages like this:
> {quote}
> 2014-05-22 08:58:59,793 DEBUG [RpcServer.handler=1,port=20020]
> master.ServerManager: RegionServer
> a2428.halxg.cloudera.com,20020,1400742071109 indicates a last flushed
> sequence id (numberOfStores=9, numberOfStorefiles=2,
> storefileUncompressedSizeMB=517, storefileSizeMB=517,
> compressionRatio=1., memstoreSizeMB=0, storefileIndexSizeMB=0,
> readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=34,
> totalStaticIndexSizeKB=381, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
> currentCompactedKVs=0, compactionProgressPct=NaN) that is less than the
> previous last flushed sequence id (605446) for region
> IntegrationTestBigLinkedList,
> �A��*t�^FU�2��0,1400740489477.a44d3e309b5a7e29355f6faa0d3a4095. Ignoring.
> {quote}
> RegionLoad.toString doesn't print out the last flushed sequence id passed in.
> Why is it less than the previous one?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14337) build error on master
[ https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14337: Attachment: HBASE-14227_v5.patch > build error on master > - > > Key: HBASE-14337 > URL: https://issues.apache.org/jira/browse/HBASE-14337 > Project: HBase > Issue Type: Bug >Reporter: Gabor Liptak >Assignee: Sean Busbey > Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch, > HBASE-14227_v5.patch, HBASE-14337.1.patch > > > In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console > /home/jenkins/tools/maven/latest/bin/mvn clean package > checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests > -DHBasePatchProcess > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt > 2>&1 > mvn exit code was 0 > mv: cannot stat 'target/checkstyle-result.xml': No such file or directory > Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [ > ... ] > Expected 2...2147483647 file arguments, found 1 > Options: > -withMessagesGenerated XML should contain msgs for external > processing > -output File in which to store combined results > Exception in thread "main" java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml > (No such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:146) > at > edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295) > at > edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235) > at > edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml' > (No such file or directory) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html' > (No such file or directory) > /home/jenkins/tools/maven/latest/bin/mvn clean install -DskipTests > -DHBasePatchProcess -Dhadoop-two.version=2.4.0 > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchJavacWithHadoop-2.4.0.txt > 2>&1 > == > There are compilation errors with Hadoop version 2.4.0. > == > [ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList > at META-INF/LICENSE.vm[line 1619, column 22] > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) > on project hbase-assembly: Error rendering velocity resource. Error invoking > method 'get(java.lang.Integer)' in java.util.ArrayList at > META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: > 0, Size: 0 -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :hbase-assembly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14337) build error on master
[
https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723514#comment-14723514
]
Hadoop QA commented on HBASE-14337:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753301/HBASE-14227_v5.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753301
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15347//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15347//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15347//console
This message is automatically generated.
> build error on master
> -
>
> Key: HBASE-14337
> URL: https://issues.apache.org/jira/browse/HBASE-14337
> Project: HBase
> Issue Type: Bug
>Reporter: Gabor Liptak
>Assignee: Sean Busbey
> Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch,
> HBASE-14227_v5.patch, HBASE-14337.1.patch
>
>
> In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console
> /home/jenkins/tools/maven/latest/bin/mvn clean package
> checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests
> -DHBasePatchProcess >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt
> 2>&1
> mvn exit code was 0
> mv: cannot stat 'target/checkstyle-result.xml': No such file or directory
> Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [
> ... ]
> Expected 2...2147483647 file arguments, found 1
> Options:
> -withMessagesGenerated XML should contain msgs for external
> processing
> -output File in which to store combined results
> Exception in thread "main" java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159)
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml:
> ERROR: cannot open
> `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml'
> (No such file or directory)
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html:
> ERROR: cannot open
> `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html'
> (No such file or directory)
> /home/jenkins/tools/maven/latest/bin/mvn clean install -DskipTests
> -DHBasePatchProcess -Dhadoop-two.version=2.4.0 >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchJavacWithHadoop-2.4.0.txt
> 2>&1
> ==
> There are compilation errors with Hadoop version 2.4.0.
> ==
> [ERROR] Error invoking method
[jira] [Resolved] (HBASE-6613) Automatically merge empty regions
[ https://issues.apache.org/jira/browse/HBASE-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6613. -- Resolution: Duplicate Resolving in favor of the work over in HBASE-13103 > Automatically merge empty regions > - > > Key: HBASE-6613 > URL: https://issues.apache.org/jira/browse/HBASE-6613 > Project: HBase > Issue Type: New Feature > Components: master, regionserver, util >Affects Versions: 0.94.1 >Reporter: Ionut Ignatescu > > Consider an usecase where row keys has an increasing value(time-series data > for example) and data retention is set to a concrete value(60 days for > example). > After a period of time, longer than retention, empty regions will appear. > This will cause high memory use on region servers. > In my opinion, regions merge could be part of major compaction or another > tool should be provided. From my understading, it is possible to merge 2 > empty regions without make table offline, but it's not possible to merge one > empty region with a non-empty region without close/unassing this regions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723536#comment-14723536
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12752388/HBASE-13158_V6.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12752388
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15348//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15348//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15348//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 13158v4.suggestion.txt, HBASE-13158.patch,
> HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723542#comment-14723542
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753305/HBASE-13158_V6.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753305
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15350//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15350//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15350//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 13158v4.suggestion.txt, HBASE-13158.patch,
> HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13158: -- Attachment: HBASE-13158_V6.patch > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, > HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, > HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14321) branch-1.1 src artifact includes shaded target dirs
[
https://issues.apache.org/jira/browse/HBASE-14321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723575#comment-14723575
]
Nick Dimiduk commented on HBASE-14321:
--
mind giving the latest patch on HBASE-14318 a spin, [~busbey]? I can update it
to correct the ref guide's instructions too.
> branch-1.1 src artifact includes shaded target dirs
> ---
>
> Key: HBASE-14321
> URL: https://issues.apache.org/jira/browse/HBASE-14321
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 1.1.2
>Reporter: Nick Dimiduk
>Priority: Blocker
>
> Fixed the make_rc script, now I find in the src binary.
> {noformat}
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-archiver/
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-archiver/
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/pom.xml
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/.plxarc
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/LICENSE
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/classes/META-INF/NOTICE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/dependency-reduced-pom.xml
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-1.1.2.jar
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-archiver/pom.properties
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/NOTICE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/original-hbase-shaded-client-1.1.2.jar
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/rat.txt
> x hbase-1.1.2/hbase-shaded/hbase-shaded-client/target/supplemental-models.xml
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/pom.xml
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/.plxarc
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/LICENSE
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/classes/META-INF/NOTICE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/dependency-reduced-pom.xml
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/hbase-shaded-server-1.1.2.jar
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-archiver/pom.properties
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/LICENSE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/maven-shared-archive-resources/META-INF/NOTICE
> x
> hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/original-hbase-shaded-server-1.1.2.jar
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/rat.txt
> x hbase-1.1.2/hbase-shaded/hbase-shaded-server/target/supplemental-models.xml
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13158: -- Attachment: HBASE-13158_V6.patch > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, > HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, > HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch, > HBASE-13158_V6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yerui Sun updated HBASE-14229: -- Attachment: HBASE-14229-branch-1-v2.patch Sorry for my delaying over one week. The new patch has passed all tests in local. CheckStyle increased one VisibilityModifier error in HRegion.PrepareFlushResult, which has same errors in current code.I think it's should be ignored. > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Yerui Sun > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch, > HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[
https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723606#comment-14723606
]
Hadoop QA commented on HBASE-14229:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753310/HBASE-14229-branch-1-v2.patch
against branch-1 branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753310
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15353//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15353//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15353//console
This message is automatically generated.
> Flushing canceled by coprocessor still leads to memstoreSize set down
> -
>
> Key: HBASE-14229
> URL: https://issues.apache.org/jira/browse/HBASE-14229
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1
>Reporter: Yerui Sun
> Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch,
> HBASE-14229-branch-1.patch
>
>
> A Coprocessor override "public InternalScanner preFlush(final Store store,
> final InternalScanner scanner)" and return NULL when calling this method,
> will cancel flush request, leaving snapshot un-flushed, and no new storefile
> created. But the HRegion.internalFlushCache still set down memstoreSize to 0
> by totalFlushableSize.
> If there's no write requests anymore, the memstoreSize will remaining as 0,
> and no more flush quests will be processed because of the checking of
> memstoreSize.get() <=0 at the beginning of internalFlushCache.
> This issue may not cause data loss, but it will confuse coprocessor users. If
> we argree with this, I'll apply a patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723616#comment-14723616
]
Lars George commented on HBASE-13103:
-
Is there follow up work or a JIRA tracking adding this to the shell? Is the
only way to enable this per table using the Java API?
> [ergonomics] add region size balancing as a feature of master
> -
>
> Key: HBASE-13103
> URL: https://issues.apache.org/jira/browse/HBASE-13103
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Usability
>Reporter: Nick Dimiduk
>Assignee: Mikhail Antonov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch,
> HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch
>
>
> Often enough, folks miss-judge split points or otherwise end up with a
> suboptimal number of regions. We should have an automated, reliable way to
> "reshape" or "balance" a table's region boundaries. This would be for tables
> that contain existing data. This might look like:
> {noformat}
> Admin#reshapeTable(TableName, int numSplits);
> {noformat}
> or from the shell:
> {noformat}
> > reshape TABLE, numSplits
> {noformat}
> Better still would be to have a maintenance process, similar to the existing
> Balancer that runs AssignmentManager on an interval, to run the above
> "reshape" operation on an interval. That way, the cluster will automatically
> self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723619#comment-14723619
]
Lars George commented on HBASE-13103:
-
Sorry, above was for [~mantonov] I guess. :) Please advise.
> [ergonomics] add region size balancing as a feature of master
> -
>
> Key: HBASE-13103
> URL: https://issues.apache.org/jira/browse/HBASE-13103
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Usability
>Reporter: Nick Dimiduk
>Assignee: Mikhail Antonov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch,
> HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch
>
>
> Often enough, folks miss-judge split points or otherwise end up with a
> suboptimal number of regions. We should have an automated, reliable way to
> "reshape" or "balance" a table's region boundaries. This would be for tables
> that contain existing data. This might look like:
> {noformat}
> Admin#reshapeTable(TableName, int numSplits);
> {noformat}
> or from the shell:
> {noformat}
> > reshape TABLE, numSplits
> {noformat}
> Better still would be to have a maintenance process, similar to the existing
> Balancer that runs AssignmentManager on an interval, to run the above
> "reshape" operation on an interval. That way, the cluster will automatically
> self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14325: Resolution: Fixed Fix Version/s: 1.0.3 0.98.15 1.1.2 1.2.0 2.0.0 Status: Resolved (was: Patch Available) > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723640#comment-14723640
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753309/HBASE-13158_V6.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753309
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 javac{color}. The patch appears to cause mvn compile goal to
fail with Hadoop version 2.7.0.
Compilation errors resume:
[ERROR] Failed to execute goal on project hbase-server: Could not resolve
dependencies for project org.apache.hbase:hbase-server:jar:2.0.0-SNAPSHOT:
Failed to collect dependencies at
org.apache.hadoop:hadoop-minicluster:jar:2.7.0 ->
org.apache.hadoop:hadoop-yarn-server-tests:jar:tests:2.7.0 ->
org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.7.0 ->
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:2.7.0:
Failed to read artifact descriptor for
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:jar:2.7.0: Could
not transfer artifact
org.apache.hadoop:hadoop-yarn-server-applicationhistoryservice:pom:2.7.0
from/to central (http://repo.maven.apache.org/maven2): Failed to transfer file:
http://repo.maven.apache.org/maven2/org/apache/hadoop/hadoop-yarn-server-applicationhistoryservice/2.7.0/hadoop-yarn-server-applicationhistoryservice-2.7.0.pom.
Return code is: 503 , ReasonPhrase:backend read error. -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hbase-server
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15352//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 13158v4.suggestion.txt, HBASE-13158.patch,
> HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch,
> HBASE-13158_V6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[
https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723639#comment-14723639
]
Ted Yu commented on HBASE-14229:
The test suite wasn't run:
{code}
WARNING: 2 rogue build processes detected, terminating.
{code}
Mind attaching your patch again ?
> Flushing canceled by coprocessor still leads to memstoreSize set down
> -
>
> Key: HBASE-14229
> URL: https://issues.apache.org/jira/browse/HBASE-14229
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1
>Reporter: Yerui Sun
> Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch,
> HBASE-14229-branch-1.patch
>
>
> A Coprocessor override "public InternalScanner preFlush(final Store store,
> final InternalScanner scanner)" and return NULL when calling this method,
> will cancel flush request, leaving snapshot un-flushed, and no new storefile
> created. But the HRegion.internalFlushCache still set down memstoreSize to 0
> by totalFlushableSize.
> If there's no write requests anymore, the memstoreSize will remaining as 0,
> and no more flush quests will be processed because of the checking of
> memstoreSize.get() <=0 at the beginning of internalFlushCache.
> This issue may not cause data loss, but it will confuse coprocessor users. If
> we argree with this, I'll apply a patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yerui Sun updated HBASE-14229: -- Attachment: HBASE-14229-branch-1-v2.patch > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Yerui Sun > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch, > HBASE-14229-branch-1-v2.patch, HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723654#comment-14723654 ] Hudson commented on HBASE-14325: SUCCESS: Integrated in HBase-1.2-IT #121 (See [https://builds.apache.org/job/HBase-1.2-IT/121/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev c331685c95e1aba8c1aad538f4acdf4fe6e36352) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723670#comment-14723670 ] Ted Yu commented on HBASE-14314: w.r.t. using new boolean flag vs change in method name , I don't have strong preference. Just want to get more review comment before making the next patch. > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723677#comment-14723677 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-1.3-IT #124 (See [https://builds.apache.org/job/HBase-1.3-IT/124/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev f30d3dc20c03523b6cf452f70190f1a55dd74978) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723674#comment-14723674 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-1.1 #643 (See [https://builds.apache.org/job/HBase-1.1/643/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev 12f4c1ace67f9b829bcb67420a52119480cf31e9) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723679#comment-14723679
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753309/HBASE-13158_V6.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753309
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:red}-1 javac{color}. The applied patch generated 26 javac compiler
warnings (more than the master's current 0 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15354//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15354//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15354//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15354//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 13158v4.suggestion.txt, HBASE-13158.patch,
> HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch,
> HBASE-13158_V6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[ https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-13158: -- Attachment: HBASE-13158_V6.patch > When client supports CellBlock, return the result Cells as controller payload > for get(Get) API also > --- > > Key: HBASE-13158 > URL: https://issues.apache.org/jira/browse/HBASE-13158 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.3.0 > > Attachments: 13158v4.suggestion.txt, HBASE-13158.patch, > HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch, > HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch, > HBASE-13158_V6.patch, HBASE-13158_V6.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723687#comment-14723687 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-1.0 #1035 (See [https://builds.apache.org/job/HBase-1.0/1035/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev 21f9910626936daaf3a8a9a642c73fef591ae0f4) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14339) HBase Bulk Load and super wide rows
[ https://issues.apache.org/jira/browse/HBASE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723684#comment-14723684 ] Nick Dimiduk commented on HBASE-14339: -- See also HBASE-7743. > HBase Bulk Load and super wide rows > --- > > Key: HBASE-14339 > URL: https://issues.apache.org/jira/browse/HBASE-14339 > Project: HBase > Issue Type: Bug >Reporter: Ted Malaska >Priority: Minor > > This may not be a huge issues but it does come up. If the number of columns > in a row are to many then KeyValueSortReducer will blow up with a out of > memory exception, because it uses a TreeMap to sort the columns with in the > memory of the reducer. > A solution would be to add the column family and qualifier to the key so the > shuffle would handle the sort. > The partitioner would only partition on the rowKey but ordering would apply > to the RowKey, Column Family, and Column Qualifier. > Look at the Spark Bulk load as an example. HBASE-14150 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14342) Recursive call in RegionMergeTransactionImpl.getJournal()
[
https://issues.apache.org/jira/browse/HBASE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723688#comment-14723688
]
Ted Yu commented on HBASE-14342:
lgtm
> Recursive call in RegionMergeTransactionImpl.getJournal()
> -
>
> Key: HBASE-14342
> URL: https://issues.apache.org/jira/browse/HBASE-14342
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.1
>Reporter: Lars George
>Assignee: Lars Francke
> Labels: beginner
> Fix For: 1.2.0, 1.1.2
>
> Attachments: HBASE-14342-branch-1.patch
>
>
> HBASE-12975 in its branch-1 patch
> (https://issues.apache.org/jira/secure/attachment/12708578/HBASE-12975-branch-1.patch)
> introduced a recursive call for {{getJournal()}}. Needs to return just the
> {{journal}} variable like master patch does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13158) When client supports CellBlock, return the result Cells as controller payload for get(Get) API also
[
https://issues.apache.org/jira/browse/HBASE-13158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723689#comment-14723689
]
Hadoop QA commented on HBASE-13158:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753320/HBASE-13158_V6.patch
against master branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753320
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15356//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15356//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15356//console
This message is automatically generated.
> When client supports CellBlock, return the result Cells as controller payload
> for get(Get) API also
> ---
>
> Key: HBASE-13158
> URL: https://issues.apache.org/jira/browse/HBASE-13158
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.3.0
>
> Attachments: 13158v4.suggestion.txt, HBASE-13158.patch,
> HBASE-13158_V2.patch, HBASE-13158_V3.patch, HBASE-13158_V4.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch, HBASE-13158_V6.patch,
> HBASE-13158_V6.patch, HBASE-13158_V6.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14344) Add timeouts to TestHttpServerLifecycle
Matteo Bertozzi created HBASE-14344: --- Summary: Add timeouts to TestHttpServerLifecycle Key: HBASE-14344 URL: https://issues.apache.org/jira/browse/HBASE-14344 Project: HBase Issue Type: Test Components: test Affects Versions: 2.0.0, 1.2.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0, 1.2.0 Attachments: HBASE-14344-v0.patch I got TestHttpServerLifecycle hanging a couple of times on my run. simple patch to add a timeout to the tests, and avoid the build to hang. (i haven't looked at them yet to see what was the source problem) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14344) Add timeouts to TestHttpServerLifecycle
[ https://issues.apache.org/jira/browse/HBASE-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-14344: Attachment: HBASE-14344-v0.patch > Add timeouts to TestHttpServerLifecycle > --- > > Key: HBASE-14344 > URL: https://issues.apache.org/jira/browse/HBASE-14344 > Project: HBase > Issue Type: Test > Components: test >Affects Versions: 2.0.0, 1.2.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 1.2.0 > > Attachments: HBASE-14344-v0.patch > > > I got TestHttpServerLifecycle hanging a couple of times on my run. > simple patch to add a timeout to the tests, and avoid the build to hang. > (i haven't looked at them yet to see what was the source problem) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723728#comment-14723728 ] Ted Yu commented on HBASE-14229: +1 on patch v2. > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Yerui Sun > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch, > HBASE-14229-branch-1-v2.patch, HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14337) build error on master
[
https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723735#comment-14723735
]
Hadoop QA commented on HBASE-14337:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753301/HBASE-14227_v5.patch
against master branch at commit df341c4299ea21e4e1ca09652f6126633f2307c5.
ATTACHMENT ID: 12753301
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1848 checkstyle errors (more than the master's current 1846 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager.test(TestReplicationWALReaderManager.java:186)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompactionInternals(TestCacheOnWrite.java:484)
at
org.apache.hadoop.hbase.io.hfile.TestCacheOnWrite.testNotCachingDataBlocksDuringCompaction(TestCacheOnWrite.java:508)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15346//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15346//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15346//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15346//console
This message is automatically generated.
> build error on master
> -
>
> Key: HBASE-14337
> URL: https://issues.apache.org/jira/browse/HBASE-14337
> Project: HBase
> Issue Type: Bug
>Reporter: Gabor Liptak
>Assignee: Sean Busbey
> Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch,
> HBASE-14227_v5.patch, HBASE-14337.1.patch
>
>
> In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console
> /home/jenkins/tools/maven/latest/bin/mvn clean package
> checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests
> -DHBasePatchProcess >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt
> 2>&1
> mvn exit code was 0
> mv: cannot stat 'target/checkstyle-result.xml': No such file or directory
> Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [
> ... ]
> Expected 2...2147483647 file arguments, found 1
> Options:
> -withMessagesGenerated XML should contain msgs for external
> processing
> -output File in which to store combined results
> Exception in thread "main" java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159)
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml:
> ERROR: cannot open
> `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml'
> (No such file or directory)
>
[jira] [Commented] (HBASE-14307) Incorrect use of positional read api in HFileBlock
[
https://issues.apache.org/jira/browse/HBASE-14307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723766#comment-14723766
]
Shradha Revankar commented on HBASE-14307:
--
Thanks [~cnauroth], I applied the patch and did some tests, while this might
not be relevant to hbase, from a webhdfs perspective, this approach has a
performance impact, mainly due the way the read() is implemented in webhdfs
client :
{code}
@Override
public int read(long position, byte[] buffer, int offset, int length)
throws IOException {
try (InputStream in = openInputStream(position).in) {
return in.read(buffer, offset, length);
}
}
{code}
Every read at a position opens a new connection and sends a new OPEN request to
the server with the specified offset. So when you loop within
{{positionalReadWithExtra}}, it sends as many requests to the server as the
loop count, the perf impact is very visible with chunked-encoding when the
returned bytes per read is only around 24 bytes.
>From an hbase perpective this fix is still useful and the right way, but I
>think we have to revisit - HDFS-8943
> Incorrect use of positional read api in HFileBlock
> --
>
> Key: HBASE-14307
> URL: https://issues.apache.org/jira/browse/HBASE-14307
> Project: HBase
> Issue Type: Bug
>Reporter: Shradha Revankar
>Assignee: Chris Nauroth
> Attachments: HBASE-14307.001.master.patch,
> HBASE-14307.002.master.patch
>
>
> Considering that {{read()}} is not guaranteed to read all bytes,
> I'm interested to understand this particular piece of code and why is partial
> read treated as an error :
> https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1446-L1450
> Particularly, if hbase were to use a different filesystem, say
> WebhdfsFileSystem, this would not work, please also see
> https://issues.apache.org/jira/browse/HDFS-8943 for discussion around this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14345) Consolidate printUsage in IntegrationTestLoadAndVerify
Nick Dimiduk created HBASE-14345:
Summary: Consolidate printUsage in IntegrationTestLoadAndVerify
Key: HBASE-14345
URL: https://issues.apache.org/jira/browse/HBASE-14345
Project: HBase
Issue Type: Improvement
Components: integration tests
Reporter: Nick Dimiduk
Priority: Trivial
Investigating the use of {{itlav}} is a little screwy. Subclasses are not
overriding the {{printUsage()}} methods correctly, so you have to pass
{{--help}} to get some info and no arguments to get the rest.
{noformat}
[hbase@ndimiduk-112rc2-7 ~]$ hbase
org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify --help
usage: bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
Options:
-h,--help Show usage
-m,--monkey Which chaos monkey to run
-monkeyProps The properties file for specifying chaos monkey
properties.
-ncc,--noClusterCleanUp Don't clean up the cluster at the end
[hbase@ndimiduk-112rc2-7 ~]$ hbase
org.apache.hadoop.hbase.test.IntegrationTestLoadAndVerify
IntegrationTestLoadAndVerify [-Doptions]
Loads a table with row dependencies and verifies the dependency chains
Options
-Dloadmapper.table=Table to write/verify (default autogen)
-Dloadmapper.backrefs=Number of backreferences per row (default 50)
-Dloadmapper.num_to_write=Number of rows per mapper (default 100,000
per mapper)
-Dloadmapper.deleteAfter= Delete after a successful verify (default
true)
-Dloadmapper.numPresplits=Number of presplit regions to start with
(default 40)
-Dloadmapper.map.tasks= Number of map tasks for load (default 200)
-Dverify.reduce.tasks=Number of reduce tasks for verify (default
35)
-Dverify.scannercaching= Number hbase scanner caching rows to read
(default 50)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723813#comment-14723813 ] Nick Dimiduk commented on HBASE-14314: -- bq. Can you give a concrete example when the above would happen ? I don't have an example off the top of my head. Please proceed; pardon me for the delay. > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723820#comment-14723820 ] Ted Yu commented on HBASE-14314: Okay. Will proceed with getHitCount and getPrimaryHitCount method names. > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13103) [ergonomics] add region size balancing as a feature of master
[
https://issues.apache.org/jira/browse/HBASE-13103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723836#comment-14723836
]
Mikhail Antonov commented on HBASE-13103:
-
Normalization enable/disable flag per table is set in HTableDescriptor like,
for example, compaction, so you should be able to do it from shell?
alter 'table1', {NORMALIZATION_ENABLED => 'true'}
> [ergonomics] add region size balancing as a feature of master
> -
>
> Key: HBASE-13103
> URL: https://issues.apache.org/jira/browse/HBASE-13103
> Project: HBase
> Issue Type: Improvement
> Components: Balancer, Usability
>Reporter: Nick Dimiduk
>Assignee: Mikhail Antonov
> Fix For: 2.0.0, 1.2.0, 1.3.0
>
> Attachments: HBASE-13103-branch-1.v3.patch, HBASE-13103-v0.patch,
> HBASE-13103-v1.patch, HBASE-13103-v2.patch, HBASE-13103-v3.patch
>
>
> Often enough, folks miss-judge split points or otherwise end up with a
> suboptimal number of regions. We should have an automated, reliable way to
> "reshape" or "balance" a table's region boundaries. This would be for tables
> that contain existing data. This might look like:
> {noformat}
> Admin#reshapeTable(TableName, int numSplits);
> {noformat}
> or from the shell:
> {noformat}
> > reshape TABLE, numSplits
> {noformat}
> Better still would be to have a maintenance process, similar to the existing
> Balancer that runs AssignmentManager on an interval, to run the above
> "reshape" operation on an interval. That way, the cluster will automatically
> self-correct toward a desirable state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723856#comment-14723856 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-1.2 #145 (See [https://builds.apache.org/job/HBase-1.2/145/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev c331685c95e1aba8c1aad538f4acdf4fe6e36352) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[
https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723857#comment-14723857
]
Hadoop QA commented on HBASE-14229:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753316/HBASE-14229-branch-1-v2.patch
against branch-1 branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753316
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
3818 checkstyle errors (more than the master's current 3816 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.TestRemoveRegionMetrics.testMoveRegion(TestRemoveRegionMetrics.java:118)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15355//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15355//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15355//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15355//console
This message is automatically generated.
> Flushing canceled by coprocessor still leads to memstoreSize set down
> -
>
> Key: HBASE-14229
> URL: https://issues.apache.org/jira/browse/HBASE-14229
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1
>Reporter: Yerui Sun
> Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch,
> HBASE-14229-branch-1-v2.patch, HBASE-14229-branch-1.patch
>
>
> A Coprocessor override "public InternalScanner preFlush(final Store store,
> final InternalScanner scanner)" and return NULL when calling this method,
> will cancel flush request, leaving snapshot un-flushed, and no new storefile
> created. But the HRegion.internalFlushCache still set down memstoreSize to 0
> by totalFlushableSize.
> If there's no write requests anymore, the memstoreSize will remaining as 0,
> and no more flush quests will be processed because of the checking of
> memstoreSize.get() <=0 at the beginning of internalFlushCache.
> This issue may not cause data loss, but it will confuse coprocessor users. If
> we argree with this, I'll apply a patch later.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14346) Typo in FamilyFilter
Joshua Batson created HBASE-14346: - Summary: Typo in FamilyFilter Key: HBASE-14346 URL: https://issues.apache.org/jira/browse/HBASE-14346 Project: HBase Issue Type: Bug Components: documentation Reporter: Joshua Batson Priority: Trivial I think there's a typo. "qualifier name" should read "column family name" Family Filter This filter takes a compare operator and a comparator. It compares each qualifier name with the comparator using the compare operator and if the comparison returns true, it returns all the key-values in that column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[ https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14317: -- Attachment: append-only-test.patch here's the test I ran. when running that no IOE is thrown. > Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL > - > > Key: HBASE-14317 > URL: https://issues.apache.org/jira/browse/HBASE-14317 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.1 >Reporter: stack >Priority: Critical > Attachments: 14317.test.txt, HBASE-14317.patch, [Java] RS stuck on > WAL sync to a dead DN - Pastebin.com.html, append-only-test.patch, raw.php, > san_dump.txt, subset.of.rs.log > > > hbase-1.1.1 and hadoop-2.7.1 > We try to roll logs because can't append (See HDFS-8960) but we get stuck. > See attached thread dump and associated log. What is interesting is that > syncers are waiting to take syncs to run and at same time we want to flush so > we are waiting on a safe point but there seems to be nothing in our ring > buffer; did we go to roll log and not add safe point sync to clear out > ringbuffer? > Needs a bit of study. Try to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14347) Add a switch to DynamicClassLoader to be disabled and make that the default
Esteban Gutierrez created HBASE-14347: - Summary: Add a switch to DynamicClassLoader to be disabled and make that the default Key: HBASE-14347 URL: https://issues.apache.org/jira/browse/HBASE-14347 Project: HBase Issue Type: Bug Components: Client, defaults, regionserver Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Since HBASE-1936 we have the option to load jars dynamically by default from HDFS or the local filesystem, however hbase.dynamic.jars.dir points to a directory that could be world writable it potentially opens a security problem in both the client side and the RS. We should consider to have a switch to enable or disable this option and it should be off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723922#comment-14723922 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-1.3 #142 (See [https://builds.apache.org/job/HBase-1.3/142/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev f30d3dc20c03523b6cf452f70190f1a55dd74978) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14347) Add a switch to DynamicClassLoader to disable it and make that the default
[ https://issues.apache.org/jira/browse/HBASE-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-14347: -- Summary: Add a switch to DynamicClassLoader to disable it and make that the default (was: Add a switch to DynamicClassLoader to be disabled and make that the default) > Add a switch to DynamicClassLoader to disable it and make that the default > -- > > Key: HBASE-14347 > URL: https://issues.apache.org/jira/browse/HBASE-14347 > Project: HBase > Issue Type: Bug > Components: Client, defaults, regionserver >Affects Versions: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > > Since HBASE-1936 we have the option to load jars dynamically by default from > HDFS or the local filesystem, however hbase.dynamic.jars.dir points to a > directory that could be world writable it potentially opens a security > problem in both the client side and the RS. We should consider to have a > switch to enable or disable this option and it should be off by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723937#comment-14723937 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-TRUNK #6767 (See [https://builds.apache.org/job/HBase-TRUNK/6767/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev 498c1845ab7b01710955153c27501fdc7492849d) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14314: --- Attachment: 14314-v2.txt > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt, 14314-v2.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723975#comment-14723975 ] Ted Yu commented on HBASE-14229: Ran patch through test suite in branch-1.1 which passed. > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Yerui Sun > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch, > HBASE-14229-branch-1-v2.patch, HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723976#comment-14723976 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-0.98 #1106 (See [https://builds.apache.org/job/HBase-0.98/1106/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev 437b58f3dcf1f058a54e1abea8be1719b76170fb) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14229) Flushing canceled by coprocessor still leads to memstoreSize set down
[ https://issues.apache.org/jira/browse/HBASE-14229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723978#comment-14723978 ] Ted Yu commented on HBASE-14229: [~sunyerui]: Mind attaching patch for master branch ? > Flushing canceled by coprocessor still leads to memstoreSize set down > - > > Key: HBASE-14229 > URL: https://issues.apache.org/jira/browse/HBASE-14229 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.98.13, 1.0.2, 1.2.0, 1.1.1 >Reporter: Yerui Sun > Attachments: HBASE-14229-0.98.patch, HBASE-14229-branch-1-v2.patch, > HBASE-14229-branch-1-v2.patch, HBASE-14229-branch-1.patch > > > A Coprocessor override "public InternalScanner preFlush(final Store store, > final InternalScanner scanner)" and return NULL when calling this method, > will cancel flush request, leaving snapshot un-flushed, and no new storefile > created. But the HRegion.internalFlushCache still set down memstoreSize to 0 > by totalFlushableSize. > If there's no write requests anymore, the memstoreSize will remaining as 0, > and no more flush quests will be processed because of the checking of > memstoreSize.get() <=0 at the beginning of internalFlushCache. > This issue may not cause data loss, but it will confuse coprocessor users. If > we argree with this, I'll apply a patch later. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14346) Typo in FamilyFilter
[ https://issues.apache.org/jira/browse/HBASE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reassigned HBASE-14346: Assignee: Lars Francke > Typo in FamilyFilter > > > Key: HBASE-14346 > URL: https://issues.apache.org/jira/browse/HBASE-14346 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: Joshua Batson >Assignee: Lars Francke >Priority: Trivial > Original Estimate: 5m > Remaining Estimate: 5m > > I think there's a typo. "qualifier name" should read "column family name" > Family Filter > This filter takes a compare operator and a comparator. It compares each > qualifier name with the comparator using the compare operator and if the > comparison returns true, it returns all the key-values in that column. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[ https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14317: -- Attachment: HBASE-14317-v1.patch Patch that at least fails the put on append erroring and sync not erroring. > Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL > - > > Key: HBASE-14317 > URL: https://issues.apache.org/jira/browse/HBASE-14317 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.1 >Reporter: stack >Priority: Critical > Attachments: 14317.test.txt, HBASE-14317-v1.patch, HBASE-14317.patch, > [Java] RS stuck on WAL sync to a dead DN - Pastebin.com.html, > append-only-test.patch, raw.php, san_dump.txt, subset.of.rs.log > > > hbase-1.1.1 and hadoop-2.7.1 > We try to roll logs because can't append (See HDFS-8960) but we get stuck. > See attached thread dump and associated log. What is interesting is that > syncers are waiting to take syncs to run and at same time we want to flush so > we are waiting on a safe point but there seems to be nothing in our ring > buffer; did we go to roll log and not add safe point sync to clear out > ringbuffer? > Needs a bit of study. Try to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[
https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724013#comment-14724013
]
stack commented on HBASE-14317:
---
This is the new bit in your patch:
{code}
1705 for (int i = 0; i < syncFutures.length; i++) {
1706if (syncFutures[i] != null) {
1707 this.syncFutures[i].done(sequence, e);
1708}
1709 }
{code}
... running through all possible syncfutures though it is as many futures as
there are handlers? You thinking we've just put a syncfuture in but have not
updated the count of futures? Is that possible since it single thread doing
syncfutures addition and count increment?
> Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
> -
>
> Key: HBASE-14317
> URL: https://issues.apache.org/jira/browse/HBASE-14317
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.1
>Reporter: stack
>Priority: Critical
> Attachments: 14317.test.txt, HBASE-14317-v1.patch, HBASE-14317.patch,
> [Java] RS stuck on WAL sync to a dead DN - Pastebin.com.html,
> append-only-test.patch, raw.php, san_dump.txt, subset.of.rs.log
>
>
> hbase-1.1.1 and hadoop-2.7.1
> We try to roll logs because can't append (See HDFS-8960) but we get stuck.
> See attached thread dump and associated log. What is interesting is that
> syncers are waiting to take syncs to run and at same time we want to flush so
> we are waiting on a safe point but there seems to be nothing in our ring
> buffer; did we go to roll log and not add safe point sync to clear out
> ringbuffer?
> Needs a bit of study. Try to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14337) build error on master
[ https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14337: Status: Open (was: Patch Available) > build error on master > - > > Key: HBASE-14337 > URL: https://issues.apache.org/jira/browse/HBASE-14337 > Project: HBase > Issue Type: Bug >Reporter: Gabor Liptak >Assignee: Sean Busbey > Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch, > HBASE-14227_v5.patch, HBASE-14337.1.patch > > > In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console > /home/jenkins/tools/maven/latest/bin/mvn clean package > checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests > -DHBasePatchProcess > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt > 2>&1 > mvn exit code was 0 > mv: cannot stat 'target/checkstyle-result.xml': No such file or directory > Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [ > ... ] > Expected 2...2147483647 file arguments, found 1 > Options: > -withMessagesGenerated XML should contain msgs for external > processing > -output File in which to store combined results > Exception in thread "main" java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml > (No such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:146) > at > edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295) > at > edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235) > at > edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml' > (No such file or directory) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html' > (No such file or directory) > /home/jenkins/tools/maven/latest/bin/mvn clean install -DskipTests > -DHBasePatchProcess -Dhadoop-two.version=2.4.0 > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchJavacWithHadoop-2.4.0.txt > 2>&1 > == > There are compilation errors with Hadoop version 2.4.0. > == > [ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList > at META-INF/LICENSE.vm[line 1619, column 22] > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) > on project hbase-assembly: Error rendering velocity resource. Error invoking > method 'get(java.lang.Integer)' in java.util.ArrayList at > META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: > 0, Size: 0 -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :hbase-assembly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14314: --- Attachment: 14314-v2.txt > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt, 14314-v2.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14314: --- Attachment: (was: 14314-v2.txt) > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt, 14314-v2.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14337) build error on master
[ https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14337: Status: Patch Available (was: Open) > build error on master > - > > Key: HBASE-14337 > URL: https://issues.apache.org/jira/browse/HBASE-14337 > Project: HBase > Issue Type: Bug >Reporter: Gabor Liptak >Assignee: Sean Busbey > Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch, > HBASE-14227_v5.patch, HBASE-14337.1.patch, HBASE-14337.2.patch > > > In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console > /home/jenkins/tools/maven/latest/bin/mvn clean package > checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests > -DHBasePatchProcess > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt > 2>&1 > mvn exit code was 0 > mv: cannot stat 'target/checkstyle-result.xml': No such file or directory > Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [ > ... ] > Expected 2...2147483647 file arguments, found 1 > Options: > -withMessagesGenerated XML should contain msgs for external > processing > -output File in which to store combined results > Exception in thread "main" java.io.FileNotFoundException: > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml > (No such file or directory) > at java.io.FileInputStream.open(Native Method) > at java.io.FileInputStream.(FileInputStream.java:146) > at > edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308) > at > edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295) > at > edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235) > at > edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml' > (No such file or directory) > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html: > ERROR: cannot open > `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html' > (No such file or directory) > /home/jenkins/tools/maven/latest/bin/mvn clean install -DskipTests > -DHBasePatchProcess -Dhadoop-two.version=2.4.0 > > /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchJavacWithHadoop-2.4.0.txt > 2>&1 > == > There are compilation errors with Hadoop version 2.4.0. > == > [ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList > at META-INF/LICENSE.vm[line 1619, column 22] > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) > on project hbase-assembly: Error rendering velocity resource. Error invoking > method 'get(java.lang.Integer)' in java.util.ArrayList at > META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index: > 0, Size: 0 -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :hbase-assembly -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14337) build error on master
[
https://issues.apache.org/jira/browse/HBASE-14337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14337:
Attachment: HBASE-14337.2.patch
-02
* adds option to test-patch to leave the patchprocess directory alone
looks like patches are now justly failing. this patch has the differences on
the branch currently running hte precommit job. needed to allow for the
following workspace layout:
{code}
%WORKSPACE%
|
|-maven_repo
|
|-patchprocess
|
|-hbase
|
\-test-patch branch
{code}
the maven_repo is needed to get around the bad dependency pom that started this
issue. breaking out hbase for the upstream repo and patchprocess for running
was needed to keep RAT just looking at the hbase code. The test-patch branch
was a side effect, but leaving it in place (but pointing to master) will keep
us better-prepared to adopt Yetus once there's a release.
> build error on master
> -
>
> Key: HBASE-14337
> URL: https://issues.apache.org/jira/browse/HBASE-14337
> Project: HBase
> Issue Type: Bug
>Reporter: Gabor Liptak
>Assignee: Sean Busbey
> Attachments: HBASE-14227_v5.patch, HBASE-14227_v5.patch,
> HBASE-14227_v5.patch, HBASE-14337.1.patch, HBASE-14337.2.patch
>
>
> In https://builds.apache.org/job/PreCommit-HBASE-Build/15325//console
> /home/jenkins/tools/maven/latest/bin/mvn clean package
> checkstyle:checkstyle-aggregate findbugs:findbugs -DskipTests
> -DHBasePatchProcess >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkJavacWarnings.txt
> 2>&1
> mvn exit code was 0
> mv: cannot stat 'target/checkstyle-result.xml': No such file or directory
> Usage: edu.umd.cs.findbugs.workflow.UnionResults [options] [
> ... ]
> Expected 2...2147483647 file arguments, found 1
> Options:
> -withMessagesGenerated XML should contain msgs for external
> processing
> -output File in which to store combined results
> Exception in thread "main" java.io.FileNotFoundException:
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml
> (No such file or directory)
> at java.io.FileInputStream.open(Native Method)
> at java.io.FileInputStream.(FileInputStream.java:146)
> at
> edu.umd.cs.findbugs.SortedBugCollection.progessMonitoredInputStream(SortedBugCollection.java:1231)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:308)
> at
> edu.umd.cs.findbugs.SortedBugCollection.readXML(SortedBugCollection.java:295)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.xslt(PrintingBugReporter.java:235)
> at
> edu.umd.cs.findbugs.PrintingBugReporter.main(PrintingBugReporter.java:159)
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml:
> ERROR: cannot open
> `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.xml'
> (No such file or directory)
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html:
> ERROR: cannot open
> `/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/trunkFindbugsWarnings.html'
> (No such file or directory)
> /home/jenkins/tools/maven/latest/bin/mvn clean install -DskipTests
> -DHBasePatchProcess -Dhadoop-two.version=2.4.0 >
> /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/patchprocess/patchJavacWithHadoop-2.4.0.txt
> 2>&1
> ==
> There are compilation errors with Hadoop version 2.4.0.
> ==
> [ERROR] Error invoking method 'get(java.lang.Integer)' in java.util.ArrayList
> at META-INF/LICENSE.vm[line 1619, column 22]
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default)
> on project hbase-assembly: Error rendering velocity resource. Error invoking
> method 'get(java.lang.Integer)' in java.util.ArrayList at
> META-INF/LICENSE.vm[line 1619, column 22]: InvocationTargetException: Index:
> 0, Size: 0 -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :hbase-assembly
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14227: Attachment: HBASE-14227_v5.patch reattaching v5 patch so QA bot can pick it up again. > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14227: Status: Open (was: Patch Available) > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[
https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724040#comment-14724040
]
Ben Lau commented on HBASE-14283:
-
Correct me if I'm wrong but I think I can't do a generalBloomFilter == null
check because there are 3 states not 2: (1) Tried to load filter and it exists
(2) Tried to load filter and it does not exist (3) Have not tried to load
filter yet. If we rely on the generalBloomFilter == null check we can't
distinguish between (2) and (3) which means we would end up trying to reload
the filter unnecessarily.
{quote}
Can information from FileInfo be included in the exception message to
facilitate debugging ?
{quote}
What information from FileInfo should be provided? The point of the message
(maybe it needs to be revised for clarity) is that we found an unexpected
BloomFilter in the HFile-- unexpected because the HFile FileInfo metadata
claims there is no bloom filter (type = NONE). I will clarify the msg a bit.
I'll fix the other issues, thanks.
> Reverse scan doesn’t work with HFile inline index/bloom blocks
> --
>
> Key: HBASE-14283
> URL: https://issues.apache.org/jira/browse/HBASE-14283
> Project: HBase
> Issue Type: Bug
>Reporter: Ben Lau
>Assignee: Ben Lau
> Attachments: HBASE-14283-v2.patch, HBASE-14283.patch,
> hfile-seek-before.patch
>
>
> Reverse scans do not work if an HFile contains inline bloom blocks or leaf
> level index blocks. The reason is because the seekBefore() call calculates
> the previous data block’s size by assuming data blocks are contiguous which
> is not the case in HFile V2 and beyond.
> Attached is a first cut patch (targeting
> bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes:
> (1) a unit test which exposes the bug and demonstrates failures for both
> inline bloom blocks and inline index blocks
> (2) a proposed fix for inline index blocks that does not require a new HFile
> version change, but is only performant for 1 and 2-level indexes and not 3+.
> 3+ requires an HFile format update for optimal performance.
> This patch does not fix the bloom filter blocks bug. But the fix should be
> similar to the case of inline index blocks. The reason I haven’t made the
> change yet is I want to confirm that you guys would be fine with me revising
> the HFile.Reader interface.
> Specifically, these 2 functions (getGeneralBloomFilterMetadata and
> getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the
> HFileReader class doesn’t have a reference to the bloom filters (and hence
> their indices) and only constructs the IO streams and hence has no way to
> know where the bloom blocks are in the HFile. It seems that the HFile.Reader
> bloom method comments state that they “know nothing about how that metadata
> is structured” but I do not know if that is a requirement of the abstraction
> (why?) or just an incidental current property.
> We would like to do 3 things with community approval:
> (1) Update the HFile.Reader interface and implementation to contain and
> return BloomFilters directly rather than unstructured IO streams
> (2) Merge the fixes for index blocks and bloom blocks into open source
> (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’
> field in the block header in the next HFile version, so that seekBefore()
> calls can not only be correct but performant in all cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14227) Fold special cased MOB APIs into existing APIs
[ https://issues.apache.org/jira/browse/HBASE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14227: Status: Patch Available (was: Open) > Fold special cased MOB APIs into existing APIs > -- > > Key: HBASE-14227 > URL: https://issues.apache.org/jira/browse/HBASE-14227 > Project: HBase > Issue Type: Task > Components: mob >Affects Versions: 2.0.0 >Reporter: Andrew Purtell >Assignee: Heng Chen >Priority: Blocker > Fix For: 2.0.0 > > Attachments: HBASE-14227.patch, HBASE-14227_v1.patch, > HBASE-14227_v2.patch, HBASE-14227_v3.patch, HBASE-14227_v4.patch, > HBASE-14227_v5.patch, HBASE-14227_v5.patch > > > There are a number of APIs that came in with MOB that are not new actions for > HBase, simply new actions for a MOB implementation: > - compactMob > - compactMobs > - majorCompactMob > - majorCompactMobs > - getMobCompactionState > And in HBaseAdmin: > - validateMobColumnFamily > Remove these special cases from the Admin API where possible by folding them > into existing APIs. > We definitely don't need one method for a singleton and another for > collections. > Ideally we will not have any APIs named *Mob when finished, whether MOBs are > in use on a table or not should be largely an internal detail. Exposing as > schema option would be fine, this conforms to existing practice for other > features. > Marking critical because I think removing the *Mob special cased APIs should > be a precondition for release of this feature either in 2.0 or as a backport. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12187) Review in source the paper "Simple Testing Can Prevent Most Critical Failures"
[ https://issues.apache.org/jira/browse/HBASE-12187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724046#comment-14724046 ] Sean Busbey commented on HBASE-12187: - we only run upstream releases of error-prone. It'd be great if you could submit a pull request to get the changes into the main repo. I suspect a PR on github (i.e. https://github.com/google/error-prone/pulls ) will get more traction than a post to their google group. > Review in source the paper "Simple Testing Can Prevent Most Critical Failures" > -- > > Key: HBASE-12187 > URL: https://issues.apache.org/jira/browse/HBASE-12187 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Critical > Attachments: HBASE-12187.patch, abortInOvercatch.warnings.txt, > emptyCatch.warnings.txt, todoInCatch.warnings.txt > > > Review the helpful paper > https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf > It describes 'catastrophic failures', especially issues where exceptions are > thrown but not properly handled. Their static analysis tool Aspirator turns > up a bunch of the obvious offenders (Lets add to test-patch.sh alongside > findbugs?). This issue is about going through code base making sub-issues to > root out these and others (Don't we have the test described in figure #6 > already? I thought we did? If we don't, need to add). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14332) Show the table state when we encounter exception while disabling / enabling table
[ https://issues.apache.org/jira/browse/HBASE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14332: Attachment: HBASE-14332_v3.patch reattaching v3 patch so QA bot will take another look. > Show the table state when we encounter exception while disabling / enabling > table > - > > Key: HBASE-14332 > URL: https://issues.apache.org/jira/browse/HBASE-14332 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Nick.han >Assignee: Nick.han >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-14332.patch, HBASE-14332_v2.patch, > HBASE-14332_v3.patch, HBASE-14332_v3.patch > > > This patch is a advice for good user experience > reason: > When we disable a table and the table is not enabled,we receive a > exception,but the exception is too brief,some time we want to know what > state is the table in,so that we can know why the table can't be disable.For > example,I once encountered a problem the table is neither disable nor enable > when my region server crash down,if we give the table state,I will find the > problem more quickly . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14342) Recursive call in RegionMergeTransactionImpl.getJournal()
[
https://issues.apache.org/jira/browse/HBASE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14342:
Attachment: HBASE-14342-branch-1.patch
reattaching patch so QA bot will take another look.
> Recursive call in RegionMergeTransactionImpl.getJournal()
> -
>
> Key: HBASE-14342
> URL: https://issues.apache.org/jira/browse/HBASE-14342
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.1
>Reporter: Lars George
>Assignee: Lars Francke
> Labels: beginner
> Fix For: 1.2.0, 1.1.2
>
> Attachments: HBASE-14342-branch-1.patch, HBASE-14342-branch-1.patch
>
>
> HBASE-12975 in its branch-1 patch
> (https://issues.apache.org/jira/secure/attachment/12708578/HBASE-12975-branch-1.patch)
> introduced a recursive call for {{getJournal()}}. Needs to return just the
> {{journal}} variable like master patch does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14338) License notification misspells 'Asciidoctor'
[
https://issues.apache.org/jira/browse/HBASE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey updated HBASE-14338:
Attachment: HBASE-14338.patch
reattaching patch so QA bot will take another look.
> License notification misspells 'Asciidoctor'
>
>
> Key: HBASE-14338
> URL: https://issues.apache.org/jira/browse/HBASE-14338
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0, 1.2.0, 1.1.2, 1.3.0
>Reporter: Sean Busbey
>Assignee: Lars Francke
>Priority: Minor
> Attachments: HBASE-14338.patch, HBASE-14338.patch
>
>
> our License file contains 'asciidoctor' but with three "i"
> {quote}
> This project bundles a derivative of portions of the 'Asciiidoctor' project
> under the terms of the MIT license.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14158) Add documentation for Initial Release for HBase-Spark Module integration
[ https://issues.apache.org/jira/browse/HBASE-14158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14158: Attachment: HBASE-14158.5.patch reattaching v5 patch so QA bot will take another look. > Add documentation for Initial Release for HBase-Spark Module integration > - > > Key: HBASE-14158 > URL: https://issues.apache.org/jira/browse/HBASE-14158 > Project: HBase > Issue Type: Improvement > Components: documentation, spark >Reporter: Ted Malaska >Assignee: Ted Malaska > Fix For: 2.0.0 > > Attachments: HBASE-14158.1.patch, HBASE-14158.2.patch, > HBASE-14158.5.patch, HBASE-14158.5.patch > > > Add documentation for Initial Release for HBase-Spark Module integration -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14181) Add Spark DataFrame DataSource to HBase-Spark Module
[ https://issues.apache.org/jira/browse/HBASE-14181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-14181: Attachment: HBASE-14181.8.patch reattaching patch v8 so QA bot will take another look. > Add Spark DataFrame DataSource to HBase-Spark Module > > > Key: HBASE-14181 > URL: https://issues.apache.org/jira/browse/HBASE-14181 > Project: HBase > Issue Type: New Feature > Components: spark >Reporter: Ted Malaska >Assignee: Ted Malaska >Priority: Minor > Attachments: HBASE-14181.1.patch, HBASE-14181.2.patch, > HBASE-14181.3.patch, HBASE-14181.4.patch, HBASE-14181.5.patch, > HBASE-14181.6.patch, HBASE-14181.7.patch, HBASE-14181.8.patch, > HBASE-14181.8.patch > > > Build a RelationProvider for HBase-Spark Module. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724056#comment-14724056 ] Ted Yu commented on HBASE-14283: Can a singleton of type BloomFilter be created so that we can use it to denote case 2 ? Meaning generalBloomFilter is initially null. When we find no bloom filter exists, assign generalBloomFilter the singleton. Thanks > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724069#comment-14724069 ] Ted Yu commented on HBASE-14283: Putting patch on reviewboard would make review easier. There're some typo's in current patch. > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14348) Update download mirror link
Andrew Purtell created HBASE-14348: -- Summary: Update download mirror link Key: HBASE-14348 URL: https://issues.apache.org/jira/browse/HBASE-14348 Project: HBase Issue Type: Task Components: documentation, site Reporter: Andrew Purtell Fix For: 2.0.0 Where we refer to www.apache.org/dyn/closer.cgi, we need to refer to www.apache.org/dyn/closer.lua instead . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[
https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724103#comment-14724103
]
Hadoop QA commented on HBASE-14317:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753362/append-only-test.patch
against master branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753362
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15358//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15358//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15358//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15358//console
This message is automatically generated.
> Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
> -
>
> Key: HBASE-14317
> URL: https://issues.apache.org/jira/browse/HBASE-14317
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.1
>Reporter: stack
>Priority: Critical
> Attachments: 14317.test.txt, HBASE-14317-v1.patch, HBASE-14317.patch,
> [Java] RS stuck on WAL sync to a dead DN - Pastebin.com.html,
> append-only-test.patch, raw.php, san_dump.txt, subset.of.rs.log
>
>
> hbase-1.1.1 and hadoop-2.7.1
> We try to roll logs because can't append (See HDFS-8960) but we get stuck.
> See attached thread dump and associated log. What is interesting is that
> syncers are waiting to take syncs to run and at same time we want to flush so
> we are waiting on a safe point but there seems to be nothing in our ring
> buffer; did we go to roll log and not add safe point sync to clear out
> ringbuffer?
> Needs a bit of study. Try to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724104#comment-14724104 ] Ben Lau commented on HBASE-14283: - The other parts of the code consider BloomFilter to be a nullable type, not just here. I don't know if it makes sense to change that in this patch. It is a bit overkill to use a null object here and seems to increase complexity more than it eliminates currently (new class to eliminate a load flag), since unlike in some common use cases of having a null object, we can't avoid checking for the null object here. > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724107#comment-14724107 ] Ben Lau commented on HBASE-14283: - Alright I'll put it on reviewboard, thanks. > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724112#comment-14724112 ] Ted Yu commented on HBASE-14283: Alright, you can keep the checking as it is now. > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[ https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14317: -- Attachment: HBASE-14317-v2.patch This patch is a little more thorough about the sync futures. We need to make sure that all of the syncfutures are marked as failed. Other wise it's possible that others could sneak in. > Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL > - > > Key: HBASE-14317 > URL: https://issues.apache.org/jira/browse/HBASE-14317 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.1 >Reporter: stack >Priority: Critical > Attachments: 14317.test.txt, HBASE-14317-v1.patch, > HBASE-14317-v2.patch, HBASE-14317.patch, [Java] RS stuck on WAL sync to a > dead DN - Pastebin.com.html, append-only-test.patch, raw.php, san_dump.txt, > subset.of.rs.log > > > hbase-1.1.1 and hadoop-2.7.1 > We try to roll logs because can't append (See HDFS-8960) but we get stuck. > See attached thread dump and associated log. What is interesting is that > syncers are waiting to take syncs to run and at same time we want to flush so > we are waiting on a safe point but there seems to be nothing in our ring > buffer; did we go to roll log and not add safe point sync to clear out > ringbuffer? > Needs a bit of study. Try to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[ https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14317: -- Attachment: HBASE-14317-v3.patch correct patch > Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL > - > > Key: HBASE-14317 > URL: https://issues.apache.org/jira/browse/HBASE-14317 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.1 >Reporter: stack >Priority: Critical > Attachments: 14317.test.txt, HBASE-14317-v1.patch, > HBASE-14317-v2.patch, HBASE-14317-v3.patch, HBASE-14317.patch, [Java] RS > stuck on WAL sync to a dead DN - Pastebin.com.html, append-only-test.patch, > raw.php, san_dump.txt, subset.of.rs.log > > > hbase-1.1.1 and hadoop-2.7.1 > We try to roll logs because can't append (See HDFS-8960) but we get stuck. > See attached thread dump and associated log. What is interesting is that > syncers are waiting to take syncs to run and at same time we want to flush so > we are waiting on a safe point but there seems to be nothing in our ring > buffer; did we go to roll log and not add safe point sync to clear out > ringbuffer? > Needs a bit of study. Try to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[ https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14317: -- Attachment: HBASE-14317-v4.patch More extreme test. Also I found that I need to reset the sync futures. > Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL > - > > Key: HBASE-14317 > URL: https://issues.apache.org/jira/browse/HBASE-14317 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.1.1 >Reporter: stack >Priority: Critical > Attachments: 14317.test.txt, HBASE-14317-v1.patch, > HBASE-14317-v2.patch, HBASE-14317-v3.patch, HBASE-14317-v4.patch, > HBASE-14317.patch, [Java] RS stuck on WAL sync to a dead DN - > Pastebin.com.html, append-only-test.patch, raw.php, san_dump.txt, > subset.of.rs.log > > > hbase-1.1.1 and hadoop-2.7.1 > We try to roll logs because can't append (See HDFS-8960) but we get stuck. > See attached thread dump and associated log. What is interesting is that > syncers are waiting to take syncs to run and at same time we want to flush so > we are waiting on a safe point but there seems to be nothing in our ring > buffer; did we go to roll log and not add safe point sync to clear out > ringbuffer? > Needs a bit of study. Try to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14325) Add snapshotinfo command to hbase script
[ https://issues.apache.org/jira/browse/HBASE-14325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724140#comment-14724140 ] Hudson commented on HBASE-14325: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1059 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1059/]) HBASE-14325 Add snapshotinfo command to hbase script. (matteo.bertozzi: rev 437b58f3dcf1f058a54e1abea8be1719b76170fb) * bin/hbase > Add snapshotinfo command to hbase script > > > Key: HBASE-14325 > URL: https://issues.apache.org/jira/browse/HBASE-14325 > Project: HBase > Issue Type: Improvement > Components: scripts >Affects Versions: 2.0.0, 1.2.0 >Reporter: Samir Ahmic >Assignee: Samir Ahmic >Priority: Minor > Fix For: 2.0.0, 1.2.0, 1.1.2, 0.98.15, 1.0.3 > > Attachments: HBASE-14325.patch > > > Since we already have commands like hbck, hfile, wal etc. that are used for > getting various types of information about HBase components it make sense to > me to add SnapshotInfo tool to collection. > If nobody objects i would add patch for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724142#comment-14724142
]
Ted Yu commented on HBASE-14331:
Can you put the three new Java classes in a single patch with proper Apache
License header ?
Looks like SemaphoreBasedLinkedBlockingQueue is better than
SemaphoreBasedPriorityBlockingQueue when waked read threads - tasks are high.
What's your interpretation ?
Thanks
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14290) Spin up less threads in tests
[ https://issues.apache.org/jira/browse/HBASE-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-14290: -- Fix Version/s: 1.1.3 1.0.3 1.3.0 1.2.0 2.0.0 > Spin up less threads in tests > - > > Key: HBASE-14290 > URL: https://issues.apache.org/jira/browse/HBASE-14290 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: lessthreads.txt, lessthreadsv2.txt, lessthreadsv2.txt, > lessthreadsv2.txt, lessthreadsv3.1.0.txt, lessthreadsv3.txt > > > TestDistributedLogReplay runs with > 450 threads which seems a little OTT (I > get OOME, cannot create native thread when I run test suite on local MBP). > This issue is about trying to tamper down the tests that are using so many > threads they can OOME because can't create native thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14290) Spin up less threads in tests
[ https://issues.apache.org/jira/browse/HBASE-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724152#comment-14724152 ] Enis Soztutar commented on HBASE-14290: --- [~stack] resolve this or keep it open for more work? > Spin up less threads in tests > - > > Key: HBASE-14290 > URL: https://issues.apache.org/jira/browse/HBASE-14290 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: lessthreads.txt, lessthreadsv2.txt, lessthreadsv2.txt, > lessthreadsv2.txt, lessthreadsv3.1.0.txt, lessthreadsv3.txt > > > TestDistributedLogReplay runs with > 450 threads which seems a little OTT (I > get OOME, cannot create native thread when I run test suite on local MBP). > This issue is about trying to tamper down the tests that are using so many > threads they can OOME because can't create native thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14331) a single callQueue related improvements
[
https://issues.apache.org/jira/browse/HBASE-14331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724156#comment-14724156
]
Ted Yu commented on HBASE-14331:
For the two BlockingQueue implementations, please add unit tests.
> a single callQueue related improvements
> ---
>
> Key: HBASE-14331
> URL: https://issues.apache.org/jira/browse/HBASE-14331
> Project: HBase
> Issue Type: Improvement
> Components: IPC/RPC, Performance
>Reporter: Hiroshi Ikeda
>Priority: Minor
> Attachments: BlockingQueuesPerformanceTestApp-output.pdf,
> BlockingQueuesPerformanceTestApp-output.txt,
> BlockingQueuesPerformanceTestApp.java, CallQueuePerformanceTestApp.java,
> SemaphoreBasedBlockingQueue.java, SemaphoreBasedLinkedBlockingQueue.java,
> SemaphoreBasedPriorityBlockingQueue.java
>
>
> {{LinkedBlockingQueue}} well separates locks between the {{take}} method and
> the {{put}} method, but not between takers, and not between putters. These
> methods are implemented to take locks at the almost beginning of their logic.
> HBASE-11355 introduces multiple call-queues to reduce such possible
> congestion, but I doubt that it is required to stick to {{BlockingQueue}}.
> There are the other shortcomings of using {{BlockingQueue}}. When using
> multiple queues, since {{BlockingQueue}} blocks threads it is required to
> prepare enough threads for each queue. It is possible that there is a queue
> starving for threads while there is another queue where threads are idle.
> Even if you can tune parameters to avoid such situations, the tuning is not
> so trivial.
> I suggest using a single {{ConcurrentLinkedQueue}} with {{Semaphore}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14290) Spin up less threads in tests
[ https://issues.apache.org/jira/browse/HBASE-14290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-14290: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) There is a load more work to be done in here but enough for now [~enis] Resolving. > Spin up less threads in tests > - > > Key: HBASE-14290 > URL: https://issues.apache.org/jira/browse/HBASE-14290 > Project: HBase > Issue Type: Task > Components: test >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3 > > Attachments: lessthreads.txt, lessthreadsv2.txt, lessthreadsv2.txt, > lessthreadsv2.txt, lessthreadsv3.1.0.txt, lessthreadsv3.txt > > > TestDistributedLogReplay runs with > 450 threads which seems a little OTT (I > get OOME, cannot create native thread when I run test suite on local MBP). > This issue is about trying to tamper down the tests that are using so many > threads they can OOME because can't create native thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[
https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724304#comment-14724304
]
Hadoop QA commented on HBASE-14317:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753396/HBASE-14317-v4.patch
against master branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753396
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:red}-1 findbugs{color}. The patch appears to cause Findbugs
(version 2.0.3) to fail.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15369//testReport/
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15369//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15369//console
This message is automatically generated.
> Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
> -
>
> Key: HBASE-14317
> URL: https://issues.apache.org/jira/browse/HBASE-14317
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.1
>Reporter: stack
>Priority: Critical
> Attachments: 14317.test.txt, HBASE-14317-v1.patch,
> HBASE-14317-v2.patch, HBASE-14317-v3.patch, HBASE-14317-v4.patch,
> HBASE-14317.patch, [Java] RS stuck on WAL sync to a dead DN -
> Pastebin.com.html, append-only-test.patch, raw.php, san_dump.txt,
> subset.of.rs.log
>
>
> hbase-1.1.1 and hadoop-2.7.1
> We try to roll logs because can't append (See HDFS-8960) but we get stuck.
> See attached thread dump and associated log. What is interesting is that
> syncers are waiting to take syncs to run and at same time we want to flush so
> we are waiting on a safe point but there seems to be nothing in our ring
> buffer; did we go to roll log and not add safe point sync to clear out
> ringbuffer?
> Needs a bit of study. Try to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14314) Metrics for block cache should take region replicas into account
[
https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724308#comment-14724308
]
Hadoop QA commented on HBASE-14314:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753376/14314-v2.txt
against master branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753376
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1851 checkstyle errors (more than the master's current 1846 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ .addCounter(Interns.info(BLOCK_CACHE_PRIMARY_HIT_COUNT,
BLOCK_CACHE_PRIMARY_HIT_COUNT_DESC),
+ .addCounter(Interns.info(BLOCK_CACHE_PRIMARY_MISS_COUNT,
BLOCK_COUNT_PRIMARY_MISS_COUNT_DESC),
+ .addCounter(Interns.info(L1_BLOCK_CACHE_EVICTION_COUNT,
L1_BLOCK_CACHE_EVICTION_COUNT_DESC),
+ .addCounter(Interns.info(BLOCK_CACHE_PRIMARY_EVICTION_COUNT,
BLOCK_CACHE_PRIMARY_EVICTION_COUNT_DESC),
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15361//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15361//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15361//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15361//console
This message is automatically generated.
> Metrics for block cache should take region replicas into account
>
>
> Key: HBASE-14314
> URL: https://issues.apache.org/jira/browse/HBASE-14314
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 14314-v1.txt, 14314-v2.txt
>
>
> Currently metrics for block cache are aggregates in the sense that they don't
> distinguish primary from secondary / tertiary replicas.
> This JIRA separates the block cache metrics for primary region replica from
> the aggregate.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-11236) Last flushed sequence id is ignored by ServerManager
[
https://issues.apache.org/jira/browse/HBASE-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724331#comment-14724331
]
Elliott Clark commented on HBASE-11236:
---
I've seen this on 1.2 a decent ammount
> Last flushed sequence id is ignored by ServerManager
>
>
> Key: HBASE-11236
> URL: https://issues.apache.org/jira/browse/HBASE-11236
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>
> I got lots of error messages like this:
> {quote}
> 2014-05-22 08:58:59,793 DEBUG [RpcServer.handler=1,port=20020]
> master.ServerManager: RegionServer
> a2428.halxg.cloudera.com,20020,1400742071109 indicates a last flushed
> sequence id (numberOfStores=9, numberOfStorefiles=2,
> storefileUncompressedSizeMB=517, storefileSizeMB=517,
> compressionRatio=1., memstoreSizeMB=0, storefileIndexSizeMB=0,
> readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=34,
> totalStaticIndexSizeKB=381, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
> currentCompactedKVs=0, compactionProgressPct=NaN) that is less than the
> previous last flushed sequence id (605446) for region
> IntegrationTestBigLinkedList,
> �A��*t�^FU�2��0,1400740489477.a44d3e309b5a7e29355f6faa0d3a4095. Ignoring.
> {quote}
> RegionLoad.toString doesn't print out the last flushed sequence id passed in.
> Why is it less than the previous one?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (HBASE-11236) Last flushed sequence id is ignored by ServerManager
[
https://issues.apache.org/jira/browse/HBASE-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elliott Clark reopened HBASE-11236:
---
> Last flushed sequence id is ignored by ServerManager
>
>
> Key: HBASE-11236
> URL: https://issues.apache.org/jira/browse/HBASE-11236
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>
> I got lots of error messages like this:
> {quote}
> 2014-05-22 08:58:59,793 DEBUG [RpcServer.handler=1,port=20020]
> master.ServerManager: RegionServer
> a2428.halxg.cloudera.com,20020,1400742071109 indicates a last flushed
> sequence id (numberOfStores=9, numberOfStorefiles=2,
> storefileUncompressedSizeMB=517, storefileSizeMB=517,
> compressionRatio=1., memstoreSizeMB=0, storefileIndexSizeMB=0,
> readRequestsCount=0, writeRequestsCount=0, rootIndexSizeKB=34,
> totalStaticIndexSizeKB=381, totalStaticBloomSizeKB=0, totalCompactingKVs=0,
> currentCompactedKVs=0, compactionProgressPct=NaN) that is less than the
> previous last flushed sequence id (605446) for region
> IntegrationTestBigLinkedList,
> �A��*t�^FU�2��0,1400740489477.a44d3e309b5a7e29355f6faa0d3a4095. Ignoring.
> {quote}
> RegionLoad.toString doesn't print out the last flushed sequence id passed in.
> Why is it less than the previous one?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14317) Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
[
https://issues.apache.org/jira/browse/HBASE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724342#comment-14724342
]
Hadoop QA commented on HBASE-14317:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12753371/HBASE-14317-v1.patch
against master branch at commit 498c1845ab7b01710955153c27501fdc7492849d.
ATTACHMENT ID: 12753371
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0 2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestProcessBasedCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/15360//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/15360//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/15360//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/15360//console
This message is automatically generated.
> Stuck FSHLog: bad disk (HDFS-8960) and can't roll WAL
> -
>
> Key: HBASE-14317
> URL: https://issues.apache.org/jira/browse/HBASE-14317
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.0, 1.1.1
>Reporter: stack
>Priority: Critical
> Attachments: 14317.test.txt, HBASE-14317-v1.patch,
> HBASE-14317-v2.patch, HBASE-14317-v3.patch, HBASE-14317-v4.patch,
> HBASE-14317.patch, [Java] RS stuck on WAL sync to a dead DN -
> Pastebin.com.html, append-only-test.patch, raw.php, san_dump.txt,
> subset.of.rs.log
>
>
> hbase-1.1.1 and hadoop-2.7.1
> We try to roll logs because can't append (See HDFS-8960) but we get stuck.
> See attached thread dump and associated log. What is interesting is that
> syncers are waiting to take syncs to run and at same time we want to flush so
> we are waiting on a safe point but there seems to be nothing in our ring
> buffer; did we go to roll log and not add safe point sync to clear out
> ringbuffer?
> Needs a bit of study. Try to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14283) Reverse scan doesn’t work with HFile inline index/bloom blocks
[ https://issues.apache.org/jira/browse/HBASE-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14724357#comment-14724357 ] Ben Lau commented on HBASE-14283: - Reviewboard link with updated version of patch (v3): https://reviews.apache.org/r/37971/ > Reverse scan doesn’t work with HFile inline index/bloom blocks > -- > > Key: HBASE-14283 > URL: https://issues.apache.org/jira/browse/HBASE-14283 > Project: HBase > Issue Type: Bug >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-14283-v2.patch, HBASE-14283.patch, > hfile-seek-before.patch > > > Reverse scans do not work if an HFile contains inline bloom blocks or leaf > level index blocks. The reason is because the seekBefore() call calculates > the previous data block’s size by assuming data blocks are contiguous which > is not the case in HFile V2 and beyond. > Attached is a first cut patch (targeting > bcef28eefaf192b0ad48c8011f98b8e944340da5 on trunk) which includes: > (1) a unit test which exposes the bug and demonstrates failures for both > inline bloom blocks and inline index blocks > (2) a proposed fix for inline index blocks that does not require a new HFile > version change, but is only performant for 1 and 2-level indexes and not 3+. > 3+ requires an HFile format update for optimal performance. > This patch does not fix the bloom filter blocks bug. But the fix should be > similar to the case of inline index blocks. The reason I haven’t made the > change yet is I want to confirm that you guys would be fine with me revising > the HFile.Reader interface. > Specifically, these 2 functions (getGeneralBloomFilterMetadata and > getDeleteBloomFilterMetadata) need to return the BloomFilter. Right now the > HFileReader class doesn’t have a reference to the bloom filters (and hence > their indices) and only constructs the IO streams and hence has no way to > know where the bloom blocks are in the HFile. It seems that the HFile.Reader > bloom method comments state that they “know nothing about how that metadata > is structured” but I do not know if that is a requirement of the abstraction > (why?) or just an incidental current property. > We would like to do 3 things with community approval: > (1) Update the HFile.Reader interface and implementation to contain and > return BloomFilters directly rather than unstructured IO streams > (2) Merge the fixes for index blocks and bloom blocks into open source > (3) Create a new Jira ticket for open source HBase to add a ‘prevBlockSize’ > field in the block header in the next HFile version, so that seekBefore() > calls can not only be correct but performant in all cases. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14314) Metrics for block cache should take region replicas into account
[ https://issues.apache.org/jira/browse/HBASE-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14314: --- Attachment: 14314-v3.txt Patch v3 fixes checkstyle warnings > Metrics for block cache should take region replicas into account > > > Key: HBASE-14314 > URL: https://issues.apache.org/jira/browse/HBASE-14314 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 14314-v1.txt, 14314-v2.txt, 14314-v3.txt > > > Currently metrics for block cache are aggregates in the sense that they don't > distinguish primary from secondary / tertiary replicas. > This JIRA separates the block cache metrics for primary region replica from > the aggregate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-14123: -- Attachment: HBASE-14123-v3.patch Patch v3. Includes HBASE-14124. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-14123-v1.patch, HBASE-14123-v2.patch, > HBASE-14123-v3.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-14124) Failed backup is not handled properly in incremental mode
[ https://issues.apache.org/jira/browse/HBASE-14124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14124. --- Resolution: Fixed As apart of patch v3 (HBASE-14123). > Failed backup is not handled properly in incremental mode > - > > Key: HBASE-14124 > URL: https://issues.apache.org/jira/browse/HBASE-14124 > Project: HBase > Issue Type: Bug >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > BackupHandler failedBackup method does not clean failed incremental backup > artefacts on HDFS (and in HBase). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14343) Fix debug message in SimpleRegionNormalizer for small regions
[
https://issues.apache.org/jira/browse/HBASE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lars Francke updated HBASE-14343:
-
Attachment: HBASE-14343-extended.patch
HBASE-14343.patch
Attached are two patches.
One just fixes the bugs mentioned here. The second one cleans up a few other
minor warnings as well. If you'd go for the second one we could rename the
issue (e.g. "Fix code style issues, inconsistencies and a potential NPE in
SimpleRegionNormalizer")
> Fix debug message in SimpleRegionNormalizer for small regions
> -
>
> Key: HBASE-14343
> URL: https://issues.apache.org/jira/browse/HBASE-14343
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 1.1.1
>Reporter: Lars George
>Assignee: Lars Francke
>Priority: Trivial
> Labels: beginner
> Attachments: HBASE-14343-extended.patch, HBASE-14343.patch
>
>
> The {{SimpleRegionNormalizer}} has this:
> {code}
> if ((smallestRegion.getSecond() +
> smallestNeighborOfSmallestRegion.getSecond()
> < avgRegionSize)) {
> LOG.debug("Table " + table + ", smallest region size: " +
> smallestRegion.getSecond()
> + " and its smallest neighbor size: " +
> smallestNeighborOfSmallestRegion.getSecond()
> + ", less than half the avg size, merging them");
> {code}
> It does *not* check for "less than half the avg size" but only "less than the
> avg size", that is, drop the "half". Fix message.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HBASE-14348) Update download mirror link
[ https://issues.apache.org/jira/browse/HBASE-14348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Francke reassigned HBASE-14348: Assignee: Lars Francke > Update download mirror link > --- > > Key: HBASE-14348 > URL: https://issues.apache.org/jira/browse/HBASE-14348 > Project: HBase > Issue Type: Task > Components: documentation, site >Reporter: Andrew Purtell >Assignee: Lars Francke > Fix For: 2.0.0 > > > Where we refer to www.apache.org/dyn/closer.cgi, we need to refer to > www.apache.org/dyn/closer.lua instead . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
