[jira] [Commented] (HBASE-13838) Fix shared TaskStatusTmpl.jamon issues (coloring, content, etc.)
[
https://issues.apache.org/jira/browse/HBASE-13838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631243#comment-14631243
]
Matt Warhaftig commented on HBASE-13838:
Note that after this change the message content of 'Client Operation' tasks
will be shown in the JSON (because the {{ClientProtos}} toString method is
used). If this is a security concern let me know and will generate a sanitized
JSON body.
Fix shared TaskStatusTmpl.jamon issues (coloring, content, etc.)
Key: HBASE-13838
URL: https://issues.apache.org/jira/browse/HBASE-13838
Project: HBase
Issue Type: Bug
Components: UI
Affects Versions: 1.1.0
Reporter: Lars George
Assignee: Matt Warhaftig
Labels: beginner
Fix For: 2.0.0, 1.3.0
Attachments: hbase-13838-v1.patch
There are a few issues with the shared TaskStatusTmpl:
- Client operations tab is always empty
For Master this is expected, but for RegionServers there is never anything
listed either. Fix for RS status page (probably caused by params not
containing Operation subclass anymore, but some PB generated classes?)
- Hide “Client Operations” tab for master UI
Since operations are RS only. Or we fix this and make other calls show here.
- The alert-error for aborted tasks is not set in CSS at all
When a task was aborted it should be amber or red, but the assigned style is
not in any of the linked stylesheets (abort-error). Add.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Open (was: Patch Available) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Attachment: HBASE-12295_16.patch Updated patch addressing comments from RB. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Patch Available (was: Open) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631385#comment-14631385 ] stack commented on HBASE-13408: --- I left some comments on rb but should have read the design first. I read the design and had these few questions (design is nicely written): FYI, Flush-by-column-family is on by default in 1.2 HBase. Will you need to do anything to accommodate this? You say In highchurn workloads, compacting the memstore can help maintain the data in memory, and thereby speed up data retrieval The pipeline entries are still skiplists sets? What is the compacted representation? Is it still a skiplist? Skip list is slow especially as we get large. Was wondering if you saw any speedup if only because you have many small skip lists rather than one big one? Therefore, we suggest applying this optimization only to inmemory column families. In testing you find that the overhead slows us down? I asked in rb what threading model was? Is there a new thread per Store memstore? Is the new forceflushsize a new config? I wasn't following why we need it? If size of current Set + pipeline is above max size, flush? I wasn't clear on why the need of 2.5. Is memstoresegment our old snapshot? it has other facility beyond old snapshot? Thanks HBase In-Memory Memstore Compaction --- Key: HBASE-13408 URL: https://issues.apache.org/jira/browse/HBASE-13408 Project: HBase Issue Type: New Feature Reporter: Eshcar Hillel Attachments: HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, InMemoryMemstoreCompactionEvaluationResults.pdf A store unit holds a column family in a region, where the memstore is its in-memory component. The memstore absorbs all updates to the store; from time to time these updates are flushed to a file on disk, where they are compacted. Unlike disk components, the memstore is not compacted until it is written to the filesystem and optionally to block-cache. This may result in underutilization of the memory due to duplicate entries per row, for example, when hot data is continuously updated. Generally, the faster the data is accumulated in memory, more flushes are triggered, the data sinks to disk more frequently, slowing down retrieval of data, even if very recent. In high-churn workloads, compacting the memstore can help maintain the data in memory, and thereby speed up data retrieval. We suggest a new compacted memstore with the following principles: 1.The data is kept in memory for as long as possible 2.Memstore data is either compacted or in process of being compacted 3.Allow a panic mode, which may interrupt an in-progress compaction and force a flush of part of the memstore. We suggest applying this optimization only to in-memory column families. A design document is attached. This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-14109:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
1.1.2
1.2.0
Status: Resolved (was: Patch Available)
Pushed trivial fix (thanks for review [~mbertozzi]) to branch-1+ Master had
this fix already. Not needed in 1.0
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631369#comment-14631369 ] Ted Malaska commented on HBASE-13992: - [~lhofhansl] * I will add the comments * As for the Result you do get the results but you need to convert it to something that can go into an RDD. * As for the memory needed. Think of it as if you are reading a file from S3 or HDFS. It is the same thing here. * Yes I tested 1.4 with -D and it seemed to work. And yes I'm super excited about this too, because this is only the beginning. I will work through the weekend to get all these reviews into the next patch. Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13788) Shell comands do not support column qualifiers containing colon (:)
[
https://issues.apache.org/jira/browse/HBASE-13788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631373#comment-14631373
]
Pankaj Kumar commented on HBASE-13788:
--
[~stack], May be we can use pattern :to[] for converter specification.
For example,
{noformat}
cf:qualifier:to[toInt] -- Bytes.toInt()
cf:qualifier:to[c(UserDefClass).toInt] -- UserDefClass.toInt()
{noformat}
Please provide your opinion.
Shell comands do not support column qualifiers containing colon (:)
---
Key: HBASE-13788
URL: https://issues.apache.org/jira/browse/HBASE-13788
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.98.0, 0.96.0, 1.0.0, 1.1.0
Reporter: Dave Latham
Assignee: Pankaj Kumar
The shell interprets the colon within the qualifier as a delimiter to a
FORMATTER instead of part of the qualifier itself.
Example from the mailing list:
Hmph, I may have spoken too soon. I know I tested this at one point and
it worked, but now I'm getting different results:
On the new cluster, I created a duplicate test table:
hbase(main):043:0 create 'content3', {NAME = 'x', BLOOMFILTER =
'NONE', REPLICATION_SCOPE = '0', VERSIONS = '3', COMPRESSION =
'NONE', MIN_VERSIONS = '0', TTL = '2147483647', BLOCKSIZE = '65536',
IN_MEMORY = 'false', BLOCKCACHE = 'true'}
Then I pull some data from the imported table:
hbase(main):045:0 scan 'content', {LIMIT=1,
STARTROW='A:9223370612089311807:twtr:57013379'}
ROW COLUMN+CELL
A:9223370612089311807:twtr:570133798827921408
column=x:twitter:username, timestamp=1424775595345, value=BERITA
INFORMASI!
Then put it:
hbase(main):046:0 put
'content3','A:9223370612089311807:twtr:570133798827921408',
'x:twitter:username', 'BERITA INFORMASI!'
But then when I query it, I see that I've lost the column qualifier
:username:
hbase(main):046:0 scan 'content3'
ROW COLUMN+CELL
A:9223370612089311807:twtr:570133798827921408 column=x:twitter,
timestamp=1432745301788, value=BERITA INFORMASI!
Even though I'm missing one of the qualifiers, I can at least filter on
columns in this sample table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631283#comment-14631283 ] Sean Busbey commented on HBASE-13992: - They were on review board Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14116) Change ByteBuff.getXXXStrictlyForward to relative position based reads
[ https://issues.apache.org/jira/browse/HBASE-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-14116: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-11425 Change ByteBuff.getXXXStrictlyForward to relative position based reads -- Key: HBASE-14116 URL: https://issues.apache.org/jira/browse/HBASE-14116 Project: HBase Issue Type: Sub-task Reporter: ramkrishna.s.vasudevan Assignee: Anoop Sam John Fix For: 2.0.0 There is a TODO added in ByteBuff.getXXXStrictlyForward to a positional based read from the current position. This could help in avoiding the initial check that is added in the API to ensure that the passed in index is always greater than the current position. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631337#comment-14631337
]
ramkrishna.s.vasudevan commented on HBASE-14117:
Yes. Directly skipping will not be possible as it is compressedInt. This JIRA
is just to explore if we can do something about it. As you said atleast try
reducing the ops.
Check DBEs where fields are being read from Bytebuffers but unused.
---
Key: HBASE-14117
URL: https://issues.apache.org/jira/browse/HBASE-14117
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
{code}
public Cell getFirstKeyCellInBlock(ByteBuff block) {
block.mark();
block.position(Bytes.SIZEOF_INT);
int keyLength = ByteBuff.readCompressedInt(block);
// TODO : See if we can avoid these reads as the read values are not
getting used
ByteBuff.readCompressedInt(block);
{code}
In DBEs many a places we read the integers just to skip them. This JIRA is to
see if we can avoid this and rather go position based, as per a review
comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631344#comment-14631344 ] Lars Hofhansl commented on HBASE-13992: --- Few nits: * Comments in what's happening in the Scala and JavaHBase*Examples * Might want to have a comment in *HBaseBulkDeleteExample.java explaining that it is probably better to use the built-in BulkDeleteEndpoint (which in turn we have to document better) * Comment explaining in *HBaseDistributedScan as to why this is preferred over using Spark's built-in HadoopRDD with TableInputFormat or the included HBaseRDD (it's obvious, but hey, somebody might look at the classes and wonder why). * I assume there's quite some heap needed to get the RDD resulting from a scan into a (RowKey, List[(columnFamily, columnQualifier, Value) (going by the article here). Can that be avoided if it is a problem? Or in other words, is there an easy to way to get a raw Result[] or ListResult? ... sorry if I'm missing something. Very excited about having this in HBase proper. Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14100) Fix high priority findbugs warnings
[ https://issues.apache.org/jira/browse/HBASE-14100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631287#comment-14631287 ] Hudson commented on HBASE-14100: SUCCESS: Integrated in HBase-1.3-IT #49 (See [https://builds.apache.org/job/HBase-1.3-IT/49/]) HBASE-14100 Fix high priority findbugs warnings (zhangduo: rev 12f6470bf39da30e6b856287d75856d8a45a56a6) * hbase-server/src/main/java/org/apache/hadoop/hbase/master/normalizer/SimpleRegionNormalizer.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/AverageIntervalRateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FixedIntervalRateLimiter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java Fix high priority findbugs warnings --- Key: HBASE-14100 URL: https://issues.apache.org/jira/browse/HBASE-14100 Project: HBase Issue Type: Bug Reporter: Duo Zhang Assignee: Duo Zhang Fix For: 2.0.0, 1.2.0, 1.1.2, 1.3.0 Attachments: HBASE-14100.patch See here: https://builds.apache.org/job/HBase-TRUNK/6654/findbugsResult/HIGH/ We have 6 high priority findbugs warnings. A high priority findbugs warning is usually a bug. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631288#comment-14631288
]
Hudson commented on HBASE-14050:
SUCCESS: Integrated in HBase-1.3-IT #49 (See
[https://builds.apache.org/job/HBase-1.3-IT/49/])
HBASE-14050 NPE in org.apache.hadoop.hbase.ipc.RpcServer.readAndProcess
(apurtell: rev d4a04d62165d4f8ce78d2f98acefb47f513b6b7d)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14116) Change ByteBuff.getXXXStrictlyForward to relative position based reads
ramkrishna.s.vasudevan created HBASE-14116: -- Summary: Change ByteBuff.getXXXStrictlyForward to relative position based reads Key: HBASE-14116 URL: https://issues.apache.org/jira/browse/HBASE-14116 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 2.0.0 There is a TODO added in ByteBuff.getXXXStrictlyForward to a positional based read from the current position. This could help in avoiding the initial check that is added in the API to ensure that the passed in index is always greater than the current position. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14116) Change ByteBuff.getXXXStrictlyForward to relative position based reads
[ https://issues.apache.org/jira/browse/HBASE-14116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John reassigned HBASE-14116: -- Assignee: Anoop Sam John Change ByteBuff.getXXXStrictlyForward to relative position based reads -- Key: HBASE-14116 URL: https://issues.apache.org/jira/browse/HBASE-14116 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Assignee: Anoop Sam John Fix For: 2.0.0 There is a TODO added in ByteBuff.getXXXStrictlyForward to a positional based read from the current position. This could help in avoiding the initial check that is added in the API to ensure that the passed in index is always greater than the current position. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan reassigned HBASE-14117:
--
Assignee: ramkrishna.s.vasudevan
Check DBEs where fields are being read from Bytebuffers but unused.
---
Key: HBASE-14117
URL: https://issues.apache.org/jira/browse/HBASE-14117
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
{code}
public Cell getFirstKeyCellInBlock(ByteBuff block) {
block.mark();
block.position(Bytes.SIZEOF_INT);
int keyLength = ByteBuff.readCompressedInt(block);
// TODO : See if we can avoid these reads as the read values are not
getting used
ByteBuff.readCompressedInt(block);
{code}
In DBEs many a places we read the integers just to skip them. This JIRA is to
see if we can avoid this and rather go position based, as per a review
comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
ramkrishna.s.vasudevan created HBASE-14117:
--
Summary: Check DBEs where fields are being read from Bytebuffers
but unused.
Key: HBASE-14117
URL: https://issues.apache.org/jira/browse/HBASE-14117
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
{code}
public Cell getFirstKeyCellInBlock(ByteBuff block) {
block.mark();
block.position(Bytes.SIZEOF_INT);
int keyLength = ByteBuff.readCompressedInt(block);
// TODO : See if we can avoid these reads as the read values are not
getting used
ByteBuff.readCompressedInt(block);
{code}
In DBEs many a places we read the integers just to skip them. This JIRA is to
see if we can avoid this and rather go position based, as per a review comment
in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631332#comment-14631332
]
Anoop Sam John commented on HBASE-14117:
We read compressed int. That is why we are not able to simply skip. Unless we
read each byte and check we can not know how many bytes this int took to store
in compressed form. May be we can have a variant of method where the read just
read bytes and skips until the compressed int bytes are over but it will NOT
compute the int value. So can save some ops. But direct skip like for
int/long we can not do right (?)
Check DBEs where fields are being read from Bytebuffers but unused.
---
Key: HBASE-14117
URL: https://issues.apache.org/jira/browse/HBASE-14117
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
{code}
public Cell getFirstKeyCellInBlock(ByteBuff block) {
block.mark();
block.position(Bytes.SIZEOF_INT);
int keyLength = ByteBuff.readCompressedInt(block);
// TODO : See if we can avoid these reads as the read values are not
getting used
ByteBuff.readCompressedInt(block);
{code}
In DBEs many a places we read the integers just to skip them. This JIRA is to
see if we can avoid this and rather go position based, as per a review
comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14117) Check DBEs where fields are being read from Bytebuffers but unused.
[
https://issues.apache.org/jira/browse/HBASE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-14117:
---
Issue Type: Sub-task (was: Improvement)
Parent: HBASE-11425
Check DBEs where fields are being read from Bytebuffers but unused.
---
Key: HBASE-14117
URL: https://issues.apache.org/jira/browse/HBASE-14117
Project: HBase
Issue Type: Sub-task
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
{code}
public Cell getFirstKeyCellInBlock(ByteBuff block) {
block.mark();
block.position(Bytes.SIZEOF_INT);
int keyLength = ByteBuff.readCompressedInt(block);
// TODO : See if we can avoid these reads as the read values are not
getting used
ByteBuff.readCompressedInt(block);
{code}
In DBEs many a places we read the integers just to skip them. This JIRA is to
see if we can avoid this and rather go position based, as per a review
comment in HBASE-12213.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[
https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631437#comment-14631437
]
Hadoop QA commented on HBASE-12295:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745808/HBASE-12295_16.patch
against master branch at commit 834f87b23de533783ba5f5b858327a6164f17f55.
ATTACHMENT ID: 12745808
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 49 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1873 checkstyle errors (more than the master's current 1871 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.master.TestDistributedLogSplitting
{color:red}-1 core zombie tests{color}. There are 4 zombie test(s):
at
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testScanWithCompactionInternals(TestBlockEvictionFromClient.java:837)
at
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient.testScanWithCompaction(TestBlockEvictionFromClient.java:799)
at
org.apache.hadoop.hbase.client.TestReplicasClient.testSmallScanWithReplicas(TestReplicasClient.java:606)
at
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient.testRestoreSnapshot(TestRestoreSnapshotFromClient.java:162)
at
org.apache.hadoop.hbase.client.TestMetaWithReplicas.testMetaAddressChange(TestMetaWithReplicas.java:392)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14816//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14816//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14816//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14816//console
This message is automatically generated.
Prevent block eviction under us if reads are in progress from the BBs
-
Key: HBASE-12295
URL: https://issues.apache.org/jira/browse/HBASE-12295
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0
Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf,
HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch,
HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_2.patch,
HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf,
HBASE-12295_9.patch, HBASE-12295_trunk.patch
While we try to serve the reads from the BBs directly from the block cache,
we need to ensure that the blocks does not get evicted under us while
reading. This JIRA is to discuss and implement a strategy for the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14114) hbase-config.sh sets MLOCK_AGENT to incorrect value
[ https://issues.apache.org/jira/browse/HBASE-14114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Faris updated HBASE-14114: --- Attachment: HBASE-14114.patch hbase-config.sh sets MLOCK_AGENT to incorrect value --- Key: HBASE-14114 URL: https://issues.apache.org/jira/browse/HBASE-14114 Project: HBase Issue Type: Bug Affects Versions: 1.0.0 Reporter: Adam Faris Assignee: Adam Faris Priority: Trivial Attachments: HBASE-14114.patch I believe there's an incorrect path when setting MLOCK_AGENT in bin/hbase-config.sh. When I run maven with -Pnative, libmlockall_agent.so gets written to $HBASE_HOME/lib/native, not $HBASE_HOME/native. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630826#comment-14630826 ] Anoop Sam John commented on HBASE-13954: HBASE-12404 added this method to Scan. We will drop it? cc [~stack] I don't think we have to make it public. Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-14114) hbase-config.sh sets MLOCK_AGENT to incorrect value
Adam Faris created HBASE-14114: -- Summary: hbase-config.sh sets MLOCK_AGENT to incorrect value Key: HBASE-14114 URL: https://issues.apache.org/jira/browse/HBASE-14114 Project: HBase Issue Type: Bug Affects Versions: 1.0.0 Reporter: Adam Faris Assignee: Adam Faris Priority: Trivial I believe there's an incorrect path when setting MLOCK_AGENT in bin/hbase-config.sh. When I run maven with -Pnative, libmlockall_agent.so gets written to $HBASE_HOME/lib/native, not $HBASE_HOME/native. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630829#comment-14630829 ] Ashish Singhi commented on HBASE-13954: --- Ok. Thanks for the comments Anoop. I will post the next version of patch by removing that method from Scan class. [~saint@gmail.com] let me know if you disagree with it. Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630843#comment-14630843
]
Hudson commented on HBASE-14050:
SUCCESS: Integrated in HBase-1.1 #586 (See
[https://builds.apache.org/job/HBase-1.1/586/])
HBASE-14050 NPE in org.apache.hadoop.hbase.ipc.RpcServer.readAndProcess
(apurtell: rev ba270917055c3d1b43bb355a4a91686212a4cab0)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630863#comment-14630863 ] Ashish Singhi commented on HBASE-12596: --- Not sure, not that easy to predict from my side :) It depends on many things like what improvement, feature they are looking for, stability... bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[
https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630917#comment-14630917
]
Hadoop QA commented on HBASE-12213:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745755/HBASE-12213_final.patch
against master branch at commit a249989b93881b40a0919dc2fbd98530e05b6cf2.
ATTACHMENT ID: 12745755
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 70 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:red}-1 javac{color}. The applied patch generated 24 javac compiler
warnings (more than the master's current 20 warnings).
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14811//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14811//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14811//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14811//console
This message is automatically generated.
HFileBlock backed by Array of ByteBuffers
-
Key: HBASE-12213
URL: https://issues.apache.org/jira/browse/HBASE-12213
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: Anoop Sam John
Assignee: ramkrishna.s.vasudevan
Attachments: HBASE-12213_1.patch, HBASE-12213_10_withBBI.patch,
HBASE-12213_11_withBBI.patch, HBASE-12213_12_withBBI.patch,
HBASE-12213_12_withBBI.patch, HBASE-12213_13_withBBI.patch,
HBASE-12213_14_withBBI.patch, HBASE-12213_2.patch, HBASE-12213_4.patch,
HBASE-12213_8_withBBI.patch, HBASE-12213_9_withBBI.patch,
HBASE-12213_final.patch, HBASE-12213_jmh.zip
In L2 cache (offheap) an HFile block might have been cached into multiple
chunks of buffers. If HFileBlock need single BB, we will end up in recreation
of bigger BB and copying. Instead we can make HFileBlock to serve data from
an array of BBs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631034#comment-14631034 ] Anoop Sam John commented on HBASE-12213: Excellent !! Appreciate your effort.. HFileBlock backed by Array of ByteBuffers - Key: HBASE-12213 URL: https://issues.apache.org/jira/browse/HBASE-12213 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12213_1.patch, HBASE-12213_10_withBBI.patch, HBASE-12213_11_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_13_withBBI.patch, HBASE-12213_14_withBBI.patch, HBASE-12213_2.patch, HBASE-12213_4.patch, HBASE-12213_8_withBBI.patch, HBASE-12213_9_withBBI.patch, HBASE-12213_final.patch, HBASE-12213_jmh.zip In L2 cache (offheap) an HFile block might have been cached into multiple chunks of buffers. If HFileBlock need single BB, we will end up in recreation of bigger BB and copying. Instead we can make HFileBlock to serve data from an array of BBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630855#comment-14630855 ] Anoop Sam John commented on HBASE-12596: If a user migrating from 98 major version to 1.x major version whether they will go with older minor versions in this 1.x line or with latest minor version? (Say by that time we had 1.6). bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Attachment: HBASE-13954-v2.patch Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954-v2.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13971) Flushes stuck since 6 hours on a regionserver.
[ https://issues.apache.org/jira/browse/HBASE-13971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630892#comment-14630892 ] Elliott Clark commented on HBASE-13971: --- Thanks Flushes stuck since 6 hours on a regionserver. -- Key: HBASE-13971 URL: https://issues.apache.org/jira/browse/HBASE-13971 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 1.3.0 Environment: Caused while running IntegrationTestLoadAndVerify for 20 M rows on cluster with 32 region servers each with max heap size of 24GBs. Reporter: Abhilash Assignee: Ted Yu Priority: Critical Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: 13971-v1.txt, 13971-v1.txt, 13971-v1.txt, jstack.1, jstack.2, jstack.3, jstack.4, jstack.5, rsDebugDump.txt, screenshot-1.png One region server stuck while flushing(possible deadlock). Its trying to flush two regions since last 6 hours (see the screenshot). Caused while running IntegrationTestLoadAndVerify for 20 M rows with 600 mapper jobs and 100 back references. ~37 Million writes on each regionserver till now but no writes happening on any regionserver from past 6 hours and their memstore size is zero(I dont know if this is related). But this particular regionserver has memstore size of 9GBs from past 6 hours. Relevant snaps from debug dump: Tasks: === Task: Flushing IntegrationTestLoadAndVerify,R\x9B\x1B\xBF\xAE\x08\xD1\xA2,1435179555993.8e2d075f94ce7699f416ec4ced9873cd. Status: RUNNING:Preparing to flush by snapshotting stores in 8e2d075f94ce7699f416ec4ced9873cd Running for 22034s Task: Flushing IntegrationTestLoadAndVerify,\x93\xA385\x81Z\x11\xE6,1435179555993.9f8d0e01a40405b835bf6e5a22a86390. Status: RUNNING:Preparing to flush by snapshotting stores in 9f8d0e01a40405b835bf6e5a22a86390 Running for 22033s Executors: === ... Thread 139 (MemStoreFlusher.1): State: WAITING Blocked count: 139711 Waited count: 239212 Waiting on java.util.concurrent.CountDownLatch$Sync@b9c094a Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2047) org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2011) org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:1902) org.apache.hadoop.hbase.regionserver.HRegion.flush(HRegion.java:1828) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:510) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:471) org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$900(MemStoreFlusher.java:75) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:259) java.lang.Thread.run(Thread.java:745) Thread 137 (MemStoreFlusher.0): State: WAITING Blocked count: 138931 Waited count: 237448 Waiting on java.util.concurrent.CountDownLatch$Sync@53f41f76 Stack: sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) org.apache.hadoop.hbase.wal.WALKey.getSequenceId(WALKey.java:305) org.apache.hadoop.hbase.regionserver.HRegion.getNextSequenceId(HRegion.java:2422) org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2168)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630938#comment-14630938
]
Hadoop QA commented on HBASE-13954:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745759/HBASE-13954-v1.patch
against master branch at commit a249989b93881b40a0919dc2fbd98530e05b6cf2.
ATTACHMENT ID: 12745759
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ ualifier\030\002
\003(\014\\336\002\n\003Get\022\013\n\003row\030\001 \002(\014\022 \n\006c +
+ \016\n\006exists\030\003 \001(\010\022\024\n\005stale\030\004
\001(\010:\005false\022\026\n +
+ d\030\r \001(\010\022\r\n\005small\030\016
\001(\010\022\027\n\010reversed\030\017 \001(\010 +
+ new java.lang.String[] { Row, Column, Attribute, Filter,
TimeRange, MaxVersions, CacheBlocks, StoreLimit, StoreOffset,
ExistenceOnly, Consistency, });
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14812//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14812//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14812//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14812//console
This message is automatically generated.
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch,
HBASE-13954-v2.patch, HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630849#comment-14630849
]
Hudson commented on HBASE-14050:
FAILURE: Integrated in HBase-TRUNK #6659 (See
[https://builds.apache.org/job/HBase-TRUNK/6659/])
HBASE-14050 NPE in org.apache.hadoop.hbase.ipc.RpcServer.readAndProcess
(apurtell: rev a249989b93881b40a0919dc2fbd98530e05b6cf2)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14115) Fix resource leak in HMasterCommandLine
[
https://issues.apache.org/jira/browse/HBASE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuhao Bi updated HBASE-14115:
-
Attachment: HBASE-14115.patch
Fix resource leak in HMasterCommandLine
---
Key: HBASE-14115
URL: https://issues.apache.org/jira/browse/HBASE-14115
Project: HBase
Issue Type: Bug
Reporter: Yuhao Bi
Assignee: Yuhao Bi
Attachments: HBASE-14115.patch
In HMasterCommandLine#stopMaster(), admin is not closed.
{code:title=HMasterCommandLine.java|borderStyle=solid}
try (Connection connection = ConnectionFactory.createConnection(conf)) {
try (Admin admin = connection.getAdmin()) {
connection.getAdmin().shutdown();
} catch (Throwable t) {
LOG.error(Failed to stop master, t);
return 1;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14050) NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
[
https://issues.apache.org/jira/browse/HBASE-14050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630970#comment-14630970
]
Hudson commented on HBASE-14050:
FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1014 (See
[https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1014/])
HBASE-14050 NPE in org.apache.hadoop.hbase.ipc.RpcServer.readAndProcess
(apurtell: rev d0fc0466ebfcf8f14c5fedb11992bbba5030c4c4)
* hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java
NPE in org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess
--
Key: HBASE-14050
URL: https://issues.apache.org/jira/browse/HBASE-14050
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.12.1
Reporter: Andrew Purtell
Assignee: Andrew Purtell
Priority: Minor
Fix For: 2.0.0, 0.98.14, 1.0.2, 1.2.0, 1.1.2
Attachments: HBASE-14050.patch
{noformat}
2015-07-02 09:49:32,028 WARN [.reader=4,port=60020] ipc.RpcServer -
RpcServer.listener,port=60020: count of bytes read: 0
java.lang.NullPointerException
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1479)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:854)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:645)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:620)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14075) HBaseClusterManager should use port(if given) to find pid
[
https://issues.apache.org/jira/browse/HBASE-14075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630972#comment-14630972
]
Hadoop QA commented on HBASE-14075:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745760/HBASE-14075-master_v7.patch
against master branch at commit a249989b93881b40a0919dc2fbd98530e05b6cf2.
ATTACHMENT ID: 12745760
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14813//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14813//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14813//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14813//console
This message is automatically generated.
HBaseClusterManager should use port(if given) to find pid
-
Key: HBASE-14075
URL: https://issues.apache.org/jira/browse/HBASE-14075
Project: HBase
Issue Type: Bug
Reporter: Yu Li
Assignee: Yu Li
Priority: Minor
Attachments: HBASE-14075-master_v2.patch,
HBASE-14075-master_v3.patch, HBASE-14075-master_v4.patch,
HBASE-14075-master_v5.patch, HBASE-14075-master_v6.patch,
HBASE-14075-master_v7.patch, HBASE-14075.patch
This issue is found while we run ITBLL in distributed cluster. Our testing
env is kind of special that we run multiple regionserver instance on a single
physical machine, so {noformat}ps -ef | grep proc_regionserver{noformat} will
return more than one line, thus cause the tool might check/kill the wrong
process
Actually in HBaseClusterManager we already introduce port as a parameter for
methods like isRunning, kill, etc. So the only thing to do here is to get pid
through port if port is given
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631061#comment-14631061 ] Hudson commented on HBASE-12213: FAILURE: Integrated in HBase-TRUNK #6660 (See [https://builds.apache.org/job/HBase-TRUNK/6660/]) HBASE-12213 HFileBlock backed by Array of ByteBuffers (Ram) (ramkrishna: rev 834f87b23de533783ba5f5b858327a6164f17f55) * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/PrefixKeyDeltaEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestCacheConfig.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestByteBufferIOEngine.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CompoundBloomFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/ByteBufferIOEngine.java * hbase-common/src/test/java/org/apache/hadoop/hbase/nio/TestMultiByteBuff.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestByteBufferInputStream.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java * hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/FileIOEngine.java * hbase-common/src/test/java/org/apache/hadoop/hbase/io/TestMultiByteBuffInputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/IOEngine.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/PrefixTreeSeeker.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/CacheTestUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/UnsafeAccess.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferArray.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DiffKeyDeltaEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/nio/MultiByteBuffer.java * hbase-common/src/test/java/org/apache/hadoop/hbase/nio/TestMultiByteBuffer.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/CopyKeyDataBlockEncoder.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestBloomFilterChunk.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/TableLockChecker.java * hbase-common/src/main/java/org/apache/hadoop/hbase/nio/SingleByteBuff.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/ByteBufferInputStream.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CacheableDeserializer.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestByteBuffUtils.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFile.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/Hash.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/MemcachedBlockCache.java * hbase-common/src/main/java/org/apache/hadoop/hbase/nio/MultiByteBuff.java * hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestByteBufferUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterChunk.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/BloomFilterUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/FastDiffDeltaEncoder.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java *
[jira] [Commented] (HBASE-14059) We should add a RS to the dead servers list if admin calls fail more than a threshold
[ https://issues.apache.org/jira/browse/HBASE-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630851#comment-14630851 ] Heng Chen commented on HBASE-14059: --- So I think just add this RS to dead server list can't solve the problem, because the bad region could be transited into other RS and cause it's call queue maxed out. I think a better solution is to add the bad region into blacklist, and skip the request on this region to avoid the rs been blocked. We should add a RS to the dead servers list if admin calls fail more than a threshold - Key: HBASE-14059 URL: https://issues.apache.org/jira/browse/HBASE-14059 Project: HBase Issue Type: Bug Components: master, regionserver, rpc Affects Versions: 0.98.13 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Critical I ran into this problem twice this week: calls from the HBase master to a RS can timeout since the RS call queue size has been maxed out, however since the RS is not dead (ephemeral znode still present) the master keeps attempting to perform admin tasks like trying to open or close a region but those operations eventually fail after we run out of retries or the assignment manager attempts to re-assign to other RSs. From the side effects of this I've noticed master operations to be fully blocked or RITs since we cannot close the region and open the region in a new location since RS is not dead. A potential solution for this is to add the RS to the list of dead RSs after certain number of calls from the master to the RS fail. I've noticed only the problem in 0.98.x but it should be present in all versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630993#comment-14630993
]
Ashish Singhi commented on HBASE-13954:
---
Attached the v2 patch which removes package protected non-deprecated unused
method {{Scan#createGetClosestRowOrBeforeReverseScan}}.
The only thing pending will be to remove {{ThriftServerRunner#getRowOrBefore}}.
I don't know how to generate the thrift classes, so not yet addressed it. If I
figure it out soon I will post another patch here otherwise I will create a new
jira to address it (to avoid the patch from getting rot here).
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch,
HBASE-13954-v2.patch, HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14115) Fix resource leak in HMasterCommandLine
Yuhao Bi created HBASE-14115:
Summary: Fix resource leak in HMasterCommandLine
Key: HBASE-14115
URL: https://issues.apache.org/jira/browse/HBASE-14115
Project: HBase
Issue Type: Bug
Reporter: Yuhao Bi
Assignee: Yuhao Bi
In HMasterCommandLine#stopMaster(), admin is not closed.
{code:title=HMasterCommandLine.java|borderStyle=solid}
try (Connection connection = ConnectionFactory.createConnection(conf)) {
try (Admin admin = connection.getAdmin()) {
connection.getAdmin().shutdown();
} catch (Throwable t) {
LOG.error(Failed to stop master, t);
return 1;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12213: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to master. Will update the release notes shortly. Thanks for the patient reviews and nice feedbacks [~anoop.hbase] and [~saint@gmail.com]. The javac warnings should be due to the Unsafe method additions. HFileBlock backed by Array of ByteBuffers - Key: HBASE-12213 URL: https://issues.apache.org/jira/browse/HBASE-12213 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12213_1.patch, HBASE-12213_10_withBBI.patch, HBASE-12213_11_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_13_withBBI.patch, HBASE-12213_14_withBBI.patch, HBASE-12213_2.patch, HBASE-12213_4.patch, HBASE-12213_8_withBBI.patch, HBASE-12213_9_withBBI.patch, HBASE-12213_final.patch, HBASE-12213_jmh.zip In L2 cache (offheap) an HFile block might have been cached into multiple chunks of buffers. If HFileBlock need single BB, we will end up in recreation of bigger BB and copying. Instead we can make HFileBlock to serve data from an array of BBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14115) Fix resource leak in HMasterCommandLine
[
https://issues.apache.org/jira/browse/HBASE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuhao Bi updated HBASE-14115:
-
Status: Patch Available (was: Open)
Fix resource leak in HMasterCommandLine
---
Key: HBASE-14115
URL: https://issues.apache.org/jira/browse/HBASE-14115
Project: HBase
Issue Type: Bug
Reporter: Yuhao Bi
Assignee: Yuhao Bi
Attachments: HBASE-14115.patch
In HMasterCommandLine#stopMaster(), admin is not closed.
{code:title=HMasterCommandLine.java|borderStyle=solid}
try (Connection connection = ConnectionFactory.createConnection(conf)) {
try (Admin admin = connection.getAdmin()) {
connection.getAdmin().shutdown();
} catch (Throwable t) {
LOG.error(Failed to stop master, t);
return 1;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631516#comment-14631516 ] ramkrishna.s.vasudevan commented on HBASE-12295: TestRegionReplica is failing for me. Will correct that. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631490#comment-14631490
]
Hudson commented on HBASE-14109:
SUCCESS: Integrated in HBase-1.1 #587 (See
[https://builds.apache.org/job/HBase-1.1/587/])
HBASE-14109 NPE if we don't load fully before we are shutdown (stack: rev
cd4668b3eb9e3718436141f3869df34e484d60c7)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Open (was: Patch Available) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Patch Available (was: Open) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Attachment: HBASE-12295_16.patch Once again uploading correcting the checkstyle. The test case failure seems suspicious. But it is passing for me locally. Trying again. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14076) ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB
[ https://issues.apache.org/jira/browse/HBASE-14076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Esteban Gutierrez updated HBASE-14076: -- Resolution: Fixed Fix Version/s: hbase-11339 Status: Resolved (was: Patch Available) ResultSerialization and MutationSerialization can throw InvalidProtocolBufferException when serializing a cell larger than 64MB --- Key: HBASE-14076 URL: https://issues.apache.org/jira/browse/HBASE-14076 Project: HBase Issue Type: Bug Affects Versions: 2.0.0, hbase-11339, 1.2.0 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Fix For: hbase-11339 Attachments: HBASE-14076.hbase-11339.patch, HBASE-14076.hbase-11339.patch, HBASE-14076.v2.hbase-11339.patch This was reported in CRUNCH-534 but is a problem how we handle deserialization of large Cells ( 64MB) in ResultSerialization and MutationSerialization. The fix is just re-using what it was done in HBASE-13230. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
[ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631097#comment-14631097 ] Lars Hofhansl commented on HBASE-13992: --- bq. -Dspark.version=1.4.0 That's what I meant. Sorry I didn't say explicitly. bq. thank you vary much for your reviews today Did these happen offline? Integrate SparkOnHBase into HBase - Key: HBASE-13992 URL: https://issues.apache.org/jira/browse/HBASE-13992 Project: HBase Issue Type: New Feature Components: spark Reporter: Ted Malaska Assignee: Ted Malaska Fix For: 2.0.0 Attachments: HBASE-13992.patch, HBASE-13992.patch.3, HBASE-13992.patch.4 This Jira is to ask if SparkOnHBase can find a home in side HBase core. Here is the github: https://github.com/cloudera-labs/SparkOnHBase I am the core author of this project and the license is Apache 2.0 A blog explaining this project is here http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/ A spark Streaming example is here http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/ A real customer using this in produce is blogged here http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/ Please debate and let me know what I can do to make this happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631104#comment-14631104
]
Hadoop QA commented on HBASE-13954:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745780/HBASE-13954-v2.patch
against master branch at commit a249989b93881b40a0919dc2fbd98530e05b6cf2.
ATTACHMENT ID: 12745780
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 34 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+ ualifier\030\002
\003(\014\\336\002\n\003Get\022\013\n\003row\030\001 \002(\014\022 \n\006c +
+ \016\n\006exists\030\003 \001(\010\022\024\n\005stale\030\004
\001(\010:\005false\022\026\n +
+ d\030\r \001(\010\022\r\n\005small\030\016
\001(\010\022\027\n\010reversed\030\017 \001(\010 +
+ new java.lang.String[] { Row, Column, Attribute, Filter,
TimeRange, MaxVersions, CacheBlocks, StoreLimit, StoreOffset,
ExistenceOnly, Consistency, });
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14814//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14814//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14814//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14814//console
This message is automatically generated.
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch,
HBASE-13954-v2.patch, HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14115) Fix resource leak in HMasterCommandLine
[
https://issues.apache.org/jira/browse/HBASE-14115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631191#comment-14631191
]
Hadoop QA commented on HBASE-14115:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745786/HBASE-14115.patch
against master branch at commit 834f87b23de533783ba5f5b858327a6164f17f55.
ATTACHMENT ID: 12745786
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14815//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14815//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14815//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14815//console
This message is automatically generated.
Fix resource leak in HMasterCommandLine
---
Key: HBASE-14115
URL: https://issues.apache.org/jira/browse/HBASE-14115
Project: HBase
Issue Type: Bug
Reporter: Yuhao Bi
Assignee: Yuhao Bi
Attachments: HBASE-14115.patch
In HMasterCommandLine#stopMaster(), admin is not closed.
{code:title=HMasterCommandLine.java|borderStyle=solid}
try (Connection connection = ConnectionFactory.createConnection(conf)) {
try (Admin admin = connection.getAdmin()) {
connection.getAdmin().shutdown();
} catch (Throwable t) {
LOG.error(Failed to stop master, t);
return 1;
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631091#comment-14631091 ] Lars Hofhansl commented on HBASE-14098: --- That makes sense. Thanks [~eclark]. A cheap approximation might be to set it for major compactions. Agree that ideally we'd decided based on the size of the output file(s) compared to the available RAM for the OS buffer cache. Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098-v1.patch, HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12596) bulkload needs to follow locality
[ https://issues.apache.org/jira/browse/HBASE-12596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631093#comment-14631093 ] Lars Hofhansl commented on HBASE-12596: --- Let's move the discussion to the mailing list as [~apurtell] suggested. Didn't mean to hijack this issue for the discussion. There's no perfect way to get this right. I also wonder whether now it's better to adopt Hadoop's RM per RC model. I do like the idea of an early 2.0 shepherd (in fact I have suggested that before), who would turn into the RM for 2.0, once ready. I'd be happy to volunteer. Anyway... Let's discuss on dev. :) bulkload needs to follow locality - Key: HBASE-12596 URL: https://issues.apache.org/jira/browse/HBASE-12596 Project: HBase Issue Type: Improvement Components: HFile, regionserver Affects Versions: 0.98.8 Environment: hadoop-2.3.0, hbase-0.98.8, jdk1.7 Reporter: Victor Xu Assignee: Victor Xu Fix For: 2.0.0, 0.98.14, 1.3.0 Attachments: HBASE-12596-0.98-v1.patch, HBASE-12596-0.98-v2.patch, HBASE-12596-0.98-v3.patch, HBASE-12596-0.98-v4.patch, HBASE-12596-0.98-v5.patch, HBASE-12596-0.98-v6.patch, HBASE-12596-branch-1-v1.patch, HBASE-12596-branch-1-v2.patch, HBASE-12596-master-v1.patch, HBASE-12596-master-v2.patch, HBASE-12596-master-v3.patch, HBASE-12596-master-v4.patch, HBASE-12596-master-v5.patch, HBASE-12596-master-v6.patch, HBASE-12596.patch Normally, we have 2 steps to perform a bulkload: 1. use a job to write HFiles to be loaded; 2. Move these HFiles to the right hdfs directory. However, the locality could be loss during the first step. Why not just write the HFiles directly into the right place? We can do this easily because StoreFile.WriterBuilder has the withFavoredNodes method, and we just need to call it in HFileOutputFormat's getNewWriter(). This feature is enabled by default, and we could use 'hbase.bulkload.locality.sensitive.enabled=false' to disable it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14000) Region server failed to report Master and stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14000:
---
Hadoop Flags: Reviewed
Fix Version/s: 1.3.0
2.0.0
Region server failed to report Master and stuck in reportForDuty retry loop
---
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12213) HFileBlock backed by Array of ByteBuffers
[ https://issues.apache.org/jira/browse/HBASE-12213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12213: --- Release Note: The read path from the Hfileblock is now backed by a new data structure represented by an abstract class ByteBuff. The name is similar to nio ByteBuffer. The new ByteBuff class has various APIs to read and write primitives to the underlying implementation similar to how nio ByteBuffer provides. We have 2 implementations of the ByteBuff now - a single ByteBuffer backed implementation (SingleByteBuff) and a multiple ByteBuffer backed implementation (MultiByteBuff). The single byte buffer (SingleByteBuff) backed implementation will be used for the HFileBlocks that come from HDFS and from LRU cachce. The blocks that gets created out of BucketCache will be of type MultiByteBuff and the read operations are performed over this data structure which intelligently moves across different Byte buffers that it is formed of. HFileBlock backed by Array of ByteBuffers - Key: HBASE-12213 URL: https://issues.apache.org/jira/browse/HBASE-12213 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: Anoop Sam John Assignee: ramkrishna.s.vasudevan Attachments: HBASE-12213_1.patch, HBASE-12213_10_withBBI.patch, HBASE-12213_11_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_12_withBBI.patch, HBASE-12213_13_withBBI.patch, HBASE-12213_14_withBBI.patch, HBASE-12213_2.patch, HBASE-12213_4.patch, HBASE-12213_8_withBBI.patch, HBASE-12213_9_withBBI.patch, HBASE-12213_final.patch, HBASE-12213_jmh.zip In L2 cache (offheap) an HFile block might have been cached into multiple chunks of buffers. If HFileBlock need single BB, we will end up in recreation of bigger BB and copying. Instead we can make HFileBlock to serve data from an array of BBs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Open (was: Patch Available) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Status: Patch Available (was: Open) Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_17.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631582#comment-14631582
]
Hudson commented on HBASE-14109:
SUCCESS: Integrated in HBase-1.2-IT #58 (See
[https://builds.apache.org/job/HBase-1.2-IT/58/])
HBASE-14109 NPE if we don't load fully before we are shutdown (stack: rev
34b162d24d9c64c3bf672aec4577e1f0fb48b71b)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631598#comment-14631598
]
Srikanth Srungarapu commented on HBASE-14106:
-
+1 lgtm. The test errors are unrelated to the patch.
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 2.0.0, 1.2.0, 1.1.0.1
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0, 1.1.2
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14092) Add -noLock and -noBalanceSwitch options to hbck
[ https://issues.apache.org/jira/browse/HBASE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14092: -- Attachment: HBASE-14092-v1.patch It's not really possible to tell if the user wants to not turn the balancer off. If the balancer is on there's a possibility that hbck will report inconsistent even though things are just fine. So not locking is all about how ok the user is with that chance. If this command is being run in the background for alerting and flakey alerts are filtered then they are ok. If they are running this command to see if action is needed then they probably aren't ok with a flakey result. Add -noLock and -noBalanceSwitch options to hbck Key: HBASE-14092 URL: https://issues.apache.org/jira/browse/HBASE-14092 Project: HBase Issue Type: Bug Components: hbck, util Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14092-v1.patch, HBASE-14092.patch HBCK is sometimes used as a way to check the health of the cluster. When doing that it's not necessary to turn off the balancer. As such it's not needed to lock other runs of hbck out. We should add the --no-lock and --no-balancer command line flags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631639#comment-14631639
]
Hudson commented on HBASE-14109:
FAILURE: Integrated in HBase-1.3-IT #50 (See
[https://builds.apache.org/job/HBase-1.3-IT/50/])
HBASE-14109 NPE if we don't load fully before we are shutdown (stack: rev
d887e4cf1a5a82ac9a29f95519256b1390167a59)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-14106:
Resolution: Fixed
Fix Version/s: (was: 1.1.2)
Status: Resolved (was: Patch Available)
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 2.0.0, 1.2.0
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matteo Bertozzi updated HBASE-14106:
Affects Version/s: (was: 1.1.0.1)
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 2.0.0, 1.2.0
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14000) Region server failed to report to Master and was stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14000:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patch, Pankaj
Thanks for the review, Jerry.
Region server failed to report to Master and was stuck in reportForDuty retry
loop
--
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[
https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631568#comment-14631568
]
ramkrishna.s.vasudevan commented on HBASE-12295:
{code}
Running org.apache.hadoop.hbase.client.TestBlockEvictionFromClient
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 59.38 sec - in
org.apache.hadoop.hbase.client.TestBlockEvictionFromClient
Results :
{code}
Prevent block eviction under us if reads are in progress from the BBs
-
Key: HBASE-12295
URL: https://issues.apache.org/jira/browse/HBASE-12295
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0
Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf,
HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch,
HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch,
HBASE-12295_17.patch, HBASE-12295_2.patch, HBASE-12295_4.patch,
HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch,
HBASE-12295_trunk.patch
While we try to serve the reads from the BBs directly from the block cache,
we need to ensure that the blocks does not get evicted under us while
reading. This JIRA is to discuss and implement a strategy for the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14110) Add CostFunction for balancing primary region replicas
[ https://issues.apache.org/jira/browse/HBASE-14110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14110: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review, Enis. Add CostFunction for balancing primary region replicas -- Key: HBASE-14110 URL: https://issues.apache.org/jira/browse/HBASE-14110 Project: HBase Issue Type: Improvement Reporter: Ted Yu Assignee: Ted Yu Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: 14110-v1.txt Currently replicas for the same region are spread across region servers. However, one region server may serve disproportionately high number of primary region replicas. This has been observed in production cluster. This issue adds PrimaryRegionCountSkewCostFunction which facilitates balancing primary region replicas evenly across region servers. Thanks to Enis for fruitful discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631611#comment-14631611
]
Stephen Yuan Jiang commented on HBASE-14106:
The
org.apache.hadoop.hbase.master.procedure.TestWALProcedureStoreOnHDFS.testWalRollOnLowReplication
test failure is very likely NOT related to this change.
{noformat}
- Caused by: org.apache.hadoop.ipc.RemoteException: File
/test-logs/state-0013.log could only be replicated to 2 nodes
instead of minReplication (=3). There are 3 datanode(s) running and 3 node(s)
are excluded in this operation.
{noformat}
However, this test is for covering the low data nodes scenario (the failure
situation). So we have another flaky test to fix.
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 2.0.0, 1.2.0, 1.1.0.1
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0, 1.1.2
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14106) TestProcedureRecovery is flaky
[
https://issues.apache.org/jira/browse/HBASE-14106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631612#comment-14631612
]
Stephen Yuan Jiang commented on HBASE-14106:
+1. LGTM for the change.
TestProcedureRecovery is flaky
--
Key: HBASE-14106
URL: https://issues.apache.org/jira/browse/HBASE-14106
Project: HBase
Issue Type: Bug
Components: proc-v2, test
Affects Versions: 2.0.0, 1.2.0, 1.1.0.1
Reporter: Andrew Purtell
Assignee: Matteo Bertozzi
Fix For: 2.0.0, 1.2.0, 1.1.2
Attachments: HBASE-14106-v0.patch
Encountered this when running master tests locally using 7u79:
{noformat}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 12.28 sec
FAILURE! - in org.apache.hadoop.hbase.procedure2.TestProcedureRecovery
testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Time elapsed: 0.318 sec ERROR!
java.lang.IllegalArgumentException: null
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
at
org.apache.hadoop.hbase.procedure2.ProcedureExecutor.submitProcedure(ProcedureExecutor.java:595)
at
org.apache.hadoop.hbase.procedure2.ProcedureTestingUtility.submitAndWait(ProcedureTestingUtility.java:137)
at
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(TestProcedureRecovery.java:321)
{noformat}
{noformat}
Flaked tests:
org.apache.hadoop.hbase.procedure2.TestProcedureRecovery.testRunningProcWithSameNonce(org.apache.hadoop.hbase.procedure2.TestProcedureRecovery)
Run 1: TestProcedureRecovery.testRunningProcWithSameNonce:321 »
IllegalArgument
Run 2: PASS
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631626#comment-14631626 ] Jesse Yates commented on HBASE-14118: - The other reason to bring it into HBase is that the hbase-maven-plugin hasn't seen any activity in the last year or so, whereas I think the HBase folks will give it more love than its seen in a while. Talked with [~gwu] offline (one of the committers there) and seemed receptive to bringing it in. My biggest question is around licensing; its already Apache 2.0 licensed, what do we need to do to bring it into HBase (besides fixing it up to work on trunk)? [~apurtell] [~stack]? minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631651#comment-14631651 ] stack commented on HBASE-14118: --- So, folks doing integration tests on a single node would use this? Any other usages? St.Ack minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631549#comment-14631549
]
Hudson commented on HBASE-14109:
SUCCESS: Integrated in HBase-1.2 #74 (See
[https://builds.apache.org/job/HBase-1.2/74/])
HBASE-14109 NPE if we don't load fully before we are shutdown (stack: rev
34b162d24d9c64c3bf672aec4577e1f0fb48b71b)
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[ https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-12295: --- Attachment: HBASE-12295_17.patch Updated patch. Test cases are passing now. There are some mock scenarios where there will not be any RpcServerContext. Prevent block eviction under us if reads are in progress from the BBs - Key: HBASE-12295 URL: https://issues.apache.org/jira/browse/HBASE-12295 Project: HBase Issue Type: Sub-task Components: regionserver, Scanners Reporter: ramkrishna.s.vasudevan Assignee: ramkrishna.s.vasudevan Fix For: 2.0.0 Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf, HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch, HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch, HBASE-12295_17.patch, HBASE-12295_2.patch, HBASE-12295_4.patch, HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch, HBASE-12295_trunk.patch While we try to serve the reads from the BBs directly from the block cache, we need to ensure that the blocks does not get evicted under us while reading. This JIRA is to discuss and implement a strategy for the same. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13788) Shell comands do not support column qualifiers containing colon (:)
[
https://issues.apache.org/jira/browse/HBASE-13788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631600#comment-14631600
]
Ben Shuai commented on HBASE-13788:
---
Pankaj, adding :to won't fix any, as existing databases may already have that
as part of qualifiers and when it happens it breaks user's systems. Only
database owners know what to set.
Shell comands do not support column qualifiers containing colon (:)
---
Key: HBASE-13788
URL: https://issues.apache.org/jira/browse/HBASE-13788
Project: HBase
Issue Type: Bug
Components: shell
Affects Versions: 0.98.0, 0.96.0, 1.0.0, 1.1.0
Reporter: Dave Latham
Assignee: Pankaj Kumar
The shell interprets the colon within the qualifier as a delimiter to a
FORMATTER instead of part of the qualifier itself.
Example from the mailing list:
Hmph, I may have spoken too soon. I know I tested this at one point and
it worked, but now I'm getting different results:
On the new cluster, I created a duplicate test table:
hbase(main):043:0 create 'content3', {NAME = 'x', BLOOMFILTER =
'NONE', REPLICATION_SCOPE = '0', VERSIONS = '3', COMPRESSION =
'NONE', MIN_VERSIONS = '0', TTL = '2147483647', BLOCKSIZE = '65536',
IN_MEMORY = 'false', BLOCKCACHE = 'true'}
Then I pull some data from the imported table:
hbase(main):045:0 scan 'content', {LIMIT=1,
STARTROW='A:9223370612089311807:twtr:57013379'}
ROW COLUMN+CELL
A:9223370612089311807:twtr:570133798827921408
column=x:twitter:username, timestamp=1424775595345, value=BERITA
INFORMASI!
Then put it:
hbase(main):046:0 put
'content3','A:9223370612089311807:twtr:570133798827921408',
'x:twitter:username', 'BERITA INFORMASI!'
But then when I query it, I see that I've lost the column qualifier
:username:
hbase(main):046:0 scan 'content3'
ROW COLUMN+CELL
A:9223370612089311807:twtr:570133798827921408 column=x:twitter,
timestamp=1432745301788, value=BERITA INFORMASI!
Even though I'm missing one of the qualifiers, I can at least filter on
columns in this sample table.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14109) NPE if we don't load fully before we are shutdown
[
https://issues.apache.org/jira/browse/HBASE-14109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631602#comment-14631602
]
Hudson commented on HBASE-14109:
SUCCESS: Integrated in HBase-1.3 #64 (See
[https://builds.apache.org/job/HBase-1.3/64/])
HBASE-14109 NPE if we don't load fully before we are shutdown (stack: rev
d887e4cf1a5a82ac9a29f95519256b1390167a59)
*
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
*
hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
NPE if we don't load fully before we are shutdown
-
Key: HBASE-14109
URL: https://issues.apache.org/jira/browse/HBASE-14109
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 1.2.0
Reporter: stack
Assignee: stack
Priority: Trivial
Fix For: 1.2.0, 1.1.2, 1.3.0
Attachments: 14109.txt
We failed to find ZK so:
{code}
2015-07-15 22:51:04,725 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Initialization of RS failed.
Hence aborting RS.
java.io.IOException: Received the shutdown message while waiting.
at
org.apache.hadoop.hbase.regionserver.HRegionServer.blockAndCheckIfStopped(HRegionServer.java:807)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.initializeZooKeeper(HRegionServer.java:756)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.preRegistrationInitialization(HRegionServer.java:732)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:876)
at java.lang.Thread.run(Thread.java:745)
{code}
... which got us this:
{code}
2015-07-15 22:51:04,734 FATAL
[regionserver/c2022.halxg.cloudera.com/10.20.84.28:16020]
regionserver.HRegionServer: ABORTING region server
c2022.halxg.cloudera.com,16020,1437025808277: Unhandled: null
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:911)
at java.lang.Thread.run(Thread.java:745)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-14118) minicluster maven plugin
Jesse Yates created HBASE-14118: --- Summary: minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14092) Add -noLock and -noBalanceSwitch options to hbck
[ https://issues.apache.org/jira/browse/HBASE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631640#comment-14631640 ] Anoop Sam John commented on HBASE-14092: +1 Add -noLock and -noBalanceSwitch options to hbck Key: HBASE-14092 URL: https://issues.apache.org/jira/browse/HBASE-14092 Project: HBase Issue Type: Bug Components: hbck, util Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14092-v1.patch, HBASE-14092.patch HBCK is sometimes used as a way to check the health of the cluster. When doing that it's not necessary to turn off the balancer. As such it's not needed to lock other runs of hbck out. We should add the --no-lock and --no-balancer command line flags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[ https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631655#comment-14631655 ] Elliott Clark commented on HBASE-14098: --- I tested this out yesterday on a cluster with 1.1. When compacting 100+ GB storfiles I can reliably make the linux kernel pause trying to thrash and find pages to clear out. With just dropping the write's out of cache things are better and there are fewer stalls. With dropping the caches behind writes and reads from a compaction there are no noticeable stalls ( ones that atop sees that take 1 second ). So I think that we should provide the ability to drop caches behind reads and writes on large compactions ( those that would run on the large compaction threads ). However because this could adversely affect people with less data per machine I think that we should have this turned off by default. Thoughts? Allow dropping caches behind compactions Key: HBASE-14098 URL: https://issues.apache.org/jira/browse/HBASE-14098 Project: HBase Issue Type: Bug Components: Compaction, hadoop2, HFile Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.3.0 Attachments: HBASE-14098-v1.patch, HBASE-14098.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631661#comment-14631661 ] Jesse Yates commented on HBASE-14118: - Basic Cross version compat testing? So you could test a new version of HBase against the minicluster from the plugin in the last version. At least, that's all I can think of right now. But still would dramatically help us in cutting down non-destructive, integration test times. For downstreamers, like Phoenix, they wouldn't need to spin up/down a minicluster for every test (which ends up being a majority of the time spent in the test). It just seems like something the hbase project should own too :) minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14000) Region server failed to report to Master and was stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-14000:
---
Summary: Region server failed to report to Master and was stuck in
reportForDuty retry loop (was: Region server failed to report Master and stuck
in reportForDuty retry loop)
Region server failed to report to Master and was stuck in reportForDuty retry
loop
--
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14059) We should add a RS to the dead servers list if admin calls fail more than a threshold
[ https://issues.apache.org/jira/browse/HBASE-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631524#comment-14631524 ] Esteban Gutierrez commented on HBASE-14059: --- I don't think thats a good idea because even if we have this blacklisted region, other regions served by this RS might not be available too. The safe side should be to add the RS to the list of dead servers, one way could be to extend the Canary to remove the ephemeral znode for this RS to trigger the recovery of the regions. We could make the threshold configurable (e.g. time based, # of failed admin ops, etc.) or we we could do this from the master in a much more coordinated way. We should add a RS to the dead servers list if admin calls fail more than a threshold - Key: HBASE-14059 URL: https://issues.apache.org/jira/browse/HBASE-14059 Project: HBase Issue Type: Bug Components: master, regionserver, rpc Affects Versions: 0.98.13 Reporter: Esteban Gutierrez Assignee: Esteban Gutierrez Priority: Critical I ran into this problem twice this week: calls from the HBase master to a RS can timeout since the RS call queue size has been maxed out, however since the RS is not dead (ephemeral znode still present) the master keeps attempting to perform admin tasks like trying to open or close a region but those operations eventually fail after we run out of retries or the assignment manager attempts to re-assign to other RSs. From the side effects of this I've noticed master operations to be fully blocked or RITs since we cannot close the region and open the region in a new location since RS is not dead. A potential solution for this is to add the RS to the list of dead RSs after certain number of calls from the master to the RS fail. I've noticed only the problem in 0.98.x but it should be present in all versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631735#comment-14631735 ] Jesse Yates commented on HBASE-14118: - bq. Adding support, -Dminicluster.verson or whatever, for launching earlier or later releases when running integration tests, for checking forwards and backwards client compat. I think with this, you would actually just specify the version of the plugin to match the version of the minicluster you want to run, since we would release the plugin alongside the rest of the codebase. If we didn't have a minicluster plugin for that version, you are SOL (or we could back port it), but otherwise it will run that version of the HBase code. minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-14118: Attachment: hbase-maven-plugin-c444b9b6228327266094df266a99daa4f53958c4.tar.gz Tarball of hbase-maven-plugin (with the commit ID from whence it came). minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates Attachments: hbase-maven-plugin-c444b9b6228327266094df266a99daa4f53958c4.tar.gz In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631709#comment-14631709 ] Andrew Purtell commented on HBASE-14118: We need to follow this process: http://incubator.apache.org/ip-clearance/ip-clearance-template.html Generally, we need: a snapshot of the code (tarball), a Software Grant, and a CLA. The code must have no dependencies on items with licenses incompatible with the ASL. I am a Member of the ASF with Incubator karma so can execute the process assuming we want to do this. To move forward, we need a snapshot, and since [~gwu] has already said offline he can represent the copyright holder and is willing to execute a software grant, let's ask him to draw up a software grant (https://www.apache.org/licenses/software-grant.txt) and CCLA (https://www.apache.org/licenses/cla-corporate.txt), if one for Wibi isn't already on file, and submit them to the Apache Secretary (secret...@apache.org) Once we have a snapshot I can open a new clearance form. Once we have acknowledgement from the Secretary that the Grant and CLA are on file, I can commit the snapshot over at the incubator and start a clearance vote. Assuming that passes, I think we should commit the snapshot as-is on a branch, massage it post commit, then call a branch merge vote to get it into master. minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[ https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-13954: -- Attachment: HBASE-13954-v3.patch Renamed the new method added in TestFromClientSide Please review. Remove HTableInterface#getRowOrBefore related server side code -- Key: HBASE-13954 URL: https://issues.apache.org/jira/browse/HBASE-13954 Project: HBase Issue Type: Sub-task Components: API Reporter: Ashish Singhi Assignee: Ashish Singhi Fix For: 2.0.0 Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch, HBASE-13954-v2.patch, HBASE-13954-v3.patch, HBASE-13954.patch As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the review board to remove all the server side related code for getRowOrBefore. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631719#comment-14631719 ] Andrew Purtell commented on HBASE-14118: Anyway, before thinking about proceeding - should we? I'd be +1 minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[
https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631715#comment-14631715
]
Andrew Purtell commented on HBASE-14118:
I find applications in hbase-it interesting:
- Refactoring hbase-it so we don't deal with local cluster setup and tear down
for 'mvn verify', let the plugin deal with it
- Adding support, {{-Dminicluster.verson}} or whatever, for launching earlier
or later releases when running integration tests, for checking forwards and
backwards client compat.
- Assist downstreamers who want to build on hbase-it
minicluster maven plugin
Key: HBASE-14118
URL: https://issues.apache.org/jira/browse/HBASE-14118
Project: HBase
Issue Type: New Feature
Components: integration tests, util
Affects Versions: 2.0.0
Reporter: Jesse Yates
In the [failsafe docs
|https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they
describe being able to startup a jetty server before your integration tests
and then bring it down after the tests. We should be able to provide the same
for the mini-cluster for downstream projects and cross-version compatibility
testing.
The folks over in Kiji made this great [plugin
|https://github.com/kijiproject/hbase-maven-plugin] that lets you do just
that! We should see about bringing that under the HBase umbrella proper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631730#comment-14631730 ] Jesse Yates commented on HBASE-14118: - Ah, ok. Sorry [~apurtell], should have RTFS. thanks! minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631729#comment-14631729 ] Jesse Yates edited comment on HBASE-14118 at 7/17/15 6:48 PM: -- FWIW, spent a little bit bringing the plugin up to trunk ([github|https://github.com/jyates/hbase/tree/add-minicluster-plugin-14118]); well, with just hadoop 2.5.1 support, but not sure how we do version support anyways, so that will do the moment. Also, brought up my [next improvements|https://github.com/jyates/hbase/tree/add-plugin-improvements] to it that allow you to do things like running mvn org.apache.hbase:hbase-maven-plugin:run to bring up (and leave up) mini-cluster was (Author: jesse_yates): FWIW, spent a little bit bringing the plugin up to trunk ([github|https://github.com/jyates/hbase/tree/add-minicluster-plugin-14118]); well, with just hadoop 2.5.1 support, but not sure how we do version support anyways, so that will do the moment. Also, brought up my [next improvements|https://github.com/jyates/hbase/tree/add-minicluster-plugin-14118] to it that allow you to do things like running mvn org.apache.hbase:hbase-maven-plugin:run to bring up (and leave up) mini-cluster minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14000) Region server failed to report to Master and was stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631674#comment-14631674
]
Pankaj Kumar commented on HBASE-14000:
--
Thanks Ted :)
Region server failed to report to Master and was stuck in reportForDuty retry
loop
--
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14000) Region server failed to report to Master and was stuck in reportForDuty retry loop
[
https://issues.apache.org/jira/browse/HBASE-14000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631672#comment-14631672
]
Pankaj Kumar commented on HBASE-14000:
--
Thanks Jerry for reviewing the patch. :)
Region server failed to report to Master and was stuck in reportForDuty retry
loop
--
Key: HBASE-14000
URL: https://issues.apache.org/jira/browse/HBASE-14000
Project: HBase
Issue Type: Bug
Reporter: Pankaj Kumar
Assignee: Pankaj Kumar
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14000.patch, HM_RS-Log_snippet.txt
In a HA cluster, region server got stuck in reportForDuty retry loop if the
active master is restarting and later on master switch happens before it
reports successfully.
Root cause is same as HBASE-13317, but the region server tried to connect
master when it was starting, so rssStub reset didnt happen as
{code}
if (ioe instanceof ServerNotRunningYetException) {
LOG.debug(Master is not running yet);
}
{code}
When master starts, master switch happened. So RS always tried to connect to
standby master.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14092) Add -noLock and -noBalanceSwitch options to hbck
[ https://issues.apache.org/jira/browse/HBASE-14092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-14092: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.0 Status: Resolved (was: Patch Available) Thanks for review [~anoop.hbase] Add -noLock and -noBalanceSwitch options to hbck Key: HBASE-14092 URL: https://issues.apache.org/jira/browse/HBASE-14092 Project: HBase Issue Type: Bug Components: hbck, util Affects Versions: 2.0.0, 1.3.0 Reporter: Elliott Clark Assignee: Elliott Clark Fix For: 2.0.0, 1.2.0, 1.3.0 Attachments: HBASE-14092-v1.patch, HBASE-14092.patch HBCK is sometimes used as a way to check the health of the cluster. When doing that it's not necessary to turn off the balancer. As such it's not needed to lock other runs of hbck out. We should add the --no-lock and --no-balancer command line flags. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631685#comment-14631685
]
Lars Hofhansl commented on HBASE-14098:
---
Awesome!
Cache flush stalls can also be avoided by configuring the Linux disk cache.
Could you do a {{sysctl -a | grep dirty}} on one of the boxes and paste
vm.dirty_background_ratio, vm.dirty_background_bytes, and vm.dirty_ratio here?
Just curious :) Your previous comment made sense anyway. Caching anything that
is usually read sequentially and larger than the available RAM never makes
sense.
Allow dropping caches behind compactions
Key: HBASE-14098
URL: https://issues.apache.org/jira/browse/HBASE-14098
Project: HBase
Issue Type: Bug
Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14098-v1.patch, HBASE-14098.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-14039) BackupHandler.deleteSnapshot MUST use HBase Snapshot API
[ https://issues.apache.org/jira/browse/HBASE-14039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov resolved HBASE-14039. --- Resolution: Invalid Closing as invalid, will reopen later once we find any issues with the current implementation (deleting snapshot directory from FS directly). BackupHandler.deleteSnapshot MUST use HBase Snapshot API - Key: HBASE-14039 URL: https://issues.apache.org/jira/browse/HBASE-14039 Project: HBase Issue Type: Improvement Reporter: Vladimir Rodionov Assignee: Vladimir Rodionov Fix For: 2.0.0 BackupHandler.deleteSnapshot MUST use HBase API for that (HBaseAdmin) - not direct FS access (deleting snapshot folder may be not enough?). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14098) Allow dropping caches behind compactions
[
https://issues.apache.org/jira/browse/HBASE-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631708#comment-14631708
]
Elliott Clark commented on HBASE-14098:
---
{code}
vm.dirty_background_ratio = 3
vm.dirty_ratio = 20
{code}
The interesting part is that these stalls aren't really caused by dirty pages.
There's only 400mb of dirty pages. These stalls seem to be caused by thrashing
the cache. Too much reading/writing actually causes reads to happen on the root
disk to re-page in executables.
Allow dropping caches behind compactions
Key: HBASE-14098
URL: https://issues.apache.org/jira/browse/HBASE-14098
Project: HBase
Issue Type: Bug
Components: Compaction, hadoop2, HFile
Affects Versions: 2.0.0, 1.3.0
Reporter: Elliott Clark
Assignee: Elliott Clark
Fix For: 2.0.0, 1.3.0
Attachments: HBASE-14098-v1.patch, HBASE-14098.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631723#comment-14631723 ] Jesse Yates commented on HBASE-14118: - Do we need to bring it into the incubator? That seems a bit much for what is really not that much code (few hundred lines). I was hoping it would just fall in as a new module in HBase. If we need to, then we need to... and I'd be +1 for that too. minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13954) Remove HTableInterface#getRowOrBefore related server side code
[
https://issues.apache.org/jira/browse/HBASE-13954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631726#comment-14631726
]
Ashish Singhi commented on HBASE-13954:
---
Regarding thrift code.. I am facing some issue in installing thrift :( so for
now I will create a new jira and handle that. As it is not that major issue now
because ThriftServerRunner#getRowOrBefore is internally using reverse scan
{code}
private Result getRowOrBefore(byte[] tableName, byte[] row, byte[] family)
throws IOException {
Scan scan = new Scan(row);
scan.setReversed(true);
scan.addFamily(family);
scan.setStartRow(row);
Table table = getTable(tableName);
try (ResultScanner scanner = table.getScanner(scan)) {
return scanner.next();
}
}
{code}
Only thing is, it is deprecated so should be removed which should not be a
problem If I am not able to do as part of this jira.
Remove HTableInterface#getRowOrBefore related server side code
--
Key: HBASE-13954
URL: https://issues.apache.org/jira/browse/HBASE-13954
Project: HBase
Issue Type: Sub-task
Components: API
Reporter: Ashish Singhi
Assignee: Ashish Singhi
Fix For: 2.0.0
Attachments: HBASE-13954(1).patch, HBASE-13954-v1.patch,
HBASE-13954-v2.patch, HBASE-13954-v3.patch, HBASE-13954.patch
As part of HBASE-13214 review, [~anoop.hbase] had a review comment on the
review board to remove all the server side related code for getRowOrBefore.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12295) Prevent block eviction under us if reads are in progress from the BBs
[
https://issues.apache.org/jira/browse/HBASE-12295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631724#comment-14631724
]
Hadoop QA commented on HBASE-12295:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12745832/HBASE-12295_16.patch
against master branch at commit 834f87b23de533783ba5f5b858327a6164f17f55.
ATTACHMENT ID: 12745832
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 49 new
or modified tests.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.7.0)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestRegionReplicas
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.flume.channel.kafka.TestKafkaChannel.testSuccessInterleave(TestKafkaChannel.java:86)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/14817//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/14817//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/14817//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/14817//console
This message is automatically generated.
Prevent block eviction under us if reads are in progress from the BBs
-
Key: HBASE-12295
URL: https://issues.apache.org/jira/browse/HBASE-12295
Project: HBase
Issue Type: Sub-task
Components: regionserver, Scanners
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Fix For: 2.0.0
Attachments: HBASE-12295.pdf, HBASE-12295_1.patch, HBASE-12295_1.pdf,
HBASE-12295_10.patch, HBASE-12295_12.patch, HBASE-12295_14.patch,
HBASE-12295_15.patch, HBASE-12295_16.patch, HBASE-12295_16.patch,
HBASE-12295_17.patch, HBASE-12295_2.patch, HBASE-12295_4.patch,
HBASE-12295_4.pdf, HBASE-12295_5.pdf, HBASE-12295_9.patch,
HBASE-12295_trunk.patch
While we try to serve the reads from the BBs directly from the block cache,
we need to ensure that the blocks does not get evicted under us while
reading. This JIRA is to discuss and implement a strategy for the same.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[
https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631725#comment-14631725
]
Andrew Purtell commented on HBASE-14118:
No, we are not bringing it into the incubator.
http://incubator.apache.org/ip-clearance/
{quote}
This form is not for new projects. This is for projects and PMCs that have
already been created and are receiving a code donation into an existing
codebase. Any code that was developed outside of the ASF SVN repository and our
public mailing lists must be processed like this, even if the external
developer is already an ASF committer.
{quote}
Yes, we need to process the donation through the IP clearance process.
minicluster maven plugin
Key: HBASE-14118
URL: https://issues.apache.org/jira/browse/HBASE-14118
Project: HBase
Issue Type: New Feature
Components: integration tests, util
Affects Versions: 2.0.0
Reporter: Jesse Yates
In the [failsafe docs
|https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they
describe being able to startup a jetty server before your integration tests
and then bring it down after the tests. We should be able to provide the same
for the mini-cluster for downstream projects and cross-version compatibility
testing.
The folks over in Kiji made this great [plugin
|https://github.com/kijiproject/hbase-maven-plugin] that lets you do just
that! We should see about bringing that under the HBase umbrella proper.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14118) minicluster maven plugin
[ https://issues.apache.org/jira/browse/HBASE-14118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14631729#comment-14631729 ] Jesse Yates commented on HBASE-14118: - FWIW, spent a little bit bringing the plugin up to trunk ([github|https://github.com/jyates/hbase/tree/add-minicluster-plugin-14118]); well, with just hadoop 2.5.1 support, but not sure how we do version support anyways, so that will do the moment. Also, brought up my [next improvements|https://github.com/jyates/hbase/tree/add-minicluster-plugin-14118] to it that allow you to do things like running mvn org.apache.hbase:hbase-maven-plugin:run to bring up (and leave up) mini-cluster minicluster maven plugin Key: HBASE-14118 URL: https://issues.apache.org/jira/browse/HBASE-14118 Project: HBase Issue Type: New Feature Components: integration tests, util Affects Versions: 2.0.0 Reporter: Jesse Yates In the [failsafe docs |https://maven.apache.org/surefire/maven-failsafe-plugin/usage.html] they describe being able to startup a jetty server before your integration tests and then bring it down after the tests. We should be able to provide the same for the mini-cluster for downstream projects and cross-version compatibility testing. The folks over in Kiji made this great [plugin |https://github.com/kijiproject/hbase-maven-plugin] that lets you do just that! We should see about bringing that under the HBase umbrella proper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
