[jira] [Updated] (HBASE-7208) Transition Offline Snapshots to ForeignExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7208: -- Attachment: pre-hbase-7208.v7.patch hbase-7208.v7.patch > Transition Offline Snapshots to ForeignExceptions > - > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch, hbase-7208.v7.patch, > pre-hbase-7208.v6.patch, pre-hbase-7208.v7.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511196#comment-13511196 ] Matt Corgan commented on HBASE-7233: few thoughts: - we can make a KEY_VALUE encoder that serializes cells in the current wire format which is pretty simple for other languages to parse. it can be a slightly more performant fallback than per-field protocol buffers - encoders will have to be backwards compatible for a while on the server anyway because people have lots of hfiles encoded with them - encoders could have versions, but they are also pretty intricate, so any changes might merit a whole new encoder like FAST_DIFF2 - the client could pass a short list of encoder options in decending order of preference like FAST_DIFF2, KEY_VALUE, PB, where PB is the forever-supported fallback I'm a little skeptical that this will be the last client hbase ever supports. If something really major changes, we could make a whole new client and the server could translate things to support the old client. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511183#comment-13511183 ] stack commented on HBASE-7233: -- bq. Will that be enough to have old clients talk to new server (or vice versa)? Should have said, new server would also have to be able to do the old datablockencoding formats too -- whatever the client proffered -- or else fall back to lowest common denominator pb all the time. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511181#comment-13511181 ] Lars Hofhansl commented on HBASE-7282: -- I see. +1 commit v2 to 0.94. This is too nice of a tool to pass on. (v2 is almost identical to your version; it just re-adds the call to preCompactScannerOpen that the first one accidentally removed). > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, > HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7253: - Fix Version/s: (was: 0.94.4) > Compaction Tool > --- > > Key: HBASE-7253 > URL: https://issues.apache.org/jira/browse/HBASE-7253 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.96.0 > > Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch > > > In HBASE-5616, as part of the compaction code refactor, a CompactionTool was > added. > but there are some issues: > * The tool is under test/ > * mockito is required, so the "test" scope should be removed from the > pom.xml, otherwise the tool doesn't start > * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some > code (Store) uses HRegion.regionInfo directly HStore.java#L2021, > HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. > * The Mocked Store uses a dummy family and the compacted files doesn't get > the same family properties specified (compression, encoding, ...) > * at the end of compaction CompactionTool.java#L155, on by default, the > compaction file is removed (note that the compacted one are already removed > inside the store.compact()... and you end up with an empty dir, if you > compact everything. > I've fixed some stuff and added support to: > * Run the compaction as a MR Job > * Specify a Table (compact each region/family) > * Specify a Region (compact each family) > * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511178#comment-13511178 ] stack commented on HBASE-7282: -- I made the fat patch by applying one after the other. Patch hiccuped? Or I did applying them. I could redo? You'd set "hbase.hstore.compaction.complete" to false when you want to compact once only (rather than compact and then when done, if we find we could compact again, go ahead and do it). Correct me if I'm wrong [~mbertozzi] > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, > HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511171#comment-13511171 ] Lars Hofhansl commented on HBASE-7233: -- bq. Yeah, will have to keep versions on datablockencoding. Will that be enough to have old clients talk to new server (or vice versa)? That's what Writable did, and it did not work so well. Client and Server have pre-negotiate what they understand? > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511169#comment-13511169 ] Hudson commented on HBASE-7253: --- Integrated in HBase-0.94 #612 (See [https://builds.apache.org/job/HBase-0.94/612/]) HBASE-7253 Backport Compaction Tool to 0.94; REVERT (Revision 1417740) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java > Compaction Tool > --- > > Key: HBASE-7253 > URL: https://issues.apache.org/jira/browse/HBASE-7253 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch > > > In HBASE-5616, as part of the compaction code refactor, a CompactionTool was > added. > but there are some issues: > * The tool is under test/ > * mockito is required, so the "test" scope should be removed from the > pom.xml, otherwise the tool doesn't start > * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some > code (Store) uses HRegion.regionInfo directly HStore.java#L2021, > HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. > * The Mocked Store uses a dummy family and the compacted files doesn't get > the same family properties specified (compression, encoding, ...) > * at the end of compaction CompactionTool.java#L155, on by default, the > compaction file is removed (note that the compacted one are already removed > inside the store.compact()... and you end up with an empty dir, if you > compact everything. > I've fixed some stuff and added support to: > * Run the compaction as a MR Job > * Specify a Table (compact each region/family) > * Specify a Region (compact each family) > * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[
https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511167#comment-13511167
]
Lars Hofhansl commented on HBASE-7282:
--
Rest of the patch looks good. What is this:
{code}
// Move the compaction into place.
- sf = completeCompaction(filesToCompact, writer);
- if (region.getCoprocessorHost() != null) {
-region.getCoprocessorHost().postCompact(this, sf);
+ if (this.conf.getBoolean("hbase.hstore.compaction.complete", true)) {
+sf = completeCompaction(filesToCompact, writer);
+if (region.getCoprocessorHost() != null) {
+ region.getCoprocessorHost().postCompact(this, sf);
+}
+ } else {
+// Create storefile around what we wrote with a reader on it.
+sf = new StoreFile(this.fs, writer.getPath(), this.conf,
this.cacheConf,
+ this.family.getBloomFilterType(), this.dataBlockEncoder);
+sf.createReader();
}
{code}
What is "hbase.hstore.compaction.complete"? And why would ever not want to have
this true? Was that wrongly pulled in from trunk?
> Backport Compaction Tool to 0.94
>
>
> Key: HBASE-7282
> URL: https://issues.apache.org/jira/browse/HBASE-7282
> Project: HBase
> Issue Type: Sub-task
> Components: Compaction
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.94.4
>
> Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch,
> HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7282: - Attachment: 7253-v2.txt Here's a patch that fixes the issue. For this I copied the original compaction from Store.java into Compactor.compact(...) and changed it accordingly. Then I checked that I did come with something similar to what you had. The only difference was the call the preCompactScannerOpen. With this the three above mentioned tests pass. Of course now I am a bit less confident in the rest of the patch. > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, 7253-v2.txt, HBASE-5616-0.94.patch, > HBASE-5693-0.94.patch, HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511156#comment-13511156 ] stack commented on HBASE-7233: -- Yeah, will have to keep versions on datablockencoding. Clients other than hbase clients will be pretty hosed; if they are doing pure pb, hbase will be dog slow marshaling and unmarshaling, and if they want to go faster, they'll have to implement datablockencoding in whatever their language. Looking, avro would let us pass schema independent of data -- say at connection setup -- and because schema is external, could have tight on the wire representation. It lets you stream too it seems (haven't looked in code). Thrift supposedly too. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511143#comment-13511143 ] Lars Hofhansl commented on HBASE-7282: -- Looking at the patch again, it seems that just the call to preCompactScannerOpen got lost in the patch. > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511141#comment-13511141 ] Lars Hofhansl commented on HBASE-7282: -- Probably just needs some verification that the code in the new Compactor class is the same as it was before... Maybe on RB it is easier to verify? (Is it even possible to put a 0.94 up on RB?) Yeah, I loved the M/R compaction and compaction tool when I read the description/patch. In fact I like that idea so much that we should still try to get into 0.94. > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511136#comment-13511136 ] stack commented on HBASE-7282: -- Reverted. Sorry for breakage. Confirmed at least TestZooKeeperScanPolicyObserver passes again. This is a nice facility and can get fellows who have overrun their hbase w/ too many files out of their hole. Will see what Matteo says. > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-7213: --- Attachment: 7213-2.8.patch bq. I can remove this on commit: Oops.. apologies for missing the removal. This patch removes the interface method. bq. This looks new (and important): Yeah (smile). I noticed it recently.. bq. Lets wait some time on hbase-3171. If it doesn't show up soon, we'll commit this. Okay. That makes sense. Thanks folks for reviewing the patch and providing good feedback. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-2.6.patch, 7213-2.8.patch, > 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511130#comment-13511130 ] Lars Hofhansl commented on HBASE-7282: -- This is quite large change for 0.94 anyway, methinks. Maybe we should revert for now and upload to RB? > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl reopened HBASE-7282: -- > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7205) Coprocessor classloader is replicated for all regions in the HRegionServer
[
https://issues.apache.org/jira/browse/HBASE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511126#comment-13511126
]
Ted Yu commented on HBASE-7205:
---
Using patch v2, I got the following test failure:
{code}
testHBase3810(org.apache.hadoop.hbase.coprocessor.TestClassLoading) Time
elapsed: 7.976 sec <<< FAILURE!
java.lang.AssertionError: Class SimpleRegionObserver was missing on a region
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at
org.apache.hadoop.hbase.coprocessor.TestClassLoading.testHBase3810(TestClassLoading.java:415)
{code}
> Coprocessor classloader is replicated for all regions in the HRegionServer
> --
>
> Key: HBASE-7205
> URL: https://issues.apache.org/jira/browse/HBASE-7205
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 0.92.2, 0.94.2
>Reporter: Adrian Muraru
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.96.0, 0.94.4
>
> Attachments: 7205-v1.txt, HBASE-7205_v2.patch
>
>
> HBASE-6308 introduced a new custom CoprocessorClassLoader to load the
> coprocessor classes and a new instance of this CL is created for each single
> HRegion opened. This leads to OOME-PermGen when the number of regions go
> above hundres / region server.
> Having the table coprocessor jailed in a separate classloader is good however
> we should create only one for all regions of a table in each HRS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7282) Backport Compaction Tool to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511125#comment-13511125 ] Lars Hofhansl commented on HBASE-7282: -- This change breaks at least these tests: * TestZooKeeperScanPolicyObserver.testScanPolicyObserver * TestCoprocessorScanPolicy.testBaseCases * TestCoprocessorScanPolicy.testTTL Verified by selectively reverting the recent patches. > Backport Compaction Tool to 0.94 > > > Key: HBASE-7282 > URL: https://issues.apache.org/jira/browse/HBASE-7282 > Project: HBase > Issue Type: Sub-task > Components: Compaction >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: 7253.txt, HBASE-5616-0.94.patch, HBASE-5693-0.94.patch, > HBASE-6327-0.94.patch, HBASE-7253-0.94.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7205) Coprocessor classloader is replicated for all regions in the HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1354#comment-1354 ] Ted Yu commented on HBASE-7205: --- In RegionCoprocessorHost.postClose(), we can retrieve activeCoprocessorClassLoaders and release the (strong) references. > Coprocessor classloader is replicated for all regions in the HRegionServer > -- > > Key: HBASE-7205 > URL: https://issues.apache.org/jira/browse/HBASE-7205 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.92.2, 0.94.2 >Reporter: Adrian Muraru >Assignee: Ted Yu >Priority: Critical > Fix For: 0.96.0, 0.94.4 > > Attachments: 7205-v1.txt, HBASE-7205_v2.patch > > > HBASE-6308 introduced a new custom CoprocessorClassLoader to load the > coprocessor classes and a new instance of this CL is created for each single > HRegion opened. This leads to OOME-PermGen when the number of regions go > above hundres / region server. > Having the table coprocessor jailed in a separate classloader is good however > we should create only one for all regions of a table in each HRS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511109#comment-13511109 ] stack commented on HBASE-7213: -- This looks new (and important): +splitLog(serverNames, META_FILTER); +splitLog(serverNames, NON_META_FILTER); I can remove this on commit: /** @return the HLog */ public HLog getWAL(); Patch looks good to me. Lets wait some time on hbase-3171. If it doesn't show up soon, we'll commit this. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-2.6.patch, > 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-7213: - Priority: Critical (was: Major) Marking critical so we don't forget it, so it goes in. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das >Priority: Critical > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-2.6.patch, > 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7205) Coprocessor classloader is replicated for all regions in the HRegionServer
[
https://issues.apache.org/jira/browse/HBASE-7205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511098#comment-13511098
]
Andrew Purtell commented on HBASE-7205:
---
The patch HBASE-7205_v2.patch looks good except for this:
{noformat}
-"com.hadoop",
{noformat}
See the discussion over on HBASE-6843.
Adding logging and ZooKeeper classes to the exceptions list is a good idea,
thanks for fixing that oversight.
Should have a test that opens a table with a few regions and confirms there's
only one CP classloader instance for all of them?
> Coprocessor classloader is replicated for all regions in the HRegionServer
> --
>
> Key: HBASE-7205
> URL: https://issues.apache.org/jira/browse/HBASE-7205
> Project: HBase
> Issue Type: Bug
> Components: Coprocessors
>Affects Versions: 0.92.2, 0.94.2
>Reporter: Adrian Muraru
>Assignee: Ted Yu
>Priority: Critical
> Fix For: 0.96.0, 0.94.4
>
> Attachments: 7205-v1.txt, HBASE-7205_v2.patch
>
>
> HBASE-6308 introduced a new custom CoprocessorClassLoader to load the
> coprocessor classes and a new instance of this CL is created for each single
> HRegion opened. This leads to OOME-PermGen when the number of regions go
> above hundres / region server.
> Having the table coprocessor jailed in a separate classloader is good however
> we should create only one for all regions of a table in each HRS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5616) Make compaction code standalone
[ https://issues.apache.org/jira/browse/HBASE-5616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511095#comment-13511095 ] Hudson commented on HBASE-5616: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java > Make compaction code standalone > --- > > Key: HBASE-5616 > URL: https://issues.apache.org/jira/browse/HBASE-5616 > Project: HBase > Issue Type: Improvement >Reporter: stack >Assignee: stack > Fix For: 0.96.0, 0.94.4 > > Attachments: 5616.txt, 5616v3.txt, 5616v6.txt, 5616v7.txt, > 5616v7.txt, addlicense.txt, standalone.txt > > > This is part of hbase-2462. Make the compaction code standalone so can run > it independent of hbase. Will make it easier to profile and try stuff out. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7279) Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511092#comment-13511092 ] Hudson commented on HBASE-7279: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7279 Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher (Revision 1417716) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Result.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java > Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher > - > > Key: HBASE-7279 > URL: https://issues.apache.org/jira/browse/HBASE-7279 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7279-0.94.txt, 7279-0.94-v2.txt, 7279-0.94-v3.txt, > 7279-0.94-v4.txt, 7279-0.96-v1.txt, 7279-0.96-v2.txt, 7279-0.96-v3.txt > > > Did some profiling again. > I we can gain some performance [1] when passing buffer, rowoffset, and > rowlength instead of making a copy of the row key. > That way we can also remove the row key caching (and this patch also removes > the timestamps caching). Considering the sheer number in which we create KVs, > every byte save is good. > [1] (15-20% when data is in the block cache we setup a Filter such that only > a single row is returned to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511094#comment-13511094 ] Hudson commented on HBASE-6327: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java > HLog can be null when create table > -- > > Key: HBASE-6327 > URL: https://issues.apache.org/jira/browse/HBASE-6327 > Project: HBase > Issue Type: Bug >Reporter: ShiXing >Assignee: ShiXing > Fix For: 0.96.0, 0.94.4 > > Attachments: 6327.txt, createTableFailedMaster.log, > HBASE-6327-trunk-V1.patch > > > As HBASE-4010 discussed, the HLog can be null. > We have meet createTable failed because the no use hlog. > When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call > the DFSClient.DFSOutputStream.sync(). > Then the hlog.closeAndDelete() was called，firstly the HLog.close() will > interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The > DFSClient.DFSOutputStream will store the exception and throw it when we > called DFSClient.close(). > The HLog.close() call the writer.close()/DFSClient.close() after interrupt > the LogSyncer. And there is no catch exception for the close(). > So the Master throw exception to the client. There is no need to throw this > exception, further， the hlog is no use. > Our cluster is 0.90, the logs is attached, after "closing hlog writer", there > is no log for the createTable(). > The trunk and 0.92, 0.94, we used just one hlog, and if the exception > happends, the client will got createTable failed, but indeed ,we expect all > the regions for the table can also be assigned. > I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5693) When creating a region, the master initializes it and creates a memstore within the master server
[ https://issues.apache.org/jira/browse/HBASE-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511090#comment-13511090 ] Hudson commented on HBASE-5693: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java > When creating a region, the master initializes it and creates a memstore > within the master server > - > > Key: HBASE-5693 > URL: https://issues.apache.org/jira/browse/HBASE-5693 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: nkeywal >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: 5593.v2.patch, 5693.v1.patch > > > I didn't do a complete analysis, but the attached patch saves more than 0.25s > for each region creation and locally all the unit tests work. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7253) Compaction Tool
[ https://issues.apache.org/jira/browse/HBASE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511093#comment-13511093 ] Hudson commented on HBASE-7253: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7253 Backport Compaction Tool to 0.94; includes HBASE-5616, HBASE-5693, HBASE-6327, HBASE-7253 (Revision 1417559) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/handler/CreateTableHandler.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/CompactionTool.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Compactor.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/compactions/CompactionProgress.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/ChecksumType.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java > Compaction Tool > --- > > Key: HBASE-7253 > URL: https://issues.apache.org/jira/browse/HBASE-7253 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-7253-v0.patch, HBASE-7253-v1.patch > > > In HBASE-5616, as part of the compaction code refactor, a CompactionTool was > added. > but there are some issues: > * The tool is under test/ > * mockito is required, so the "test" scope should be removed from the > pom.xml, otherwise the tool doesn't start > * The mock, used by the tool, is mocking HRegion.getRegionInfo() but some > code (Store) uses HRegion.regionInfo directly HStore.java#L2021, > HStore.java#L1389, HStore.java#L1402 and you end up with a NPE in the tool. > * The Mocked Store uses a dummy family and the compacted files doesn't get > the same family properties specified (compression, encoding, ...) > * at the end of compaction CompactionTool.java#L155, on by default, the > compaction file is removed (note that the compacted one are already removed > inside the store.compact()... and you end up with an empty dir, if you > compact everything. > I've fixed some stuff and added support to: > * Run the compaction as a MR Job > * Specify a Table (compact each region/family) > * Specify a Region (compact each family) > * Specify a Family (as before) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5888) Clover profile in build
[ https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511091#comment-13511091 ] Hudson commented on HBASE-5888: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-5888: Clover profile in build (Andrey Klochkov) (Revision 1417231) Result = FAILURE jyates : Files : * /hbase/branches/0.94/pom.xml > Clover profile in build > --- > > Key: HBASE-5888 > URL: https://issues.apache.org/jira/browse/HBASE-5888 > Project: HBase > Issue Type: Improvement > Components: build, test >Affects Versions: 0.92.2, 0.94.1, 0.96.0 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-5888-0.94.patch, HBASE-5888-0.94.patch, > hbase-clover_v1.patch, hbase-clover_v2.patch > > > Clover is disabled right now. I would like to add a profile that enables > clover reports. We can also backport this to 0.92, and 0.94, since we are > also interested in test coverage for those branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7260) Upgrade hadoop 1 dependency to hadoop 1.1.1
[ https://issues.apache.org/jira/browse/HBASE-7260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511087#comment-13511087 ] Hudson commented on HBASE-7260: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7260 Upgrade hadoop 1 dependency to hadoop 1.1.1 (Revision 1417107) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/pom.xml > Upgrade hadoop 1 dependency to hadoop 1.1.1 > --- > > Key: HBASE-7260 > URL: https://issues.apache.org/jira/browse/HBASE-7260 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.96.0, 0.94.4 > > Attachments: 7260.94, 7260.txt > > > hadoop 1.1.1 has been released with 20 bug fixes and improvements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6206) Large tests fail with jdk1.7
[ https://issues.apache.org/jira/browse/HBASE-6206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511088#comment-13511088 ] Hudson commented on HBASE-6206: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-6206 Large tests fail with jdk1.7 (Revision 1417249) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Get.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/client/Scan.java > Large tests fail with jdk1.7 > > > Key: HBASE-6206 > URL: https://issues.apache.org/jira/browse/HBASE-6206 > Project: HBase > Issue Type: Sub-task > Components: Thrift >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: 6206-hbase.patch > > > Failed tests: > testSimplePutDelete(org.apache.hadoop.hbase.replication.TestMasterReplication): > Waited too much time for put replication > > testCyclicReplication(org.apache.hadoop.hbase.replication.TestMasterReplication): > Waited too much time for put replication > > testMultiSlaveReplication(org.apache.hadoop.hbase.replication.TestMultiSlaveReplication): > Waited too much time for put replication > testDeleteTypes(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for put replication > testSimplePutDelete(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for put replication > testSmallBatch(org.apache.hadoop.hbase.replication.TestReplication): Waited > too much time for normal batch replication > testStartStop(org.apache.hadoop.hbase.replication.TestReplication): Waited > too much time for put replication > testDisableEnable(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for put replication > > testDisableInactivePeer(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for put replication > > testAddAndRemoveClusters(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for put replication > loadTesting(org.apache.hadoop.hbase.replication.TestReplication): Waited > too much time for normal batch replication, 0 instead of 1000 > testVerifyRepJob(org.apache.hadoop.hbase.replication.TestReplication): > Waited too much time for normal batch replication > > testRSSplitEphemeralsDisappearButDaughtersAreOnlinedAfterShutdownHandling(org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster) > Tests in error: > testNull(org.apache.hadoop.hbase.client.TestFromClientSide) > testPut(org.apache.hadoop.hbase.client.TestFromClientSide) > testSplit(org.apache.hadoop.hbase.regionserver.wal.TestHLog): 3 exceptions > [org.apache.hadoop.ipc.RemoteException: > org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on > /user/jxiang/hbase/TestHLog/08fec3b92db148f74a1599c3db6fb1ee/recovered.edits/004.temp > File is not open for writing. Holder DFSClient_388157531 does not have any > open files.(..) > Tests run: 333, Failures: 4, Errors: 3, Skipped: 7 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7273) Upgrade zookeeper dependency to 3.4.5 for 0.94
[ https://issues.apache.org/jira/browse/HBASE-7273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511089#comment-13511089 ] Hudson commented on HBASE-7273: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7273 Upgrade zookeeper dependency to 3.4.5 for 0.94 (Revision 1417255) Result = FAILURE stack : Files : * /hbase/branches/0.94/pom.xml > Upgrade zookeeper dependency to 3.4.5 for 0.94 > -- > > Key: HBASE-7273 > URL: https://issues.apache.org/jira/browse/HBASE-7273 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Fix For: 0.94.4 > > Attachments: 7273.txt > > > 0.94 is using zookeeper 3.4.3 > We should upgrade to 3.4.5 > HBASE-4791, e.g., needs 3.4.5 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511086#comment-13511086 ] Hudson commented on HBASE-6423: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-6423 Writes should not block reads on blocking updates to memstores - Addendum 2 (Revision 1417247) HBASE-6423 Writes should not block reads on blocking updates to memstores - Addendum (Revision 1417243) HBASE-6423 Writes should not block reads on blocking updates to memstores (Revision 1417079) Result = FAILURE tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java tedyu : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java jxiang : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/RegionTooBusyException.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Fix For: 0.96.0, 0.94.4 > > Attachments: 0.94-6423.patch, 0.94-6423_v4.patch, 6423.addendum, > trunk-6423.patch, trunk-6423_v2.1.patch, trunk-6423_v2.patch, > trunk-6423_v3.2.patch, trunk-6423_v3.3.patch, trunk-6423_v3.4.patch, > trunk-6423_v4.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511085#comment-13511085 ] Hudson commented on HBASE-7249: --- Integrated in HBase-0.94 #611 (See [https://builds.apache.org/job/HBase-0.94/611/]) HBASE-7249 add test name filter to IntegrationTestsDriver (Revision 1417236) Result = FAILURE stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3, 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0-094.patch, > HBASE-7249-v0.patch, HBASE-7249-v1.patch, HBASE-7249-v2-094.patch, > HBASE-7249-v2.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7280) TableNotFoundException thrown in peer cluster will incur endless retry for shipEdits, which in turn block following normal replication
[ https://issues.apache.org/jira/browse/HBASE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511076#comment-13511076 ] Feng Honghua commented on HBASE-7280: - yes, that's what I hope for the finer-grained cluster replication. for such design by default (without any table/cf configuration) peer receives all the edits from master cluster. Since in real-world scenario, we may have a master cluster, and a backup cluster which need to replicate the whole copy of the master cluster and it receives all edits, but at the same time maybe there are some experiment/down-stream clusters which just need a certain table or even some CF of a table from master cluster. by providing table/cf configurable peer we can enable such scenarios. ReplicationSource need to parse out the peer's table/cf configuration on creation, and filter the edits while reading the HLog files to determine which edits needs to be shipped to the corresponding peer. Looks like no more change in peer-side (ReplicationSink), right? Yes, my current change in ReplicationSink doesn't save the unnecessary edits to peers, but it's enough to unblocks us. A wiser treatment should be in ReplicationSource where we can filter out unnecessary edits before shipping out to peer cluster by checking if the table exists at peer cluster for each edit. > TableNotFoundException thrown in peer cluster will incur endless retry for > shipEdits, which in turn block following normal replication > -- > > Key: HBASE-7280 > URL: https://issues.apache.org/jira/browse/HBASE-7280 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.2 >Reporter: Feng Honghua > Fix For: 0.94.4 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > in cluster replication, if the master cluster have 2 tables which have > column-family declared with replication scope = 1, and add a peer cluster > which has only 1 table with the same name as the master cluster, in the > ReplicationSource (thread in master cluster) for this peer, edits (logs) for > both tables will be shipped to the peer, the peer will fail applying the > edits due to TableNotFoundException, and this exception will also be > responsed to the original shipper (ReplicationSource in master cluster), and > the shipper will fall into an endless retry for shipping the failed edits > without proceeding to read the remained(newer) log files and to ship > following edits(maybe the normal, expected edit for the registered table). > the symptom looks like the TableNotFoundException incurs endless retry and > blocking normal table replication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5258) Move coprocessors set out of RegionLoad
[ https://issues.apache.org/jira/browse/HBASE-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511077#comment-13511077 ] Andrew Purtell commented on HBASE-5258: --- bq. I don't see getting coprocessors from RegionLoad actually being used anywhere. +1 I searched for users of RegionLoad#getCoprocessors and didn't find any either. Originally we put this in so the loaded coprocessors on a region would show up next to other region information in the master UI. Unfortunately the result was not that useful, just clutter. > Move coprocessors set out of RegionLoad > --- > > Key: HBASE-5258 > URL: https://issues.apache.org/jira/browse/HBASE-5258 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Priority: Critical > > When I worked on HBASE-5256, I revisited the code related to Ser/De of > coprocessors set in RegionLoad. > I think the rationale for embedding coprocessors set is for maximum > flexibility where each region can load different coprocessors. > This flexibility is causing extra cost in the region server to Master > communication and increasing the footprint of Master heap. > Would HServerLoad be a better place for this set ? > If required, region server should calculate disparity of loaded coprocessors > among regions and send report through HServerLoad -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511065#comment-13511065 ] Hudson commented on HBASE-6423: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-6423 Writes should not block reads on blocking updates to memstores - Addendum (Revision 1417242) HBASE-6423 Writes should not block reads on blocking updates to memstores (Revision 1417078) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java jxiang : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionTooBusyException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionBusyWait.java > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Jimmy Xiang > Fix For: 0.96.0, 0.94.4 > > Attachments: 0.94-6423.patch, 0.94-6423_v4.patch, 6423.addendum, > trunk-6423.patch, trunk-6423_v2.1.patch, trunk-6423_v2.patch, > trunk-6423_v3.2.patch, trunk-6423_v3.3.patch, trunk-6423_v3.4.patch, > trunk-6423_v4.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7279) Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511067#comment-13511067 ] Hudson commented on HBASE-7279: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7279 Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher (Revision 1417715) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java > Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher > - > > Key: HBASE-7279 > URL: https://issues.apache.org/jira/browse/HBASE-7279 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7279-0.94.txt, 7279-0.94-v2.txt, 7279-0.94-v3.txt, > 7279-0.94-v4.txt, 7279-0.96-v1.txt, 7279-0.96-v2.txt, 7279-0.96-v3.txt > > > Did some profiling again. > I we can gain some performance [1] when passing buffer, rowoffset, and > rowlength instead of making a copy of the row key. > That way we can also remove the row key caching (and this patch also removes > the timestamps caching). Considering the sheer number in which we create KVs, > every byte save is good. > [1] (15-20% when data is in the block cache we setup a Filter such that only > a single row is returned to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7285) HMaster fails to start with secure Hadoop
[
https://issues.apache.org/jira/browse/HBASE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511066#comment-13511066
]
Hudson commented on HBASE-7285:
---
Integrated in HBase-TRUNK #3594 (See
[https://builds.apache.org/job/HBase-TRUNK/3594/])
HBASE-7285 HMaster fails to start with secure Hadoop (Revision 1417690)
Result = FAILURE
stack :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> HMaster fails to start with secure Hadoop
> -
>
> Key: HBASE-7285
> URL: https://issues.apache.org/jira/browse/HBASE-7285
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.96.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.96.0
>
> Attachments: HBASE-7285.patch
>
>
> In current trunk, HMaster will fail to start with secure Hadoop if the user
> starting the process has not obtained a kerberos TGT. The user starting the
> process should not be required to have a TGT, as the HMaster process self
> logs in using the configured keytab and principal.
> This is due to a log line in the HMaster constructor executing prior to the
> {{User.login()}} step:
> {code}
> LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
> ", hbase.cluster.distributed=" +
> this.conf.getBoolean("hbase.cluster.distributed", false));
> {code}
> Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
> trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7008) Set scanner caching to a better default, disable Nagles
[ https://issues.apache.org/jira/browse/HBASE-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511064#comment-13511064 ] Hudson commented on HBASE-7008: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7008 Set scanner caching to a better default, disable Nagles (Revision 1417233) Result = FAILURE larsh : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HTable.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java > Set scanner caching to a better default, disable Nagles > --- > > Key: HBASE-7008 > URL: https://issues.apache.org/jira/browse/HBASE-7008 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: liang xie >Assignee: liang xie > Fix For: 0.96.0 > > Attachments: 7008-0.94.txt, 7008-0.94-v2.txt, 7008-0.94-v3.txt, > 7008-trunk-v5.txt, 7008-trunk-v6.txt, 7008-v3.txt, 7008-v4.txt, > HBASE-7008.patch, HBASE-7008-v2.patch > > > per > http://search-hadoop.com/m/qaRu9iM2f02/Set+scanner+caching+to+a+better+default%253F&subj=Set+scanner+caching+to+a+better+default+ > let's set to 100 by default -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7277) Thrift default JMX port should be 10103 instead of 8093
[ https://issues.apache.org/jira/browse/HBASE-7277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511061#comment-13511061 ] Hudson commented on HBASE-7277: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7277 Thrift default JMX port should be 10103 instead of 8093 (Revision 1417530) Result = FAILURE jxiang : Files : * /hbase/trunk/bin/hbase-config.sh > Thrift default JMX port should be 10103 instead of 8093 > --- > > Key: HBASE-7277 > URL: https://issues.apache.org/jira/browse/HBASE-7277 > Project: HBase > Issue Type: Bug >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: 7277.patch > > > HBASE-5879 set Thrift JMX port to 8093. In conf/hbase-env.sh, the default > one is 10103 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7249) add test name filter to IntegrationTestsDriver
[ https://issues.apache.org/jira/browse/HBASE-7249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511063#comment-13511063 ] Hudson commented on HBASE-7249: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7249 add test name filter to IntegrationTestsDriver (Revision 1417230) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/ClassTestFinder.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestsDriver.java * /hbase/trunk/src/docbkx/developer.xml > add test name filter to IntegrationTestsDriver > -- > > Key: HBASE-7249 > URL: https://issues.apache.org/jira/browse/HBASE-7249 > Project: HBase > Issue Type: Improvement > Components: test >Affects Versions: 0.94.3, 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Fix For: 0.96.0, 0.94.4 > > Attachments: HBASE-7249-v0-094.patch, HBASE-7249-v0-094.patch, > HBASE-7249-v0.patch, HBASE-7249-v1.patch, HBASE-7249-v2-094.patch, > HBASE-7249-v2.patch > > > As the number of tests grows, one might want to just run one, or a subset -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7270) Remove MultiPut and MultiPutResponse to satisfy rat-check
[ https://issues.apache.org/jira/browse/HBASE-7270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511062#comment-13511062 ] Hudson commented on HBASE-7270: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7270: Remove MultiPut and MultiPutResponse to satisfy rat-check (Revision 1417015) Result = FAILURE jyates : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPut.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/MultiPutResponse.java > Remove MultiPut and MultiPutResponse to satisfy rat-check > - > > Key: HBASE-7270 > URL: https://issues.apache.org/jira/browse/HBASE-7270 > Project: HBase > Issue Type: Bug >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0 > > Attachments: hbase-7270.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7274) Enable JMX metrics collection for REST Server
[ https://issues.apache.org/jira/browse/HBASE-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511060#comment-13511060 ] Hudson commented on HBASE-7274: --- Integrated in HBase-TRUNK #3594 (See [https://builds.apache.org/job/HBase-TRUNK/3594/]) HBASE-7274 Enable JMX metrics collection for REST Server (Revision 1417527) Result = FAILURE jxiang : Files : * /hbase/trunk/bin/hbase-config.sh * /hbase/trunk/conf/hbase-env.sh > Enable JMX metrics collection for REST Server > - > > Key: HBASE-7274 > URL: https://issues.apache.org/jira/browse/HBASE-7274 > Project: HBase > Issue Type: Improvement > Components: REST >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 0.96.0 > > Attachments: trunk-7274.patch > > > REST server should accept JMX options. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7280) TableNotFoundException thrown in peer cluster will incur endless retry for shipEdits, which in turn block following normal replication
[ https://issues.apache.org/jira/browse/HBASE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511053#comment-13511053 ] Jieshan Bean commented on HBASE-7280: - I agree with your suggesion of adding configuration list for each peer. So we need to maitain this list in Zookeeper for each peer. e.g. peer-1 -> table1[fam1, fam2], table2[fam1] peer-2 -> table1[fam1] So the related properties in table is use-less. right? Hope I understand you correctly. But this will make things more difficult. Change in ReplicationSink seems simple, but master cluster will send some unneccessary edits to peers. > TableNotFoundException thrown in peer cluster will incur endless retry for > shipEdits, which in turn block following normal replication > -- > > Key: HBASE-7280 > URL: https://issues.apache.org/jira/browse/HBASE-7280 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.2 >Reporter: Feng Honghua > Fix For: 0.94.4 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > in cluster replication, if the master cluster have 2 tables which have > column-family declared with replication scope = 1, and add a peer cluster > which has only 1 table with the same name as the master cluster, in the > ReplicationSource (thread in master cluster) for this peer, edits (logs) for > both tables will be shipped to the peer, the peer will fail applying the > edits due to TableNotFoundException, and this exception will also be > responsed to the original shipper (ReplicationSource in master cluster), and > the shipper will fall into an endless retry for shipping the failed edits > without proceeding to read the remained(newer) log files and to ship > following edits(maybe the normal, expected edit for the registered table). > the symptom looks like the TableNotFoundException incurs endless retry and > blocking normal table replication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511051#comment-13511051 ] Lars Hofhansl commented on HBASE-7233: -- bq. As I see it then, we'll send a pb Result and then on the wire, it'll be directly followed by an encoded block of KVs. That makes sense. Would need to be extremely careful to still have wire compatibility. I.e. when a new serialization format comes along for the KV block, we cannot just send the new encoding along (even when announced in the header), the other side would not know what to do with it. bq. except when doing secure connection.. there we need to sasl wrap the byte array response That's interesting. Is there no way around this? We could use a GatheringByteChannel and then assemble the response piecemeal. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7247) Assignment performances decreased by 50% because of regionserver.OpenRegionHandler#tickleOpening
[
https://issues.apache.org/jira/browse/HBASE-7247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511034#comment-13511034
]
Andrew Purtell commented on HBASE-7247:
---
{quote}
bq. Stepping back (after looking at code), could we drop the notion that a
master can intercede and assign a region elsewhere because it is proceeding too
slow on a particular region in the name of simplifying the region open handling
interaction? There would be less noise in the logs and less states to deal with.
It would be a huge simplification imho. It's worth trying, I would say. It
actually makes sense to do it now, because once the current trunk code will be
production proven, touching it will be scarier.
{quote}
+1
> Assignment performances decreased by 50% because of
> regionserver.OpenRegionHandler#tickleOpening
>
>
> Key: HBASE-7247
> URL: https://issues.apache.org/jira/browse/HBASE-7247
> Project: HBase
> Issue Type: Improvement
> Components: master, Region Assignment, regionserver
>Affects Versions: 0.96.0
>Reporter: nkeywal
>Assignee: nkeywal
>Priority: Critical
> Fix For: 0.96.0
>
> Attachments: 7247.v1.patch
>
>
> The regionserver.OpenRegionHandler#tickleOpening updates the region znode as
> "Do this so master doesn't timeout this region-in-transition.".
> However, on the usual test, this makes the assignment time of 1500 regions
> goes from 70s to 100s, that is, we're 50% slower because of this.
> More generally, ZooKeper commits to disk all the data update, and this takes
> time. Using it to provide a keep alive seems overkill. At the very list, it
> could be made asynchronous.
> I'm not sure how necessary these updates are required (I need to go deeper in
> the internal, feedback welcome), but it seems very important to optimize
> this... The trival fix would be to make this optional.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511030#comment-13511030 ] Devaraj Das commented on HBASE-7213: I ran tests locally with the patch and they passed. I also ran some manual tests (killing region servers after making sure some edits weren't persisted, etc.). > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-2.6.patch, > 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7288) Add snapshot verification admin tool
[ https://issues.apache.org/jira/browse/HBASE-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7288: -- Summary: Add snapshot verification admin tool (was: Add snapshot verification to tool) > Add snapshot verification admin tool > > > Key: HBASE-7288 > URL: https://issues.apache.org/jira/browse/HBASE-7288 > Project: HBase > Issue Type: Sub-task > Components: Client, master, regionserver, snapshots, Zookeeper >Reporter: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > > Snapshots have a new file layout and there should be some admin command line > tool to verify that they are not corrupt. This could potentially be added to > hbck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7288) Add snapshot verification to tool
Jonathan Hsieh created HBASE-7288: - Summary: Add snapshot verification to tool Key: HBASE-7288 URL: https://issues.apache.org/jira/browse/HBASE-7288 Project: HBase Issue Type: Sub-task Reporter: Jonathan Hsieh Snapshots have a new file layout and there should be some admin command line tool to verify that they are not corrupt. This could potentially be added to hbck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7280) TableNotFoundException thrown in peer cluster will incur endless retry for shipEdits, which in turn block following normal replication
[ https://issues.apache.org/jira/browse/HBASE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13511021#comment-13511021 ] Feng Honghua commented on HBASE-7280: - I can understand the initiative of current design. A master cluster may have multiple tables with REPLICATION_SCOPE=1, but not all peer clusters want to replicate all these tables, current design prevents only replicating selective table(s). In our scenario, I expect peer cluster(sink) can omit the edits for which the table doesn't exist in peer cluster and only apply edits for which the table(s) exist in peer cluster(we really want to replicate). I make a minor change in ReplicationSink.java which just omits edits for non-existing table(s) in peer cluster and the behavior is what we want. Though this change doesn't reduce the needless network bandwidth it's at least doesn't block the normal replication. Seems current replication's per-cluster granularity is a bit coarse-grained for many real-world scenarios. In my opinion adding such as table- or columnfamily- list configuration for peer when adding peer is more reasonable. > TableNotFoundException thrown in peer cluster will incur endless retry for > shipEdits, which in turn block following normal replication > -- > > Key: HBASE-7280 > URL: https://issues.apache.org/jira/browse/HBASE-7280 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.2 >Reporter: Feng Honghua > Fix For: 0.94.4 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > in cluster replication, if the master cluster have 2 tables which have > column-family declared with replication scope = 1, and add a peer cluster > which has only 1 table with the same name as the master cluster, in the > ReplicationSource (thread in master cluster) for this peer, edits (logs) for > both tables will be shipped to the peer, the peer will fail applying the > edits due to TableNotFoundException, and this exception will also be > responsed to the original shipper (ReplicationSource in master cluster), and > the shipper will fall into an endless retry for shipping the failed edits > without proceeding to read the remained(newer) log files and to ship > following edits(maybe the normal, expected edit for the registered table). > the symptom looks like the TableNotFoundException incurs endless retry and > blocking normal table replication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-7180 started by Lars Hofhansl. > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7180-0.94-SKETCH.txt, 7180-0.94-v1.txt, > 7180-0.94-v2.txt, 7180-0.94-v3.txt, 7180-0.96-v1.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7180: - Status: Patch Available (was: In Progress) > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7180-0.94-SKETCH.txt, 7180-0.94-v1.txt, > 7180-0.94-v2.txt, 7180-0.94-v3.txt, 7180-0.96-v1.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7180: - Fix Version/s: 0.94.4 0.96.0 Assignee: Lars Hofhansl > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7180-0.94-SKETCH.txt, 7180-0.94-v1.txt, > 7180-0.94-v2.txt, 7180-0.94-v3.txt, 7180-0.96-v1.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7180: - Attachment: 7180-0.94-v3.txt Just changes a comment slightly > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 7180-0.94-SKETCH.txt, 7180-0.94-v1.txt, > 7180-0.94-v2.txt, 7180-0.94-v3.txt, 7180-0.96-v1.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7180: - Attachment: 7180-0.96-v1.txt And a 0.96 version > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 7180-0.94-SKETCH.txt, 7180-0.94-v1.txt, > 7180-0.94-v2.txt, 7180-0.94-v3.txt, 7180-0.96-v1.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7055) port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice (not configurable by cf or dynamically)
[ https://issues.apache.org/jira/browse/HBASE-7055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7055: Attachment: HBASE-7055-v2.patch Added javadoc, based on the doc. > port HBASE-6371 tier-based compaction from 0.89-fb to trunk - first slice > (not configurable by cf or dynamically) > - > > Key: HBASE-7055 > URL: https://issues.apache.org/jira/browse/HBASE-7055 > Project: HBase > Issue Type: Task > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 0.96.0 > > Attachments: HBASE-6371-squashed.patch, HBASE-6371-v2-squashed.patch, > HBASE-6371-v3-refactor-only-squashed.patch, > HBASE-6371-v4-refactor-only-squashed.patch, > HBASE-6371-v5-refactor-only-squashed.patch, HBASE-7055-v0.patch, > HBASE-7055-v1.patch, HBASE-7055-v2.patch > > > There's divergence in the code :( > See HBASE-6371 for details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7280) TableNotFoundException thrown in peer cluster will incur endless retry for shipEdits, which in turn block following normal replication
[ https://issues.apache.org/jira/browse/HBASE-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510993#comment-13510993 ] Jieshan Bean commented on HBASE-7280: - Yes, this is the expected behavior. In current implementation, backup cluster should create the tables by itself. > TableNotFoundException thrown in peer cluster will incur endless retry for > shipEdits, which in turn block following normal replication > -- > > Key: HBASE-7280 > URL: https://issues.apache.org/jira/browse/HBASE-7280 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 0.94.2 >Reporter: Feng Honghua > Fix For: 0.94.4 > > Original Estimate: 0.5h > Remaining Estimate: 0.5h > > in cluster replication, if the master cluster have 2 tables which have > column-family declared with replication scope = 1, and add a peer cluster > which has only 1 table with the same name as the master cluster, in the > ReplicationSource (thread in master cluster) for this peer, edits (logs) for > both tables will be shipped to the peer, the peer will fail applying the > edits due to TableNotFoundException, and this exception will also be > responsed to the original shipper (ReplicationSource in master cluster), and > the shipper will fall into an endless retry for shipping the failed edits > without proceeding to read the remained(newer) log files and to ship > following edits(maybe the normal, expected edit for the registered table). > the symptom looks like the TableNotFoundException incurs endless retry and > blocking normal table replication -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7268: Attachment: HBASE-7268-v0.patch > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7268-v0.patch, HBASE-7268-v0.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-7213: --- Attachment: 7213-2.6.patch bq. One more spin and I'd say this patch is ready to go in. Good on you DD. Sweet... Here is an updated patch with the filename extn change, and with passing PathFilter in splitLogDistributed as per your last comments. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-2.6.patch, > 7213-in-progress.2.2.patch, 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7236) add per-table/per-cf compaction configuration via metadata
[ https://issues.apache.org/jira/browse/HBASE-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510973#comment-13510973 ] Sergey Shelukhin commented on HBASE-7236: - Hi; are there any responses/objections? I'd like to go ahead with making commit-ready patch asap :) > add per-table/per-cf compaction configuration via metadata > -- > > Key: HBASE-7236 > URL: https://issues.apache.org/jira/browse/HBASE-7236 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HBASE-7236-PROTOTYPE.patch, HBASE-7236-PROTOTYPE.patch, > HBASE-7236-PROTOTYPE-v1.patch > > > Regardless of the compaction policy, it makes sense to have separate > configuration for compactions for different tables and column families, as > their access patterns and workloads can be different. In particular, for > tiered compactions that are being ported from 0.89-fb branch it is necessary > to have, to use it properly. > We might want to add support for compaction configuration via metadata on > table/cf. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7279) Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7279: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to 0.94 and 0.96, thanks for the review > Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher > - > > Key: HBASE-7279 > URL: https://issues.apache.org/jira/browse/HBASE-7279 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7279-0.94.txt, 7279-0.94-v2.txt, 7279-0.94-v3.txt, > 7279-0.94-v4.txt, 7279-0.96-v1.txt, 7279-0.96-v2.txt, 7279-0.96-v3.txt > > > Did some profiling again. > I we can gain some performance [1] when passing buffer, rowoffset, and > rowlength instead of making a copy of the row key. > That way we can also remove the row key caching (and this patch also removes > the timestamps caching). Considering the sheer number in which we create KVs, > every byte save is good. > [1] (15-20% when data is in the block cache we setup a Filter such that only > a single row is returned to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7287) HBase inconsistencies when merge failing on "Files have same sequenceid"
[ https://issues.apache.org/jira/browse/HBASE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7287: --- Status: Patch Available (was: Open) > HBase inconsistencies when merge failing on "Files have same sequenceid" > > > Key: HBASE-7287 > URL: https://issues.apache.org/jira/browse/HBASE-7287 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7287.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When regions merge is failing because the files have the same sequenceID, the > expected region is still created even if it's not used, which leaves the > system with inconsistencies. The new region creation should be moved after > the sequenceID test to avoid this issue, until we find a way to merge regions > with the same sequenceID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7287) HBase inconsistencies when merge failing on "Files have same sequenceid"
[ https://issues.apache.org/jira/browse/HBASE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7287: --- Attachment: HBASE-7287.patch Patch attached. > HBase inconsistencies when merge failing on "Files have same sequenceid" > > > Key: HBASE-7287 > URL: https://issues.apache.org/jira/browse/HBASE-7287 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Attachments: HBASE-7287.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When regions merge is failing because the files have the same sequenceID, the > expected region is still created even if it's not used, which leaves the > system with inconsistencies. The new region creation should be moved after > the sequenceID test to avoid this issue, until we find a way to merge regions > with the same sequenceID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7287) HBase inconsistencies when merge failing on "Files have same sequenceid"
[ https://issues.apache.org/jira/browse/HBASE-7287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510965#comment-13510965 ] Jean-Marc Spaggiari commented on HBASE-7287: Finally, it's not possible to move the newRegionDir creation after the sequenceID check since some sequenceID for some Familly files might be correct and will need to create the new column family dirs (makeColumnFamilyDirs). So instead of moving the directory creation, here is a patch to remove the created directory if the merge is failing, to avoid system inconsistencies. > HBase inconsistencies when merge failing on "Files have same sequenceid" > > > Key: HBASE-7287 > URL: https://issues.apache.org/jira/browse/HBASE-7287 > Project: HBase > Issue Type: Bug >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari > Original Estimate: 1h > Remaining Estimate: 1h > > When regions merge is failing because the files have the same sequenceID, the > expected region is still created even if it's not used, which leaves the > system with inconsistencies. The new region creation should be moved after > the sequenceID test to avoid this issue, until we find a way to merge regions > with the same sequenceID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7279) Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510963#comment-13510963 ] Lars Hofhansl commented on HBASE-7279: -- Ran all tests in trunk with 7279-0.96-v3 and they all pass. Going to commit. > Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher > - > > Key: HBASE-7279 > URL: https://issues.apache.org/jira/browse/HBASE-7279 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7279-0.94.txt, 7279-0.94-v2.txt, 7279-0.94-v3.txt, > 7279-0.94-v4.txt, 7279-0.96-v1.txt, 7279-0.96-v2.txt, 7279-0.96-v3.txt > > > Did some profiling again. > I we can gain some performance [1] when passing buffer, rowoffset, and > rowlength instead of making a copy of the row key. > That way we can also remove the row key caching (and this patch also removes > the timestamps caching). Considering the sheer number in which we create KVs, > every byte save is good. > [1] (15-20% when data is in the block cache we setup a Filter such that only > a single row is returned to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7268: Status: Patch Available (was: Open) > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7268-v0.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HBASE-7268: Attachment: HBASE-7268-v0.patch Apparently meta TS is set. Using it. Also made some logging more helpful. > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > Attachments: HBASE-7268-v0.patch > > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5258) Move coprocessors set out of RegionLoad
[ https://issues.apache.org/jira/browse/HBASE-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510958#comment-13510958 ] Sergey Shelukhin commented on HBASE-5258: - Hi. Is there any objections to removing them for now? I don't see getting coprocessors from RegionLoad actually being used anywhere. > Move coprocessors set out of RegionLoad > --- > > Key: HBASE-5258 > URL: https://issues.apache.org/jira/browse/HBASE-5258 > Project: HBase > Issue Type: Task >Reporter: Ted Yu >Priority: Critical > > When I worked on HBASE-5256, I revisited the code related to Ser/De of > coprocessors set in RegionLoad. > I think the rationale for embedding coprocessors set is for maximum > flexibility where each region can load different coprocessors. > This flexibility is causing extra cost in the region server to Master > communication and increasing the footprint of Master heap. > Would HServerLoad be a better place for this set ? > If required, region server should calculate disparity of loaded coprocessors > among regions and send report through HServerLoad -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7284) Rename rest server from Main to RestServer
[ https://issues.apache.org/jira/browse/HBASE-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510955#comment-13510955 ] Andrew Purtell commented on HBASE-7284: --- Sounds good to me. > Rename rest server from Main to RestServer > -- > > Key: HBASE-7284 > URL: https://issues.apache.org/jira/browse/HBASE-7284 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Priority: Trivial > > With jps, the rest server is shown as Main because the class is Main. Is > there any special reason for it to be called Main? If not, I suggest we > change it to RestServer instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7284) Rename rest server from Main to RestServer
[ https://issues.apache.org/jira/browse/HBASE-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510947#comment-13510947 ] Jimmy Xiang commented on HBASE-7284: Is RESTServer a better one? > Rename rest server from Main to RestServer > -- > > Key: HBASE-7284 > URL: https://issues.apache.org/jira/browse/HBASE-7284 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Priority: Trivial > > With jps, the rest server is shown as Main because the class is Main. Is > there any special reason for it to be called Main? If not, I suggest we > change it to RestServer instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7284) Rename rest server from Main to RestServer
[ https://issues.apache.org/jira/browse/HBASE-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510941#comment-13510941 ] Andrew Purtell commented on HBASE-7284: --- +1 The choice of name for the main class was just not so imaginative. > Rename rest server from Main to RestServer > -- > > Key: HBASE-7284 > URL: https://issues.apache.org/jira/browse/HBASE-7284 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Priority: Trivial > > With jps, the rest server is shown as Main because the class is Main. Is > there any special reason for it to be called Main? If not, I suggest we > change it to RestServer instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5991) Introduce sequential ZNode based read/write locks
[ https://issues.apache.org/jira/browse/HBASE-5991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510937#comment-13510937 ] Enis Soztutar commented on HBASE-5991: -- @Alex thats for the pointers and the feedback. jcommons seems interesting, but one motivator for curator was that it already has reentrant read/write lock, barriers (for hbase snapshots), and queues (for log splitting?). I did not see those recipes in jcommons. With your patch already having locks, John's barrier implementation, and the rest of the stuff, there is definitely less reason to switch, but agreed with Stack in that this is the perfect time to switch, if we chose to :) bq. One approach could be to just implement the interfaces with curator and then run this through the unit tests. Agreed. That was my intention as well. > Introduce sequential ZNode based read/write locks > -- > > Key: HBASE-5991 > URL: https://issues.apache.org/jira/browse/HBASE-5991 > Project: HBase > Issue Type: Improvement >Reporter: Alex Feinberg >Assignee: Alex Feinberg > Fix For: 0.89-fb > > > This is a continuation of HBASE-5494: > Currently table-level write locks have been implemented using non-sequential > ZNodes as part of HBASE-5494 and committed to 89-fb branch. This issue is to > track converting the table-level locks to sequential ZNodes and supporting > read-write locks, as to solve the issue of preventing schema changes during > region splits or merges. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7208) Transition Offline Snapshots to ForeignExceptions
[ https://issues.apache.org/jira/browse/HBASE-7208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7208: -- Summary: Transition Offline Snapshots to ForeignExceptions (was: Transition Offline Snapshots to ExternalExceptions) > Transition Offline Snapshots to ForeignExceptions > - > > Key: HBASE-7208 > URL: https://issues.apache.org/jira/browse/HBASE-7208 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: hbase-6055 >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055 > > Attachments: hbase-7208.v6.patch, pre-hbase-7208.v6.patch > > > This will eliminate the old errorhandling code, and update existing code to > use the ExternalException mechanisms. > I'd like this to be done before attempt merging to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7285) HMaster fails to start with secure Hadoop
[
https://issues.apache.org/jira/browse/HBASE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510926#comment-13510926
]
Gary Helmling commented on HBASE-7285:
--
Thanks for quick review guys!
> HMaster fails to start with secure Hadoop
> -
>
> Key: HBASE-7285
> URL: https://issues.apache.org/jira/browse/HBASE-7285
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.96.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.96.0
>
> Attachments: HBASE-7285.patch
>
>
> In current trunk, HMaster will fail to start with secure Hadoop if the user
> starting the process has not obtained a kerberos TGT. The user starting the
> process should not be required to have a TGT, as the HMaster process self
> logs in using the configured keytab and principal.
> This is due to a log line in the HMaster constructor executing prior to the
> {{User.login()}} step:
> {code}
> LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
> ", hbase.cluster.distributed=" +
> this.conf.getBoolean("hbase.cluster.distributed", false));
> {code}
> Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
> trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7206) Foreign Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7206: -- Attachment: pre-hbase-7206.v4.patch hbase-7206.v4.patch > Foreign Exception framework v2 (simplifies and replaces HBASE-6571) > --- > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch, > hbase-7206.v4.patch, pre-hbase-7206.v4.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7206) Foreign Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-7206: -- Description: This provides a way of sending exceptions from 'external' threads/processes (not the main executing thread) to others that poll cooperatively for external exceptions. Some examples of how this can be used include: having a separate timeout thread that injects an exception when a time limit has elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an exception from an separate process delivered to a local thread. This simplified version is centered around the ExternalException class. Instead of using generics and ErrorListener interfaces, this more straight-forward implementation eliminates many of the builders/factories and generics. was: This provides a way of sending exceptions from 'external' threads/processes (not the main executing thread) to others that poll cooperatively for external exceptions. Some examples of how this can be used include: having a separate timeout thread that injects an exception when a time limit has elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an exception from an separate process delivered to a local thread. This simplified version is centered around the ExternalException class. Instead of using generics and ErrorListener interfaces, this more straight-forward implementation eliminates many of the builders/factories and generics. Summary: Foreign Exception framework v2 (simplifies and replaces HBASE-6571) (was: External Exception framework v2 (simplifies and replaces HBASE-6571)) > Foreign Exception framework v2 (simplifies and replaces HBASE-6571) > --- > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch, > hbase-7206.v4.patch, pre-hbase-7206.v4.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[ https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510920#comment-13510920 ] Enis Soztutar commented on HBASE-7268: -- bq. I'd prefer to have TS in meta but this is a simpler fix for now. Can't we use the TS from the META Puts/Gets to not to invalidate the client cache? > correct local region location cache information can be overwritten w/stale > information from an old server > - > > Key: HBASE-7268 > URL: https://issues.apache.org/jira/browse/HBASE-7268 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.0 >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Minor > > Discovered via HBASE-7250; related to HBASE-5877. > Test is writing from multiple threads. > Server A has region R; client knows that. > R gets moved from A to server B. > B gets killed. > R gets moved by master to server C. > ~15 seconds later, client tries to write to it (on A?). > Multiple client threads report from RegionMoved exception processing logic "R > moved from C to B", even though such transition never happened (neither in > nor before the sequence described below). Not quite sure how the client > learned of the transition to C, I assume it's from meta from some other > thread... > Then, put fails (it may fail due to accumulated errors that are not logged, > which I am investigating... but the bogus cache update is there > nonwithstanding). > I have a patch but not sure if it works, test still fails locally for yet > unknown reason. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7287) HBase inconsistencies when merge failing on "Files have same sequenceid"
Jean-Marc Spaggiari created HBASE-7287: -- Summary: HBase inconsistencies when merge failing on "Files have same sequenceid" Key: HBASE-7287 URL: https://issues.apache.org/jira/browse/HBASE-7287 Project: HBase Issue Type: Bug Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari When regions merge is failing because the files have the same sequenceID, the expected region is still created even if it's not used, which leaves the system with inconsistencies. The new region creation should be moved after the sequenceID test to avoid this issue, until we find a way to merge regions with the same sequenceID. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7284) Rename rest server from Main to RestServer
[ https://issues.apache.org/jira/browse/HBASE-7284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510886#comment-13510886 ] stack commented on HBASE-7284: -- Sounds good Jimmy. > Rename rest server from Main to RestServer > -- > > Key: HBASE-7284 > URL: https://issues.apache.org/jira/browse/HBASE-7284 > Project: HBase > Issue Type: Improvement >Reporter: Jimmy Xiang >Priority: Trivial > > With jps, the rest server is shown as Main because the class is Main. Is > there any special reason for it to be called Main? If not, I suggest we > change it to RestServer instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7285) HMaster fails to start with secure Hadoop
[
https://issues.apache.org/jira/browse/HBASE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-7285.
--
Resolution: Fixed
Fix Version/s: 0.96.0
Assignee: Gary Helmling
Hadoop Flags: Reviewed
Thanks Gary. I think that log line was my bright idea. Committed to trunk
(Problem is not in 0.94).
> HMaster fails to start with secure Hadoop
> -
>
> Key: HBASE-7285
> URL: https://issues.apache.org/jira/browse/HBASE-7285
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.96.0
>Reporter: Gary Helmling
>Assignee: Gary Helmling
> Fix For: 0.96.0
>
> Attachments: HBASE-7285.patch
>
>
> In current trunk, HMaster will fail to start with secure Hadoop if the user
> starting the process has not obtained a kerberos TGT. The user starting the
> process should not be required to have a TGT, as the HMaster process self
> logs in using the configured keytab and principal.
> This is due to a log line in the HMaster constructor executing prior to the
> {{User.login()}} step:
> {code}
> LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
> ", hbase.cluster.distributed=" +
> this.conf.getBoolean("hbase.cluster.distributed", false));
> {code}
> Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
> trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7286) Record time when a dead server is detected
Ted Yu created HBASE-7286: - Summary: Record time when a dead server is detected Key: HBASE-7286 URL: https://issues.apache.org/jira/browse/HBASE-7286 Project: HBase Issue Type: Bug Reporter: Ted Yu DeadServer uses a Set to store ServerName's of dead servers. On master UI, it would help admin observe when each dead server was detected. Normally newly dead server(s) would reveal more about recent cluster health. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7285) HMaster fails to start with secure Hadoop
[
https://issues.apache.org/jira/browse/HBASE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510881#comment-13510881
]
Ted Yu commented on HBASE-7285:
---
+1 on patch.
> HMaster fails to start with secure Hadoop
> -
>
> Key: HBASE-7285
> URL: https://issues.apache.org/jira/browse/HBASE-7285
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.96.0
>Reporter: Gary Helmling
> Attachments: HBASE-7285.patch
>
>
> In current trunk, HMaster will fail to start with secure Hadoop if the user
> starting the process has not obtained a kerberos TGT. The user starting the
> process should not be required to have a TGT, as the HMaster process self
> logs in using the configured keytab and principal.
> This is due to a log line in the HMaster constructor executing prior to the
> {{User.login()}} step:
> {code}
> LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
> ", hbase.cluster.distributed=" +
> this.conf.getBoolean("hbase.cluster.distributed", false));
> {code}
> Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
> trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7285) HMaster fails to start with secure Hadoop
[
https://issues.apache.org/jira/browse/HBASE-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Helmling updated HBASE-7285:
-
Attachment: HBASE-7285.patch
Patch moving the log line to follow the login step.
> HMaster fails to start with secure Hadoop
> -
>
> Key: HBASE-7285
> URL: https://issues.apache.org/jira/browse/HBASE-7285
> Project: HBase
> Issue Type: Bug
> Components: security
>Affects Versions: 0.96.0
>Reporter: Gary Helmling
> Attachments: HBASE-7285.patch
>
>
> In current trunk, HMaster will fail to start with secure Hadoop if the user
> starting the process has not obtained a kerberos TGT. The user starting the
> process should not be required to have a TGT, as the HMaster process self
> logs in using the configured keytab and principal.
> This is due to a log line in the HMaster constructor executing prior to the
> {{User.login()}} step:
> {code}
> LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
> ", hbase.cluster.distributed=" +
> this.conf.getBoolean("hbase.cluster.distributed", false));
> {code}
> Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
> trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7285) HMaster fails to start with secure Hadoop
Gary Helmling created HBASE-7285:
Summary: HMaster fails to start with secure Hadoop
Key: HBASE-7285
URL: https://issues.apache.org/jira/browse/HBASE-7285
Project: HBase
Issue Type: Bug
Components: security
Affects Versions: 0.96.0
Reporter: Gary Helmling
In current trunk, HMaster will fail to start with secure Hadoop if the user
starting the process has not obtained a kerberos TGT. The user starting the
process should not be required to have a TGT, as the HMaster process self logs
in using the configured keytab and principal.
This is due to a log line in the HMaster constructor executing prior to the
{{User.login()}} step:
{code}
LOG.info("hbase.rootdir=" + FSUtils.getRootDir(this.conf) +
", hbase.cluster.distributed=" +
this.conf.getBoolean("hbase.cluster.distributed", false));
{code}
Here the FSUtils.getRootDir() winds up hitting the NameNode. The fix is
trivial, moving the log line to follow {{User.login()}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7284) Rename rest server from Main to RestServer
Jimmy Xiang created HBASE-7284: -- Summary: Rename rest server from Main to RestServer Key: HBASE-7284 URL: https://issues.apache.org/jira/browse/HBASE-7284 Project: HBase Issue Type: Improvement Reporter: Jimmy Xiang Priority: Trivial With jps, the rest server is shown as Main because the class is Main. Is there any special reason for it to be called Main? If not, I suggest we change it to RestServer instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[ https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510843#comment-13510843 ] stack commented on HBASE-7213: -- bq. So I am wondering whether I should just have ".meta" as the suffix for the meta log files, and be done with it (as I had it in the previous patch). Yes. The '.hlog' is extraneous (I thought all log files had .log suffix -- sorry if I mislead) One more spin and I'd say this patch is ready to go in. Good on you DD. > Have HLog files for .META. edits only > - > > Key: HBASE-7213 > URL: https://issues.apache.org/jira/browse/HBASE-7213 > Project: HBase > Issue Type: Improvement > Components: master, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 7213-2.4.patch, 7213-in-progress.2.2.patch, > 7213-in-progress.2.patch, 7213-in-progress.patch > > > Over on HBASE-6774, there is a discussion on separating out the edits for > .META. regions from the other regions' edits w.r.t where the edits are > written. This jira is to track an implementation of that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510838#comment-13510838 ] stack commented on HBASE-7233: -- As I see it then, we'll send a pb Result and then on the wire, it'll be directly followed by an encoded block of KVs. The Result will describe the block that is coming immediately after. Need to do same for Mutation sending in the data. Hopefully, can doctor the rpc so I can get better access to the channel. Currently we are composing the response in a bytebuffer that we give to a WritableByteChannel (this is after pb has done similar when we build the messages). The composing of the response in a bytebuffer is a known temporary stopgap while moving to pb but we'll need to undo it before we ship (except when doing secure connection.. there we need to sasl wrap the byte array response). Let me finish the baseline case where we do pure pb throughout. Then will have a go at trying to send a follow-along encoded block. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7213) Have HLog files for .META. edits only
[
https://issues.apache.org/jira/browse/HBASE-7213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510829#comment-13510829
]
Devaraj Das commented on HBASE-7213:
bq. Do other logs have a '.log' suffix? You add .hlog with the below?
Currently (in the trunk codebase), the log files don't have extension like .log
or something (the files look like
192.168.1.8%2C60020%2C1354695930505.1354695931168). They are in directory
called ".logs/". So I am wondering whether I should just have
".meta" as the suffix for the meta log files, and be done with it (as I had it
in the previous patch). In the current patch, the meta logs have .meta.hlog as
the extension. Or, would it make sense to change the log files to have ".hlog"
extension as well? Please let me know..
bq. The SSH changes look good. There is a hole though at the moment around ROOT
handling? We need to wait on ROOT removal before this can go in, right? (The
RootServerShutdownHandler would goor should RSSH be calling MSSH when it is
done w/ its process?)
It can be done either way in theory (HBASE-3171 first or this first). On the
hole, yes, if this patch goes in first, and if there is a case where the same
RS hosts both the root and meta regions, and that RS goes down, and the root's
edits weren't already persisted, there will be a problem. But given that the
root doesn't have edits other than the one row to do with .META., it is
unlikely that we will run into the hole in practice. But am okay to wait till
HBASE-3171 goes in (and that's a better way to line the commits, but would like
to close on on the other comments).
bq. This below is given to an executor? Should have a better name than handler?
bq. + private UncaughtExceptionHandler handler;
Will take a look and update.
bq. Will we always create the metahlog just because getMetaHLog was called?
Even if we are NOT carrying .META.:
No. It'll be created on the first call to getMetaWAL (and getMetaWAL gets
indirectly called only when we are asked to open the meta region in
OpenRegionHandler)
bq. If .META. moves away from a RS and then comes back, we'll just use the
already made meta log roller, etc.?
Yes.
bq. Replication will skip this .META. log?
Yes. The MetaLogRoller doesn't "register" with the Replication folks when
instantiated.
bq. It is intentional that MetaServices is still in the patch?
Mistake.. Will update.
bq. In SplitLogManager, we have added splitMetaLogDistributed. A more generic
method might have been splitLogDistributed that took a file path filter
instead...
Hmm.. Let me see..
bq. There is a define for .META. HRegionInfo in
HRegionInfo#FIRST_META_REGIONINFO so you don't have to make it each time, FYI.
I had seen this one but had forgotten to update code to use it.. will update.
bq. Don't have to deprecate the below I'd say. This is for 0.96 the singularity
and this is on a class with annotation @InterfaceAudience.Private
bq. /** @return the HLog */
bq. + @Deprecated
bq. public HLog getWAL();
Okay..
{quote}
Here
return new Path(dir, prefix + "." + filenum);
+ String child = prefix + "." + filenum;
+ return new Path(dir, child);
the 'normal' WALs do not seem to pick up the suffix in here. Is it possible
that the '.log' is appended elsewhere? Are your .META. logs getting double
'.log'? Just wondering.
{quote}
No.. as I said before, the log files don't have the string extensions.. So what
do you think - should I just remove the .hlog from the .meta.hlog extension
that I did for meta hlogs? Or, leave .meta.hlog as it is and add .hlog to the
non-meta hlog files, or, leave the non-meta hlog names as they are now?
> Have HLog files for .META. edits only
> -
>
> Key: HBASE-7213
> URL: https://issues.apache.org/jira/browse/HBASE-7213
> Project: HBase
> Issue Type: Improvement
> Components: master, regionserver
>Reporter: Devaraj Das
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 7213-2.4.patch, 7213-in-progress.2.2.patch,
> 7213-in-progress.2.patch, 7213-in-progress.patch
>
>
> Over on HBASE-6774, there is a discussion on separating out the edits for
> .META. regions from the other regions' edits w.r.t where the edits are
> written. This jira is to track an implementation of that.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7233) Serializing KeyValues when passing them over RPC
[ https://issues.apache.org/jira/browse/HBASE-7233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510802#comment-13510802 ] Matt Corgan commented on HBASE-7233: Most of the ProtocolBuffer uses are not performance critical and PB gives great flexibility and a well-known paradigm, but sending big chunks of Cells over the wire as fast as possible in a long scan is worth a special case i'd say. Using the DataBlockEncoding stuff might consume roughly the same cpu as PB encoding on the server, but will save a ton of network bandwith for many tables and would be much easier for the client to decode. > Serializing KeyValues when passing them over RPC > > > Key: HBASE-7233 > URL: https://issues.apache.org/jira/browse/HBASE-7233 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack >Priority: Blocker > Attachments: 7233.txt, 7233-v2.txt > > > Undo KeyValue being a Writable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510801#comment-13510801 ] stack commented on HBASE-7206: -- bq. I'm just going to use Foreign instead of the longer ForeignThread in most places I was just going to say... having 'Thread' in the name is a little redundant on the day after. +1 on talking up the peer aspect of partipants. Once I got that part, the proposal made way more sense. Thanks. > External Exception framework v2 (simplifies and replaces HBASE-6571) > > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7268) correct local region location cache information can be overwritten w/stale information from an old server
[
https://issues.apache.org/jira/browse/HBASE-7268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510798#comment-13510798
]
Sergey Shelukhin commented on HBASE-7268:
-
ok, I got repro... will attach patch after cleanup of debug logging/etc.
I'd prefer to have TS in meta but this is a simpler fix for now.
The logging with patch looks like this:
{code}
2012-12-05 12:06:08,285 DEBUG [Thread-521] util.ChaosMonkey$Action(203):
Removing 13 regions from 10.10.11.17,53406,1354737903944
...
2012-12-05 12:06:08,765 INFO [am-zkevent-worker-pool-2-thread-2]
master.RegionStates(249): Region {NAME =>
'IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.',
STARTKEY => '7ff8', ENDKEY => '8cc4',
ENCODED => 89483778064d05b1f2e1c0d20bcabc16,} transitioned from
{IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.
state=PENDING_OPEN, ts=1354737968742, server=10.10.11.17,53407,1354737903960}
to
{IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.
state=OPENING, ts=1354737968765, server=10.10.11.17,53407,1354737903960}
...
2012-12-05 12:06:10,549 INFO [Thread-521] util.ChaosMonkey$Action(179):
Killing region server:10.10.11.17,53407,1354737903960
...
2012-12-05 12:06:39,233 INFO [am-zkevent-worker-pool-2-thread-2]
master.RegionStates(249): Region {NAME =>
'IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.',
STARTKEY => '7ff8', ENDKEY => '8cc4',
ENCODED => 89483778064d05b1f2e1c0d20bcabc16,} transitioned from
{IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.
state=OPENING, ts=1354737999228, server=10.10.11.17,53404,1354737903902} to
{IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.
state=OPEN, ts=1354737999232, server=10.10.11.17,53404,1354737903902}
...
2012-12-05 12:06:40,276 INFO [HBaseWriterThread_4]
client.HConnectionManager$HConnectionImplementation(1776): Received an error
from 10.10.11.17:53407 for region
IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.;
not removing 10.10.11.17:53404 from cache.
...
2012-12-05 12:06:40,381 INFO [HBaseWriterThread_15]
client.HConnectionManager$HConnectionImplementation(1809): Region
IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.
moved to 10.10.11.17:53407 according to 10.10.11.17:53406
2012-12-05 12:06:40,381 DEBUG [HBaseWriterThread_15]
client.HConnectionManager$HConnectionImplementation(1342): Ignoring stale
location update for
IntegrationTestRebalanceAndKillServersTargeted,7ff8,1354737916774.89483778064d05b1f2e1c0d20bcabc16.:
10.10.11.17:53407 at 1354737968725; local 10.10.11.17:53404 at 1354738000265
{code}
> correct local region location cache information can be overwritten w/stale
> information from an old server
> -
>
> Key: HBASE-7268
> URL: https://issues.apache.org/jira/browse/HBASE-7268
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Minor
>
> Discovered via HBASE-7250; related to HBASE-5877.
> Test is writing from multiple threads.
> Server A has region R; client knows that.
> R gets moved from A to server B.
> B gets killed.
> R gets moved by master to server C.
> ~15 seconds later, client tries to write to it (on A?).
> Multiple client threads report from RegionMoved exception processing logic "R
> moved from C to B", even though such transition never happened (neither in
> nor before the sequence described below). Not quite sure how the client
> learned of the transition to C, I assume it's from meta from some other
> thread...
> Then, put fails (it may fail due to accumulated errors that are not logged,
> which I am investigating... but the bogus cache update is there
> nonwithstanding).
> I have a patch but not sure if it works, test still fails locally for yet
> unknown reason.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7269) Testing in place does not work if not building with default profile
[
https://issues.apache.org/jira/browse/HBASE-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510797#comment-13510797
]
stack commented on HBASE-7269:
--
[~eclark] did the above hbase-it trickery... hey Elliott!
> Testing in place does not work if not building with default profile
> ---
>
> Key: HBASE-7269
> URL: https://issues.apache.org/jira/browse/HBASE-7269
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Andrew Purtell
> Attachments: 7269.v1.patch
>
>
> If I build with the Hadoop 2 profile, for example:
> {{mvn -Dhadoop.profile=2.0 -Dhadoop.version=2.0.3-SNAPSHOT}}
> and then try to run daemons like so:
> {{./bin/hbase master start}}
> this will fail, because the launch script will invoke Maven to build the
> cached classpath selecting whatever is the default profile, currently Hadoop
> 1. Startup will actually get pretty far, until:
> {noformat}
> 12/12/04 11:42:13 WARN regionserver.HRegionServer: error telling master we
> are up
> com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:189)
> at $Proxy10.regionServerStartup(Unknown Source)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1844)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:843)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:317)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1415)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1278)
> at
> org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:177)
> ... 4 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.net.SocketInputWrapper
> {noformat}
> There doesn't appear to be a way to supply additional arguments to the launch
> script for directing Maven which profile(s) to activate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7269) Testing in place does not work if not building with default profile
[
https://issues.apache.org/jira/browse/HBASE-7269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510793#comment-13510793
]
Enis Soztutar commented on HBASE-7269:
--
N, do you know what this was in bin/hbase in the first place? Does not make
sense to me.
{code} we need hbase-it to always be the last module run. since it has the
largest classpath
{code}
Thinking about it, the patch makes sense. We should not be doing mvn invocation
from bin/hbase.
> Testing in place does not work if not building with default profile
> ---
>
> Key: HBASE-7269
> URL: https://issues.apache.org/jira/browse/HBASE-7269
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.96.0
>Reporter: Andrew Purtell
> Attachments: 7269.v1.patch
>
>
> If I build with the Hadoop 2 profile, for example:
> {{mvn -Dhadoop.profile=2.0 -Dhadoop.version=2.0.3-SNAPSHOT}}
> and then try to run daemons like so:
> {{./bin/hbase master start}}
> this will fail, because the launch script will invoke Maven to build the
> cached classpath selecting whatever is the default profile, currently Hadoop
> 1. Startup will actually get pretty far, until:
> {noformat}
> 12/12/04 11:42:13 WARN regionserver.HRegionServer: error telling master we
> are up
> com.google.protobuf.ServiceException: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:189)
> at $Proxy10.regionServerStartup(Unknown Source)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:1844)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:843)
> at java.lang.Thread.run(Thread.java:679)
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/hadoop/net/SocketInputWrapper
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.createConnection(HBaseClient.java:317)
> at
> org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:1415)
> at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1278)
> at
> org.apache.hadoop.hbase.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:177)
> ... 4 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.net.SocketInputWrapper
> {noformat}
> There doesn't appear to be a way to supply additional arguments to the launch
> script for directing Maven which profile(s) to activate.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-1212) merge tool expects regions all have different sequence ids
[ https://issues.apache.org/jira/browse/HBASE-1212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-1212: --- Attachment: failure.log Attached logs from a merge failure. > merge tool expects regions all have different sequence ids > -- > > Key: HBASE-1212 > URL: https://issues.apache.org/jira/browse/HBASE-1212 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: failure.log > > > Currently merging two regions, the merge tool will compare their sequence > ids. If same, it will decrement one. It needs to do this because on region > open, files are keyed by their sequenceid; if two the same, one will erase > the other. > Well, with the move to the aggregating hfile format, the sequenceid is > written when the file is created and its no longer written into an aside file > but as metadata on to the end of the file. Changing the sequenceid is no > longer an option. > This issue is about figuring a solution for the rare case where two store > files have same sequence id AND we want to merge the two regions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7279) Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher
[ https://issues.apache.org/jira/browse/HBASE-7279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-7279: - Attachment: 7279-0.96-v3.txt A reset of the rowcounter (for intra row pagination new to 0.96) got lost in the patch. Found that through running the tests locally. > Avoid copying the rowkey in RegionScanner, StoreScanner, and ScanQueryMatcher > - > > Key: HBASE-7279 > URL: https://issues.apache.org/jira/browse/HBASE-7279 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.4 > > Attachments: 7279-0.94.txt, 7279-0.94-v2.txt, 7279-0.94-v3.txt, > 7279-0.94-v4.txt, 7279-0.96-v1.txt, 7279-0.96-v2.txt, 7279-0.96-v3.txt > > > Did some profiling again. > I we can gain some performance [1] when passing buffer, rowoffset, and > rowlength instead of making a copy of the row key. > That way we can also remove the row key caching (and this patch also removes > the timestamps caching). Considering the sheer number in which we create KVs, > every byte save is good. > [1] (15-20% when data is in the block cache we setup a Filter such that only > a single row is returned to the client). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510787#comment-13510787 ] Ted Yu commented on HBASE-7283: --- Jenkins is still down. Can you provide test results based on the combined patch ? Thanks > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: HBASE-6564-0.94.patch, HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510781#comment-13510781 ] Jonathan Hsieh commented on HBASE-7206: --- I'm just going to use Foreign instead of the longer ForeignThread in most place. > External Exception framework v2 (simplifies and replaces HBASE-6571) > > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7206) External Exception framework v2 (simplifies and replaces HBASE-6571)
[ https://issues.apache.org/jira/browse/HBASE-7206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510781#comment-13510781 ] Jonathan Hsieh edited comment on HBASE-7206 at 12/5/12 9:19 PM: I'm just going to use Foreign instead of the longer ForeignThread in most places was (Author: jmhsieh): I'm just going to use Foreign instead of the longer ForeignThread in most place. > External Exception framework v2 (simplifies and replaces HBASE-6571) > > > Key: HBASE-7206 > URL: https://issues.apache.org/jira/browse/HBASE-7206 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Fix For: hbase-6055, 0.96.0 > > Attachments: 121122-external-exceptions.pdf, hbase-7206.v3.patch > > > This provides a way of sending exceptions from 'external' threads/processes > (not the main executing thread) to others that poll cooperatively for > external exceptions. Some examples of how this can be used include: having a > separate timeout thread that injects an exception when a time limit has > elapsed (TimeoutExceptionInjector, was OperationAttemptTimer), or having an > exception from an separate process delivered to a local thread. > This simplified version is centered around the ExternalException class. > Instead of using generics and ErrorListener interfaces, this more > straight-forward implementation eliminates many of the builders/factories and > generics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7283) Backport HBASE-6564 + HBASE-7202 to 0.94
[ https://issues.apache.org/jira/browse/HBASE-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7283: --- Attachment: HBASE-7202-0.94.patch HBASE-6564-0.94.patch > Backport HBASE-6564 + HBASE-7202 to 0.94 > > > Key: HBASE-7283 > URL: https://issues.apache.org/jira/browse/HBASE-7283 > Project: HBase > Issue Type: Task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.94.4 > > Attachments: HBASE-6564-0.94.patch, HBASE-7202-0.94.patch > > > * HBASE-6564: HDFS space is not reclaimed when a column family is deleted > * HBASE-7202: Family Store Files are not archived on admin.deleteColumn() > (the second one is a fix for the first, to use the archiver) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
