[jira] [Commented] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986407#comment-13986407 ] stack commented on HBASE-11092: --- Server is top level Interface. I think ConsensusProvider is looking like it should be w/ implemenations and supporting Interfaces down in the consensus package? Then Server doesn't have to reach down into the consensus package. Factory should be top level too? Otherwise patch lgtm. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.diff, HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. > Since Server interface lives in hbase-client maven module, the following > approach is implemented in this patch: > - hbase-client module has very basic (almost marker) interface > ConsensusProvider to return instance of consensus provider from the Server > - hbase-server module has BaseConsensusProvider which defines the consensus > interfaces > - Implementations shall subclass BaseConsensusProvider > - whoever wants to get ConsensusProvider from raw Server interface on > hbase-server side, has to typecast: (BaseConsensusProvider) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[ https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986405#comment-13986405 ] stack commented on HBASE-10915: --- bq. When I think at high level, ZK is used know mostly to coordinate ops between HM and HRS, so operations which has to be abstracted presumably, mostly involve both HM and HRS side. So may be for now put it all under o.a.h.h.consensus? This is reasonable rationale. Mess around w/ it. If it feels right, lets go w/ it (make a note somewhere that this is what we decided maybe in a class comment or so...) Let me look at hbase-11092. > Decouple region closing (HM and HRS) from ZK > > > Key: HBASE-10915 > URL: https://issues.apache.org/jira/browse/HBASE-10915 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch > > > Decouple region closing from ZK. > Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the > code using (HRegionServer, RSRpcServices etc). > May need small changes in AssignmentManager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986374#comment-13986374
]
stack commented on HBASE-11099:
---
[~jeffreyz] I don't think so. Down in append we do this:
{code}
...
long sequence = this.disruptor.getRingBuffer().next();
try {
RingBufferTruck truck = this.disruptor.getRingBuffer().get(sequence);
FSWALEntry entry =
new FSWALEntry(sequence, logKey, edits, sequenceId, inMemstore, htd,
info);
truck.loadPayload(entry, scope.detach());
} finally {
this.disruptor.getRingBuffer().publish(sequence);
}
...
{code}
So we get a slot on the ring buffer and load it up. When ready to go, we
publish to the ring.
Threads contend here abouts so publishing can be happening in any order (that
could be ok).
(Reading the setAvailable, called when we publish, I can't tell how it works
w/o running some tests; i.e. does publishing make it available for processing
though there are sequences ahead of this one not yet published?. I could do
that.)
The ring buffer sequence number is an internal detail not related to region
sequence id. Wouldn't I have to relate them doing the above (ringbuffer is
regionserver scoped)? Otherwise, I would have to synchronize -- i.e. block --
the disruptor so I could tie the disruptor id getting and the upping of the
region sequence id together? Unless I used the disruptor id as region sequence
id? (would need to check that publish respected disruptor id). Disruptor id is
a long. Say 100k writes a second, I think its 3M years till we roll over
(would have to check -- disruptor might be using some of the higher order bits
as flags).
Also at flush time, don't we want all that could be in the snapshot sync'd
rather than just appended? I know sync is a pretty faint guarantee but it
would be better than our using a seqid of an edit not sync'd? Thinking on it,
this might not be necessay. If the flush succeeds, we probably had a sync come
in in in the meantime. Could do a sync outside of the update lock to be sure.
What you think boss? (thanks for the help here).
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[ https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986366#comment-13986366 ] Mikhail Antonov commented on HBASE-10915: - If the approach in HBASE-11092 looks right I think it'd be better to commit it first, as it's another kind of infrastrusture task (also [~sergey.soldatov] needs that in his work on splitting and merging regions). > Decouple region closing (HM and HRS) from ZK > > > Key: HBASE-10915 > URL: https://issues.apache.org/jira/browse/HBASE-10915 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch > > > Decouple region closing from ZK. > Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the > code using (HRegionServer, RSRpcServices etc). > May need small changes in AssignmentManager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11092: Description: As discussed in comments to HBASE-10915, we need to have a proper way to retrieve instance of consensus provider, and Server interface seems the right one. Since Server interface lives in hbase-client maven module, the following approach is implemented in this patch: - hbase-client module has very basic (almost marker) interface ConsensusProviderto return instance of consensus provider from the Server - hbase-server module has BaseConsensusProvider which defines the consensus interfaces - Implementations shall subclass BaseConsensusProvider - whoever wants to get ConsensusProvider from raw Server interface on hbase-server side, has to typecast: (BaseConsensusProvider) was:As discussed in comments to HBASE-10915, we need to have a proper way to retrieve instance of consensus provider, and Server interface seems the right one. > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.diff, HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. > Since Server interface lives in hbase-client maven module, the following > approach is implemented in this patch: > - hbase-client module has very basic (almost marker) interface > ConsensusProviderto return instance of consensus provider from the Server > - hbase-server module has BaseConsensusProvider which defines the consensus > interfaces > - Implementations shall subclass BaseConsensusProvider > - whoever wants to get ConsensusProvider from raw Server interface on > hbase-server side, has to typecast: (BaseConsensusProvider) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11092: Description: As discussed in comments to HBASE-10915, we need to have a proper way to retrieve instance of consensus provider, and Server interface seems the right one. Since Server interface lives in hbase-client maven module, the following approach is implemented in this patch: - hbase-client module has very basic (almost marker) interface ConsensusProvider to return instance of consensus provider from the Server - hbase-server module has BaseConsensusProvider which defines the consensus interfaces - Implementations shall subclass BaseConsensusProvider - whoever wants to get ConsensusProvider from raw Server interface on hbase-server side, has to typecast: (BaseConsensusProvider) was: As discussed in comments to HBASE-10915, we need to have a proper way to retrieve instance of consensus provider, and Server interface seems the right one. Since Server interface lives in hbase-client maven module, the following approach is implemented in this patch: - hbase-client module has very basic (almost marker) interface ConsensusProviderto return instance of consensus provider from the Server - hbase-server module has BaseConsensusProvider which defines the consensus interfaces - Implementations shall subclass BaseConsensusProvider - whoever wants to get ConsensusProvider from raw Server interface on hbase-server side, has to typecast: (BaseConsensusProvider) > Server interface should have method getConsensusProvider() > -- > > Key: HBASE-11092 > URL: https://issues.apache.org/jira/browse/HBASE-11092 > Project: HBase > Issue Type: Sub-task > Components: Consensus >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Fix For: 0.99.0 > > Attachments: HBASE-11092.diff, HBASE-11092.patch > > > As discussed in comments to HBASE-10915, we need to have a proper way to > retrieve instance of consensus provider, and Server interface seems the right > one. > Since Server interface lives in hbase-client maven module, the following > approach is implemented in this patch: > - hbase-client module has very basic (almost marker) interface > ConsensusProvider to return instance of consensus provider from the Server > - hbase-server module has BaseConsensusProvider which defines the consensus > interfaces > - Implementations shall subclass BaseConsensusProvider > - whoever wants to get ConsensusProvider from raw Server interface on > hbase-server side, has to typecast: (BaseConsensusProvider) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986365#comment-13986365 ] Anoop Sam John commented on HBASE-10835: https://reviews.apache.org/r/20936 Also comment from Matt is fixed in this version. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[ https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986364#comment-13986364 ] Mikhail Antonov commented on HBASE-10915: - Well, it's actually open question about packaging. We talked before that there's is option to put it all in o.a.h.h.consensus.*, because those transactions have HM and HRS side. But, for closing operations specifically, there's pretty much no ZK operations on HM side, so I was not entirely sure what the right package is. When I think at high level, ZK is used know mostly to coordinate ops between HM and HRS, so operations which has to be abstracted presumably, mostly involve both HM and HRS side. So may be for now put it all under o.a.h.h.consensus? Regarding interfaces - could you please take a look at HBASE-11092? (the latest patch is combination of 2 diffs I need to squash and rebase to make it go thru hadoop-qa, but it's working patch. > Decouple region closing (HM and HRS) from ZK > > > Key: HBASE-10915 > URL: https://issues.apache.org/jira/browse/HBASE-10915 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch > > > Decouple region closing from ZK. > Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the > code using (HRegionServer, RSRpcServices etc). > May need small changes in AssignmentManager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10835: --- Attachment: HBASE-10835_V3.patch > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch, > HBASE-10835_V3.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986360#comment-13986360
]
Jeffrey Zhong commented on HBASE-8763:
--
The reason using MutableLong object is that at the very beginning we don't know
the real sync sequence number(due to the late binding) so I use MutableLong
object which keeps a "faked" big sequence number. All new KVs and related of
this transation reference this mvcc mutablelong object. Once after the
corresponding WALEdit is synced(after SyncOrDefer call), we have the real
sequence number and I reset the value of the MutaleLong in one place so all new
KVs in MemStore will see the updated sequence number(because they keep the
reference to this MVCC(MutableLong) instance.
If our WAL Sync doesn't late binding then I don't need to use MutableLong.
[~enis] is suggesting not to use MutableLong while keeping all new KVs and
reset their MVCC values in an extra loop. This may be hard to implement because
our pre & post co-processor copies MVCC values as in the code you pasted
above(where I changed to copy reference)
{code}
newKv.setMvccVersion(kv.getMvccVersionReference());
{code}
My plan is to get all tests pass and then do enhancement/refactoring that you
and [~enis] are suggesting.
> [BRAINSTORM] Combine MVCC and SeqId
> ---
>
> Key: HBASE-8763
> URL: https://issues.apache.org/jira/browse/HBASE-8763
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Reporter: Enis Soztutar
>Assignee: Jeffrey Zhong
>Priority: Critical
> Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch,
> hbase-8763_wip1.patch
>
>
> HBASE-8701 and a lot of recent issues include good discussions about mvcc +
> seqId semantics. It seems that having mvcc and the seqId complicates the
> comparator semantics a lot in regards to flush + WAL replay + compactions +
> delete markers and out of order puts.
> Thinking more about it I don't think we need a MVCC write number which is
> different than the seqId. We can keep the MVCC semantics, read point and
> smallest read points intact, but combine mvcc write number and seqId. This
> will allow cleaner semantics + implementation + smaller data files.
> We can do some brainstorming for 0.98. We still have to verify that this
> would be semantically correct, it should be so by my current understanding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[
https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986357#comment-13986357
]
stack commented on HBASE-10915:
---
You think +++
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/consensus/CloseRegionConsensus.java
should be in a subpackage? Strikes me that it core to HRS operation so so
should be at same level.
This is very nice:
{code}
-int versionOfClosingNode = -1;
-if (request.hasVersionOfClosingNode()) {
- versionOfClosingNode = request.getVersionOfClosingNode();
-}
-boolean zk = request.getTransitionInZK();
final ServerName sn = (request.hasDestinationServer() ?
ProtobufUtil.toServerName(request.getDestinationServer()) : null);
@@ -887,10 +883,11 @@
}
requestCount.increment();
- LOG.info("Close " + encodedRegionName + ", via zk=" + (zk ? "yes" : "no")
-+ ", znode version=" + versionOfClosingNode + ", on " + sn);
+ LOG.info("Close " + encodedRegionName + ", on " + sn);
+ CloseRegionConsensus.CloseRegionDetails crd =
regionServer.getConsensusProvider()
+.getCloseRegionConsensus().parseFromProtoRequest(request);
{code}
Hide that ugly zk stuff!
Good:
- boolean closed = regionServer.closeRegion(encodedRegionName, false, zk,
versionOfClosingNode, sn);
+ boolean closed = regionServer.closeRegion(encodedRegionName, false, crd,
sn);
What can we do about this Mikhail?
+import org.apache.hadoop.hbase.regionserver.consensus.CloseRegionConsensus;
+import org.apache.hadoop.hbase.regionserver.consensus.ZkCloseRegionConsensus;
It seems odd that a o.a.h.h.consensus package would import lower-level RS
stuff. Can Interface only be up at this level but the impementation down in
RS? I suppose you want to have a listing of all the ZK implementations up
here at the top level though they are implemented all over. I suppose we could
live w/ that but given the little trick above you do w/ details, you might have
another up your sleeve to solve this package tangle.
Otherwise, this is close IMO ([~jxiang] You should take a look at this if you
get a chance -- I think you'll like it)
> Decouple region closing (HM and HRS) from ZK
>
>
> Key: HBASE-10915
> URL: https://issues.apache.org/jira/browse/HBASE-10915
> Project: HBase
> Issue Type: Sub-task
> Components: Consensus, Zookeeper
>Affects Versions: 0.99.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
> Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch,
> HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch
>
>
> Decouple region closing from ZK.
> Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the
> code using (HRegionServer, RSRpcServices etc).
> May need small changes in AssignmentManager.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[ https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986354#comment-13986354 ] stack commented on HBASE-8763: -- (Pardon me; am excited about this one so went back to do some more study...) So yeah, why do a MutableLong in KV rather than just a volatile? Probably same in the end... I see you want to get a reference to a KV. You trying to tie a KV to something else in case the mvcc gets updated? Maybe this is it: -newKv.setMvccVersion(kv.getMvccVersion()); +newKv.setMvccVersion(kv.getMvccVersionReference()); We are cloning. The original gets updated later? You want to make sure clone is updated too? Would be cool if we could get rid of this clone one day. > [BRAINSTORM] Combine MVCC and SeqId > --- > > Key: HBASE-8763 > URL: https://issues.apache.org/jira/browse/HBASE-8763 > Project: HBase > Issue Type: Improvement > Components: regionserver >Reporter: Enis Soztutar >Assignee: Jeffrey Zhong >Priority: Critical > Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch, > hbase-8763_wip1.patch > > > HBASE-8701 and a lot of recent issues include good discussions about mvcc + > seqId semantics. It seems that having mvcc and the seqId complicates the > comparator semantics a lot in regards to flush + WAL replay + compactions + > delete markers and out of order puts. > Thinking more about it I don't think we need a MVCC write number which is > different than the seqId. We can keep the MVCC semantics, read point and > smallest read points intact, but combine mvcc write number and seqId. This > will allow cleaner semantics + implementation + smaller data files. > We can do some brainstorming for 0.98. We still have to verify that this > would be semantically correct, it should be so by my current understanding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10771) Primitive type put/get APIs in ByteRange
[
https://issues.apache.org/jira/browse/HBASE-10771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986353#comment-13986353
]
Matt Corgan commented on HBASE-10771:
-
Looks good to me. It looks like the encoding matches the LittleEndian
representation from Bits.java, so a ByteBuffer implementation should be able to
call straight through to its internal putInt method if we deem that's faster
than writing 4 individual bytes to the ByteBuffer.
putInt from java.io.Bits.java{code}
private static byte int3(int x) { return (byte)(x >> 24); }
private static byte int2(int x) { return (byte)(x >> 16); }
private static byte int1(int x) { return (byte)(x >> 8); }
private static byte int0(int x) { return (byte)(x ); }
static void putIntL(ByteBuffer bb, int bi, int x) {
bb._put(bi + 3, int3(x));
bb._put(bi + 2, int2(x));
bb._put(bi + 1, int1(x));
bb._put(bi, int0(x));
}
{code}
putInt from hbase's Bytes.java{code}
public static int putInt(byte[] bytes, int offset, int val) {
if (bytes.length - offset < SIZEOF_INT) {
throw new IllegalArgumentException("Not enough room to put an int at"
+ " offset " + offset + " in a " + bytes.length + " byte array");
}
for(int i= offset + 3; i > offset; i--) {
bytes[i] = (byte) val;
val >>>= 8;
}
bytes[offset] = (byte) val;
return offset + SIZEOF_INT;
}{code}
Might be interesting to add a test to confirm that. You could write to an
array using a ByteRange and then ByteBuffer.wrap(br.getArray()) and read the
values back.
> Primitive type put/get APIs in ByteRange
> -
>
> Key: HBASE-10771
> URL: https://issues.apache.org/jira/browse/HBASE-10771
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10771.patch, HBASE-10771_V2.patch,
> HBASE-10771_V3.patch
>
>
> While doing HBASE-10713 I came across the need to write int/long (and read
> also) from a ByteRange. CellBlocks are backed by ByteRange. So we can add
> such APIs.
> Also as per HBASE-10750 we return a ByteRange from MSLAB and also discussion
> under HBASE-10191 suggest we can have BR backed HFileBlocks etc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10915) Decouple region closing (HM and HRS) from ZK
[ https://issues.apache.org/jira/browse/HBASE-10915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986346#comment-13986346 ] Mikhail Antonov commented on HBASE-10915: - TestVisibilityLabelsWithDistributedLogReplay passes locally for me (timed out by hadoop-qa), but TestRegionObserverInterface is genuine regression, looking into it. > Decouple region closing (HM and HRS) from ZK > > > Key: HBASE-10915 > URL: https://issues.apache.org/jira/browse/HBASE-10915 > Project: HBase > Issue Type: Sub-task > Components: Consensus, Zookeeper >Affects Versions: 0.99.0 >Reporter: Mikhail Antonov >Assignee: Mikhail Antonov > Attachments: HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch, > HBASE-10915.patch, HBASE-10915.patch, HBASE-10915.patch > > > Decouple region closing from ZK. > Includes RS side (CloseRegionHandler), HM side (ClosedRegionHandler) and the > code using (HRegionServer, RSRpcServices etc). > May need small changes in AssignmentManager. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986343#comment-13986343
]
Jeffrey Zhong commented on HBASE-11099:
---
[~saint@gmail.com] Is that possible that we assign the sequence number in
AppendNoSync call without late binding assignment because the order of changes
in the "ring buffer" will be preserved till real WAL sync?
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986336#comment-13986336 ] Anoop Sam John commented on HBASE-10835: Sure I will put it in RB. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[ https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986334#comment-13986334 ] Matt Corgan commented on HBASE-10835: - Anoop, i don't see anything wrong but could get a better feel for it on reviewboard. Let me know if that's possible or if I missed a review request. > DBE encode path improvements > > > Key: HBASE-10835 > URL: https://issues.apache.org/jira/browse/HBASE-10835 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-10835.patch, HBASE-10835_V2.patch > > > Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder. > Encoder again reads kvs one by one from the buffer and encodes and creates a > new buffer. > There is no need to have this model now. Previously we had option of no > encode in disk and encode only in cache. At that time the read buffer from a > HFile block was passed to this and encodes. > So encode cell by cell can be done now. Making this change will need us to > have a NoOp DBE impl which just do the write of a cell as it is with out any > encoding. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10835) DBE encode path improvements
[
https://issues.apache.org/jira/browse/HBASE-10835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986333#comment-13986333
]
Matt Corgan commented on HBASE-10835:
-
{code}
+@InterfaceAudience.Private
+public class EncoderPoolImpl implements EncoderPool {
+
+ private List unusedEncoders = new
ArrayList();
+
+ @Override
+ public PrefixTreeEncoder checkOut(OutputStream outputStream, boolean
includeMvccVersion) {
+PrefixTreeEncoder encoder = null;
+synchronized (this) {
+ if (!unusedEncoders.isEmpty()) {
+encoder = unusedEncoders.remove(0);
+ }
+}
+if (encoder == null) {
+ encoder = new PrefixTreeEncoder(outputStream, includeMvccVersion);
+} else {
+ encoder.reset(outputStream, includeMvccVersion);
+}
+return encoder;
+ }
+
+ @Override
+ public void checkIn(PrefixTreeEncoder encoder) {
+synchronized (this) {
+ this.unusedEncoders.add(encoder);
+}
+ }
+}
{code}
instead of a plain ArrayList for unusedEncoders, maybe you should use a
LinkedBlockingQueue? then you wouldn't have to synchronize the methods
> DBE encode path improvements
>
>
> Key: HBASE-10835
> URL: https://issues.apache.org/jira/browse/HBASE-10835
> Project: HBase
> Issue Type: Improvement
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.99.0
>
> Attachments: HBASE-10835.patch, HBASE-10835_V2.patch
>
>
> Here 1st we write KVs (Cells) into a buffer and then passed to DBE encoder.
> Encoder again reads kvs one by one from the buffer and encodes and creates a
> new buffer.
> There is no need to have this model now. Previously we had option of no
> encode in disk and encode only in cache. At that time the read buffer from a
> HFile block was passed to this and encodes.
> So encode cell by cell can be done now. Making this change will need us to
> have a NoOp DBE impl which just do the write of a cell as it is with out any
> encoding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986320#comment-13986320 ] ramkrishna.s.vasudevan commented on HBASE-10885: Let me change the code to track the visibility expression per ts. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986319#comment-13986319 ] ramkrishna.s.vasudevan commented on HBASE-10885: In the current case only one gets applied. So need to track the vis expression based on ts. One more thing to note here is bq.For deleteFamily case all cells with ts <=103 are candidates for delete Here because deleteFamily with ts = 103 satisfies the condition I think from that cell onwards all the cells less than 103 should also be covered because it is deleteFamily. If it is deleteFamilyVersion then we need to check for exact match only. But here I think once we find the cell with matching visibility expression the ones below that also should get deleted. Which means if a deleteFamily says ts = 103 and assume in this case the ts =101 only matches the vis expression then from 101 on wards all cells less than 101 should be masked. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986315#comment-13986315 ] Anoop Sam John commented on HBASE-10885: Yes as u said , without vis exp the delete will remove all cells except ts=105 With vis exp in delete, the exact version delete should delete cell ts=104 as the vis exps are matching. For deleteFamily case all cells with ts <=103 are candidates for delete but u can see only one cell with ts=103 is having matching vis exp withh that in the delete cell. Cells with ts 102 and 101 are not having matching vis exps and so should not be getting deleted. > Support visibility expressions on Deletes > - > > Key: HBASE-10885 > URL: https://issues.apache.org/jira/browse/HBASE-10885 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan >Priority: Blocker > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch > > > Accumulo can specify visibility expressions for delete markers. During > compaction the cells covered by the tombstone are determined in part by > matching the visibility expression. This is useful for the use case of data > set coalescing, where entries from multiple data sets carrying different > labels are combined into one common large table. Later, a subset of entries > can be conveniently removed using visibility expressions. > Currently doing the same in HBase would only be possible with a custom > coprocessor. Otherwise, a Delete will affect all cells covered by the > tombstone regardless of any visibility expression scoping. This is correct > behavior in that no data spill is possible, but certainly could be > surprising, and is only meant to be transitional. We decided not to support > visibility expressions on Deletes to control the complexity of the initial > implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11051) checkJavacWarnings in test-patch.sh should bail out early if there is compilation error
[
https://issues.apache.org/jira/browse/HBASE-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-11051:
---
Description:
Currently checkJavacWarnings doesn't exit QA script in the presence of
compilation error.
Here is one example:
https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
checkJavacWarnings should do the following so that it is clear what caused the
QA run to fail:
{code}
if [[ $? != 0 ]] ; then
ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
echo "Trunk compilation is broken?
\{code\}$ERR\{code\}"
cleanupAndExit 1
fi
{code}
was:
Currently checkJavacWarnings doesn't exit QA script in the presence of
compilation error.
Here is one example:
https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
checkJavacWarnings should do the following so that it is clear what caused the
QA run to fail:
{code}
if [[ $? != 0 ]] ; then
ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
echo "Trunk compilation is broken?
{code}$ERR{code}"
cleanupAndExit 1
fi
{code}
> checkJavacWarnings in test-patch.sh should bail out early if there is
> compilation error
> ---
>
> Key: HBASE-11051
> URL: https://issues.apache.org/jira/browse/HBASE-11051
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-11051-v1.patch, HBASE-11051.patch
>
>
> Currently checkJavacWarnings doesn't exit QA script in the presence of
> compilation error.
> Here is one example:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
> checkJavacWarnings should do the following so that it is clear what caused
> the QA run to fail:
> {code}
> if [[ $? != 0 ]] ; then
> ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
> echo "Trunk compilation is broken?
> \{code\}$ERR\{code\}"
> cleanupAndExit 1
> fi
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11051) checkJavacWarnings in test-patch.sh should bail out early if there is compilation error
[
https://issues.apache.org/jira/browse/HBASE-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly updated HBASE-11051:
Attachment: HBASE-11051-v1.patch
Commented line removed.
> checkJavacWarnings in test-patch.sh should bail out early if there is
> compilation error
> ---
>
> Key: HBASE-11051
> URL: https://issues.apache.org/jira/browse/HBASE-11051
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-11051-v1.patch, HBASE-11051.patch
>
>
> Currently checkJavacWarnings doesn't exit QA script in the presence of
> compilation error.
> Here is one example:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
> checkJavacWarnings should do the following so that it is clear what caused
> the QA run to fail:
> {code}
> if [[ $? != 0 ]] ; then
> ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
> echo "Trunk compilation is broken?
> {code}$ERR{code}"
> cleanupAndExit 1
> fi
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11051) checkJavacWarnings in test-patch.sh should bail out early if there is compilation error
[
https://issues.apache.org/jira/browse/HBASE-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly updated HBASE-11051:
Status: Patch Available (was: Open)
> checkJavacWarnings in test-patch.sh should bail out early if there is
> compilation error
> ---
>
> Key: HBASE-11051
> URL: https://issues.apache.org/jira/browse/HBASE-11051
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-11051.patch
>
>
> Currently checkJavacWarnings doesn't exit QA script in the presence of
> compilation error.
> Here is one example:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
> checkJavacWarnings should do the following so that it is clear what caused
> the QA run to fail:
> {code}
> if [[ $? != 0 ]] ; then
> ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
> echo "Trunk compilation is broken?
> {code}$ERR{code}"
> cleanupAndExit 1
> fi
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11051) checkJavacWarnings in test-patch.sh should bail out early if there is compilation error
[
https://issues.apache.org/jira/browse/HBASE-11051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gustavo Anatoly updated HBASE-11051:
Attachment: HBASE-11051.patch
> checkJavacWarnings in test-patch.sh should bail out early if there is
> compilation error
> ---
>
> Key: HBASE-11051
> URL: https://issues.apache.org/jira/browse/HBASE-11051
> Project: HBase
> Issue Type: Test
>Reporter: Ted Yu
>Assignee: Gustavo Anatoly
>Priority: Minor
> Attachments: HBASE-11051.patch
>
>
> Currently checkJavacWarnings doesn't exit QA script in the presence of
> compilation error.
> Here is one example:
> https://builds.apache.org/job/PreCommit-HBASE-Build/9360/console .
> checkJavacWarnings should do the following so that it is clear what caused
> the QA run to fail:
> {code}
> if [[ $? != 0 ]] ; then
> ERR=`$GREP -A 5 'Compilation failure' $PATCH_DIR/trunkJavacWarnings.txt`
> echo "Trunk compilation is broken?
> {code}$ERR{code}"
> cleanupAndExit 1
> fi
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10942) support parallel request cancellation for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986301#comment-13986301 ] Sergey Shelukhin commented on HBASE-10942: -- yeah ping? > support parallel request cancellation for multi-get > --- > > Key: HBASE-10942 > URL: https://issues.apache.org/jira/browse/HBASE-10942 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10942.01.patch, HBASE-10942.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[ https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-11104: -- Assignee: Ted Yu > IntegrationTestImportTsv#testRunFromOutputCommitter misses credential > initialization > > > Key: HBASE-11104 > URL: https://issues.apache.org/jira/browse/HBASE-11104 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Assignee: Ted Yu >Priority: Minor > Attachments: 11104-v1.txt > > > IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships > the HBase dependencies. > However, call to TableMapReduceUtil.initCredentials(job) is missing - making > this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10184) [Online Schema Change]: Add additional tests for online schema change
[ https://issues.apache.org/jira/browse/HBASE-10184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10184: --- Fix Version/s: (was: 0.98.2) 0.98.3 > [Online Schema Change]: Add additional tests for online schema change > - > > Key: HBASE-10184 > URL: https://issues.apache.org/jira/browse/HBASE-10184 > Project: HBase > Issue Type: Task > Components: test >Affects Versions: 0.96.1, 0.99.0 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.99.0, 0.98.3 > > Attachments: 10184-4.patch, 10184.addendum, HBASE-10184-trunk.diff > > > There are some gaps in testing for Online Schema Change: > Examples of some tests that should be added: > 1. Splits with online schema change > 2. Merge during online schema change > 3. MR over HBase during online schema change > 4. Bulk Load during online schema change > 5. Online change table owner > 6. Online Replication scope change > 7. Online Bloom Filter change > 8. Snapshots during online schema change (HBASE-10136) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10052) use HDFS advisory caching to avoid caching HFiles that are not going to be read again (because they are being compacted)
[
https://issues.apache.org/jira/browse/HBASE-10052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10052:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> use HDFS advisory caching to avoid caching HFiles that are not going to be
> read again (because they are being compacted)
>
>
> Key: HBASE-10052
> URL: https://issues.apache.org/jira/browse/HBASE-10052
> Project: HBase
> Issue Type: Improvement
>Reporter: Colin Patrick McCabe
>Priority: Minor
> Fix For: 0.99.0, 0.98.3
>
>
> HBase can benefit from doing dropbehind during compaction since compacted
> files are not read again. HDFS advisory caching, introduced in HDFS-4817,
> can help here. The right API here is {{DataInputStream#setDropBehind}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-5554) "hadoop.native.lib" config is deprecated
[
https://issues.apache.org/jira/browse/HBASE-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-5554:
--
Fix Version/s: (was: 0.98.3)
(was: 0.96.3)
(was: 0.99.0)
> "hadoop.native.lib" config is deprecated
>
>
> Key: HBASE-5554
> URL: https://issues.apache.org/jira/browse/HBASE-5554
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Esteban Gutierrez
> Attachments: HBASE-5554.v0.patch
>
>
> When using HBase shell, we see:
> {code}
> 12/03/09 09:06:58 WARN conf.Configuration: hadoop.native.lib is deprecated.
> Instead, use io.native.lib.available
> {code}
> "io.native.lib.available" should be used.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10844) Coprocessor failure during batchmutation leaves the memstore datastructs in an inconsistent state
[ https://issues.apache.org/jira/browse/HBASE-10844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10844: --- Fix Version/s: (was: 0.94.20) (was: 0.96.3) (was: 0.98.2) (was: 0.99.0) > Coprocessor failure during batchmutation leaves the memstore datastructs in > an inconsistent state > - > > Key: HBASE-10844 > URL: https://issues.apache.org/jira/browse/HBASE-10844 > Project: HBase > Issue Type: Bug > Components: regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Attachments: 10844-1.txt > > > Observed this in the testing with Phoenix. The test in Phoenix - > MutableIndexFailureIT deliberately fails the batchmutation call via the > installed coprocessor. But the update is not rolled back. That leaves the > memstore inconsistent. In particular, I observed that getFlushableSize is > updated before the coprocessor was called but the update is not rolled back. > When the region is being closed at some later point, the assert introduced in > HBASE-10514 in the HRegion.doClose() causes the RegionServer to shutdown > abnormally. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10883) Restrict the universe of labels and authorizations
[
https://issues.apache.org/jira/browse/HBASE-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10883:
---
Fix Version/s: (was: 0.98.3)
0.98.2
> Restrict the universe of labels and authorizations
> --
>
> Key: HBASE-10883
> URL: https://issues.apache.org/jira/browse/HBASE-10883
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.1
>Reporter: Andrew Purtell
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10883.patch, HBASE-10883_1.patch,
> HBASE-10883_2.patch, HBASE-10883_3.patch, HBASE-10883_4.patch,
> HBASE-10883_5.patch, HBASE-10883_6.patch, HBASE-10883_7.patch
>
>
> Currently we allow any string as visibility label or request authorization.
> However as seen on HBASE-10878, we accept for authorizations strings that
> would not work if provided as labels in visibility expressions. We should
> throw an exception at least in cases where someone tries to define or use a
> label or authorization including visibility expression operators '&', '|',
> '!', '(', ')'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11048) Support setting custom priority per client RPC
[ https://issues.apache.org/jira/browse/HBASE-11048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986280#comment-13986280 ] Andrew Purtell commented on HBASE-11048: This missed 0.98.2. Perhaps for 0.98.3 next month? > Support setting custom priority per client RPC > -- > > Key: HBASE-11048 > URL: https://issues.apache.org/jira/browse/HBASE-11048 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 0.99.0, 0.98.2 >Reporter: Jesse Yates >Assignee: Jesse Yates > Labels: Phoenix > Fix For: 0.99.0, 0.98.3 > > Attachments: hbase-11048-trunk-v0.patch > > > Servers have the ability to handle custom rpc priority levels, but currently > we are only using it to differentiate META/ROOT updates from replication and > other 'priority' updates (as specified by annotation tags per RS method). > However, some clients need the ability to create custom handlers (e.g. > PHOENIX-938) which can really only be cleanly tied together to requests by > the request priority. The disconnect is in that there is no way for the > client to overwrite the priority per table - the PayloadCarryingRpcController > will always just set priority per ROOT/META and otherwise just use the > generic priority. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-10883) Restrict the universe of labels and authorizations
[
https://issues.apache.org/jira/browse/HBASE-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-10883.
Resolution: Fixed
Hadoop Flags: Reviewed
This was committed.
> Restrict the universe of labels and authorizations
> --
>
> Key: HBASE-10883
> URL: https://issues.apache.org/jira/browse/HBASE-10883
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.1
>Reporter: Andrew Purtell
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-10883.patch, HBASE-10883_1.patch,
> HBASE-10883_2.patch, HBASE-10883_3.patch, HBASE-10883_4.patch,
> HBASE-10883_5.patch, HBASE-10883_6.patch, HBASE-10883_7.patch
>
>
> Currently we allow any string as visibility label or request authorization.
> However as seen on HBASE-10878, we accept for authorizations strings that
> would not work if provided as labels in visibility expressions. We should
> throw an exception at least in cases where someone tries to define or use a
> label or authorization including visibility expression operators '&', '|',
> '!', '(', ')'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11048) Support setting custom priority per client RPC
[ https://issues.apache.org/jira/browse/HBASE-11048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986280#comment-13986280 ] Andrew Purtell commented on HBASE-11048: This missed 0.98.2. Perhaps for 0.98.3 next month? > Support setting custom priority per client RPC > -- > > Key: HBASE-11048 > URL: https://issues.apache.org/jira/browse/HBASE-11048 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 0.99.0, 0.98.2 >Reporter: Jesse Yates >Assignee: Jesse Yates > Labels: Phoenix > Fix For: 0.99.0, 0.98.3 > > Attachments: hbase-11048-trunk-v0.patch > > > Servers have the ability to handle custom rpc priority levels, but currently > we are only using it to differentiate META/ROOT updates from replication and > other 'priority' updates (as specified by annotation tags per RS method). > However, some clients need the ability to create custom handlers (e.g. > PHOENIX-938) which can really only be cleanly tied together to requests by > the request priority. The disconnect is in that there is no way for the > client to overwrite the priority per table - the PayloadCarryingRpcController > will always just set priority per ROOT/META and otherwise just use the > generic priority. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11070) [AccessController] Restore early-out access denial if the user has no access at the table or CF level
[ https://issues.apache.org/jira/browse/HBASE-11070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11070: --- Fix Version/s: (was: 0.98.3) (was: 0.99.0) > [AccessController] Restore early-out access denial if the user has no access > at the table or CF level > - > > Key: HBASE-11070 > URL: https://issues.apache.org/jira/browse/HBASE-11070 > Project: HBase > Issue Type: Task >Reporter: Andrew Purtell >Assignee: Andrew Purtell > > We want to support two different use cases for cell ACLs: > 1. The user can see all cells in a table or CF unless a cell ACL denies access > 2. The user cannot see any cells in a table or CF unless a cell ACL grants > access > For the sake of flexibility we made it a toggle on an operation by operation > basis. However this changed the behavior of the AccessController with respect > to how requests for which a user has no grant at the table or CF level are > handled. Prior to the cell ACL changes if a user had no grant at the table or > CF level, they would see an AccessDeniedException. We can't do that if we > want cell ACLs to provide exceptional access. Subsequent to the cell ACL > changes if a user has no grant at the table or CF level, there is no > exception, they simply won't see any cells except those granting exceptional > access at the cell level. This also brings the AccessController semantics in > line with those of the new VisibilityController. > Feedback on dev@ is this change is a bridge too far for at least three > reasons. First, it is surprising (Enis and Vandana). Second, the audit trail > is affected or missing (Enis). Third, it allows any user on the cluster to > mount targeted queries against all tables looking for timing differences, > that depending on schema design could possibly leak the existence in row keys > of sensitive information, or leak the size of the table (Todd). Although we > can't prevent timing attacks in general we can limit the scope of what a user > can explore by restoring early-out access denial if the user has no access at > the table or CF level. > We can make early-out access denial if the user has no access at the table or > CF level configurable on a per table basis. Setting the default to "false", > with a release note and paragraph in the security guide explaining how to > reintroduce the old behavior, would address the above and not introduce > another surprising change among 0.98 releases. If the consensus is > (presumably a milder) surprise due to this change is fine, then the default > could be "true" -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9445) Snapshots should create column family dirs for empty regions
[
https://issues.apache.org/jira/browse/HBASE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-9445:
--
Fix Version/s: (was: 0.98.2)
0.98.3
> Snapshots should create column family dirs for empty regions
>
>
> Key: HBASE-9445
> URL: https://issues.apache.org/jira/browse/HBASE-9445
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.99.0, 0.98.3
>
> Attachments: hbase-9445_v1.patch, hbase-9445_v2.patch,
> hbase-9445_v3.patch, hbase-9445_v4.patch
>
>
> Currently, taking a snapshot will not create the family directory under a
> region if the family does not have any files in it.
> Subsequent verification fails because of this. There is some logic in the
> SnapshotTestingUtils.confirmSnapshotValid() to deal with empty family
> directories, but I think we should create the family directories regardless
> of whether there are any hfiles referencing them.
> {code}
> 2013-09-05 11:07:21,566 DEBUG [Thread-208] util.FSUtils(1687): |-data/
> 2013-09-05 11:07:21,567 DEBUG [Thread-208] util.FSUtils(1687): |default/
> 2013-09-05 11:07:21,568 DEBUG [Thread-208] util.FSUtils(1687): |---test/
> 2013-09-05 11:07:21,569 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tabledesc/
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1690):
> |-.tableinfo.01
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tmp/
> 2013-09-05 11:07:21,571 DEBUG [Thread-208] util.FSUtils(1687):
> |--accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1690):
> |-.regioninfo
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1687):
> |-fam/
> 2013-09-05 11:07:21,555 DEBUG [Thread-208] util.FSUtils(1687):
> |-.hbase-snapshot/
> 2013-09-05 11:07:21,556 DEBUG [Thread-208] util.FSUtils(1687): |.tmp/
> 2013-09-05 11:07:21,557 DEBUG [Thread-208] util.FSUtils(1687):
> |offlineTableSnapshot/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |---.snapshotinfo
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1687):
> |---.tabledesc/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |--.tableinfo.01
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687): |---.tmp/
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687):
> |---accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,560 DEBUG [Thread-208] util.FSUtils(1690):
> |--.regioninfo
> {code}
> I think this is important for 0.96.0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10536) ImportTsv should fail fast if any of the column family passed to the job is not present in the table
[
https://issues.apache.org/jira/browse/HBASE-10536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10536:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> ImportTsv should fail fast if any of the column family passed to the job is
> not present in the table
>
>
> Key: HBASE-10536
> URL: https://issues.apache.org/jira/browse/HBASE-10536
> Project: HBase
> Issue Type: Bug
> Components: mapreduce
>Affects Versions: 0.98.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
> Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
>
>
> While checking 0.98 rc, running bulkload tools. By mistake passed wrong
> column family to importtsv. LoadIncrementalHfiles failed with following
> exception
> {code}
> Exception in thread "main" java.io.IOException: Unmatched family names found:
> unmatched family names in HFiles to be bulkloaded: [f1]; valid family names
> of table test are: [f]
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:241)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:823)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.main(LoadIncrementalHFiles.java:828)
> {code}
>
> Its better to fail fast if any of the passed column family is not present in
> table.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10414) Distributed replay should re-encode WAL entries with its own RPC codec
[ https://issues.apache.org/jira/browse/HBASE-10414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10414: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Distributed replay should re-encode WAL entries with its own RPC codec > -- > > Key: HBASE-10414 > URL: https://issues.apache.org/jira/browse/HBASE-10414 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0, 0.99.0 >Reporter: Andrew Purtell >Assignee: ramkrishna.s.vasudevan > Fix For: 0.99.0, 0.98.3 > > > HBASE-10412 allows distributed replay to send WAL entries with tags intact > between RegionServers by substituting a WALCodec directly for the RPC codec. > We should instead have distributed replay handle WAL entries including tags > with its own tag-aware RPC codec and drop the direct use of WALCodecs for > that purpose. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10560) Per cell TTLs
[ https://issues.apache.org/jira/browse/HBASE-10560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10560: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Per cell TTLs > - > > Key: HBASE-10560 > URL: https://issues.apache.org/jira/browse/HBASE-10560 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 0.99.0, 0.98.3 > > > Now that we have cell tags, we can optionally store TTLs per cell. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10499) In write heavy scenario one of the regions does not get flushed causing RegionTooBusyException
[
https://issues.apache.org/jira/browse/HBASE-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10499:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> In write heavy scenario one of the regions does not get flushed causing
> RegionTooBusyException
> --
>
> Key: HBASE-10499
> URL: https://issues.apache.org/jira/browse/HBASE-10499
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-10499.patch,
> hbase-root-master-ip-10-157-0-229.zip,
> hbase-root-regionserver-ip-10-93-128-92.zip, master_4e39.log,
> master_576f.log, rs_4e39.log, rs_576f.log, t1.dump, t2.dump,
> workloada_0.98.dat
>
>
> I got this while testing 0.98RC. But am not sure if it is specific to this
> version. Doesn't seem so to me.
> Also it is something similar to HBASE-5312 and HBASE-5568.
> Using 10 threads i do writes to 4 RS using YCSB. The table created has 200
> regions. In one of the run with 0.98 server and 0.98 client I faced this
> problem like the hlogs became more and the system requested flushes for those
> many regions.
> One by one everything was flushed except one and that one thing remained
> unflushed. The ripple effect of this on the client side
> {code}
> com.yahoo.ycsb.DBException:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 54 actions: RegionTooBusyException: 54 times,
> at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:245)
> at com.yahoo.ycsb.DBWrapper.cleanup(DBWrapper.java:73)
> at com.yahoo.ycsb.ClientThread.run(Client.java:307)
> Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed
> 54 actions: RegionTooBusyException: 54 times,
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.makeException(AsyncProcess.java:187)
> at
> org.apache.hadoop.hbase.client.AsyncProcess$BatchErrors.access$500(AsyncProcess.java:171)
> at
> org.apache.hadoop.hbase.client.AsyncProcess.getErrors(AsyncProcess.java:897)
> at
> org.apache.hadoop.hbase.client.HTable.backgroundFlushCommits(HTable.java:961)
> at
> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1225)
> at com.yahoo.ycsb.db.HBaseClient.cleanup(HBaseClient.java:232)
> ... 2 more
> {code}
> On one of the RS
> {code}
> 2014-02-11 08:45:58,714 INFO [regionserver60020.logRoller] wal.FSHLog: Too
> many hlogs: logs=38, maxlogs=32; forcing flush of 23 regions(s):
> 97d8ae2f78910cc5ded5fbb1ddad8492, d396b8a1da05c871edcb68a15608fdf2,
> 01a68742a1be3a9705d574ad68fec1d7, 1250381046301e7465b6cf398759378e,
> 127c133f47d0419bd5ab66675aff76d4, 9f01c5d25ddc6675f750968873721253,
> 29c055b5690839c2fa357cd8e871741e, ca4e33e3eb0d5f8314ff9a870fc43463,
> acfc6ae756e193b58d956cb71ccf0aa3, 187ea304069bc2a3c825bc10a59c7e84,
> 0ea411edc32d5c924d04bf126fa52d1e, e2f9331fc7208b1b230a24045f3c869e,
> d9309ca864055eddf766a330352efc7a, 1a71bdf457288d449050141b5ff00c69,
> 0ba9089db28e977f86a27f90bbab9717, fdbb3242d3b673bbe4790a47bc30576f,
> bbadaa1f0e62d8a8650080b824187850, b1a5de30d8603bd5d9022e09c574501b,
> cc6a9fabe44347ed65e7c325faa72030, 313b17dbff2497f5041b57fe13fa651e,
> 6b788c498503ddd3e1433a4cd3fb4e39, 3d71274fe4f815882e9626e1cfa050d1,
> acc43e4b42c1a041078774f4f20a3ff5
> ..
> 2014-02-11 08:47:49,580 INFO [regionserver60020.logRoller] wal.FSHLog: Too
> many hlogs: logs=53, maxlogs=32; forcing flush of 2 regions(s):
> fdbb3242d3b673bbe4790a47bc30576f, 6b788c498503ddd3e1433a4cd3fb4e39
> {code}
> {code}
> 2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
> flush for region
> usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
> delay of 16689
> 2014-02-11 09:42:44,237 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
> flush for region
> usertable,user6264,1392107806983.6b788c498503ddd3e1433a4cd3fb4e39. after a
> delay of 15868
> 2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
> flush for region
> usertable,user3654,1392107806977.fdbb3242d3b673bbe4790a47bc30576f. after a
> delay of 20847
> 2014-02-11 09:42:54,238 INFO [regionserver60020.periodicFlusher]
> regionserver.HRegionServer: regionserver60020.periodicFlusher requesting
> flush for region
> usertable,user6264,1392107806983.6b788c498503dd
[jira] [Updated] (HBASE-8039) Make HDFS replication number configurable for a column family
[ https://issues.apache.org/jira/browse/HBASE-8039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-8039: -- Fix Version/s: (was: 0.98.2) 0.98.3 > Make HDFS replication number configurable for a column family > - > > Key: HBASE-8039 > URL: https://issues.apache.org/jira/browse/HBASE-8039 > Project: HBase > Issue Type: Improvement > Components: HFile >Reporter: Maryann Xue >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > > To allow users to decide which column family's data is more important and > which is less important by specifying a replica number instead of using the > default replica number. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9804) Startup option for holding user table deployment
[ https://issues.apache.org/jira/browse/HBASE-9804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9804: -- Fix Version/s: (was: 0.98.2) 0.98.3 > Startup option for holding user table deployment > > > Key: HBASE-9804 > URL: https://issues.apache.org/jira/browse/HBASE-9804 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > > Introduce a boolean configuration option, false by default, that if set to > 'true' will cause the master to set all user tables to disabled state at > startup. From there, individual tables can be onlined as normal. Add a new > admin method HBA#enableAll for convenience, also a new HBA#disableAll for > symmetry. Add shell support for sending those new admin commands. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9845) Add MR support for cell ACLs
[ https://issues.apache.org/jira/browse/HBASE-9845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9845: -- Fix Version/s: (was: 0.98.2) 0.98.3 > Add MR support for cell ACLs > > > Key: HBASE-9845 > URL: https://issues.apache.org/jira/browse/HBASE-9845 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.0 >Reporter: Andrew Purtell > Fix For: 0.98.3 > > > We should support adding cell ACLs through TableOutputFormat and > HFileOutputFormat. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6581) Build with hadoop.profile=3.0
[ https://issues.apache.org/jira/browse/HBASE-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6581: -- Fix Version/s: (was: 0.98.2) 0.98.3 > Build with hadoop.profile=3.0 > - > > Key: HBASE-6581 > URL: https://issues.apache.org/jira/browse/HBASE-6581 > Project: HBase > Issue Type: Bug >Reporter: Eric Charles >Assignee: Eric Charles > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE-6581-1.patch, HBASE-6581-2.patch, > HBASE-6581-20130821.patch, HBASE-6581-3.patch, HBASE-6581-4.patch, > HBASE-6581-5.patch, HBASE-6581-6.patch, HBASE-6581-7.patch, > HBASE-6581-8-pom.patch, HBASE-6581.diff, HBASE-6581.diff > > > Building trunk with hadoop.profile=3.0 gives exceptions (see [1]) due to > change in the hadoop maven modules naming (and also usage of 3.0-SNAPSHOT > instead of 3.0.0-SNAPSHOT in hbase-common). > I can provide a patch that would move most of hadoop dependencies in their > respective profiles and will define the correct hadoop deps in the 3.0 > profile. > Please tell me if that's ok to go this way. > Thx, Eric > [1] > $ mvn clean install -Dhadoop.profile=3.0 > [INFO] Scanning for projects... > [ERROR] The build could not read 3 projects -> [Help 1] > [ERROR] > [ERROR] The project org.apache.hbase:hbase-server:0.95-SNAPSHOT > (/d/hbase.svn/hbase-server/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 655, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 659, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 663, column 21 > [ERROR] > [ERROR] The project org.apache.hbase:hbase-common:0.95-SNAPSHOT > (/d/hbase.svn/hbase-common/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 170, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 174, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 178, column 21 > [ERROR] > [ERROR] The project org.apache.hbase:hbase-it:0.95-SNAPSHOT > (/d/hbase.svn/hbase-it/pom.xml) has 3 errors > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-common:jar is missing. @ line 220, column 18 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-annotations:jar is missing. @ line 224, column 21 > [ERROR] 'dependencies.dependency.version' for > org.apache.hadoop:hadoop-minicluster:jar is missing. @ line 228, column 21 > [ERROR] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9527) Review all old api that takes a table name as a byte array and ensure none can pass ns + tablename
[ https://issues.apache.org/jira/browse/HBASE-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-9527: -- Fix Version/s: (was: 0.98.2) 0.98.3 > Review all old api that takes a table name as a byte array and ensure none > can pass ns + tablename > -- > > Key: HBASE-9527 > URL: https://issues.apache.org/jira/browse/HBASE-9527 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Critical > Fix For: 0.99.0, 0.98.3 > > > Go over all old APIs that take a table name and ensure that it is not > possible to pass in a byte array that is a namespace + tablename; instead > throw an exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10561) Forward port: HBASE-10212 New rpc metric: number of active handler
[ https://issues.apache.org/jira/browse/HBASE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10561: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Forward port: HBASE-10212 New rpc metric: number of active handler > -- > > Key: HBASE-10561 > URL: https://issues.apache.org/jira/browse/HBASE-10561 > Project: HBase > Issue Type: Sub-task > Components: IPC/RPC >Reporter: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.98.3 > > > The metrics implementation has changed a lot in 0.96. > Forward port HBASE-10212 to 0.96 and later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10507) Proper filter tests for TestImportExport
[ https://issues.apache.org/jira/browse/HBASE-10507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10507: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Proper filter tests for TestImportExport > > > Key: HBASE-10507 > URL: https://issues.apache.org/jira/browse/HBASE-10507 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl > Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 > > > See parent. TestImportExport.testWithFilter used to passed by accident (until > parent is fixed and until very recently also in trunk). > This is as simple as just added some non-matching rows to the tests. Other > than parent that should be added to all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10613) Discuss WAL device tiering in the Security section of the online manual
[ https://issues.apache.org/jira/browse/HBASE-10613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10613: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Discuss WAL device tiering in the Security section of the online manual > --- > > Key: HBASE-10613 > URL: https://issues.apache.org/jira/browse/HBASE-10613 > Project: HBase > Issue Type: Task >Affects Versions: 0.98.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > > Add text to the Security section of the online manual that discusses how the > WAL can be provisioned to a separate device tier, and how tiers can be > managed with different sensitivity considerations, with and without WAL > encryption. > Something to call out and discuss. This issue is a reminder for me to do that > before spinning 0.98.1. > After HBASE-10077, the new section can be updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10758) TestAdmin runs too long
[
https://issues.apache.org/jira/browse/HBASE-10758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10758:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> TestAdmin runs too long
> ---
>
> Key: HBASE-10758
> URL: https://issues.apache.org/jira/browse/HBASE-10758
> Project: HBase
> Issue Type: Test
>Affects Versions: 0.98.0
> Environment: Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
> HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
>Reporter: Andrew Purtell
> Fix For: 0.99.0, 0.98.3
>
>
> About ~430 seconds here.
> {noformat}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.hbase.client.TestAdmin
> Tests run: 44, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 427.247 sec
> Results :
> Tests run: 44, Failures: 0, Errors: 0, Skipped: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11063) Normalize logging between AccessController and VisibilityController
[ https://issues.apache.org/jira/browse/HBASE-11063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11063: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Normalize logging between AccessController and VisibilityController > --- > > Key: HBASE-11063 > URL: https://issues.apache.org/jira/browse/HBASE-11063 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.1 >Reporter: Andrew Purtell > Fix For: 0.99.0, 0.98.3 > > > The AccessController and VisibilityController should log auditable events > using a format parsable with the same regular expression. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10883) Restrict the universe of labels and authorizations
[
https://issues.apache.org/jira/browse/HBASE-10883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10883:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> Restrict the universe of labels and authorizations
> --
>
> Key: HBASE-10883
> URL: https://issues.apache.org/jira/browse/HBASE-10883
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.98.1
>Reporter: Andrew Purtell
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.99.0, 0.98.3
>
> Attachments: HBASE-10883.patch, HBASE-10883_1.patch,
> HBASE-10883_2.patch, HBASE-10883_3.patch, HBASE-10883_4.patch,
> HBASE-10883_5.patch, HBASE-10883_6.patch, HBASE-10883_7.patch
>
>
> Currently we allow any string as visibility label or request authorization.
> However as seen on HBASE-10878, we accept for authorizations strings that
> would not work if provided as labels in visibility expressions. We should
> throw an exception at least in cases where someone tries to define or use a
> label or authorization including visibility expression operators '&', '|',
> '!', '(', ')'.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10831) IntegrationTestIngestWithACL is not setting up LoadTestTool correctly
[
https://issues.apache.org/jira/browse/HBASE-10831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10831:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> IntegrationTestIngestWithACL is not setting up LoadTestTool correctly
> -
>
> Key: HBASE-10831
> URL: https://issues.apache.org/jira/browse/HBASE-10831
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.1
>Reporter: Andrew Purtell
>Assignee: ramkrishna.s.vasudevan
> Fix For: 0.99.0, 0.98.3
>
>
> IntegrationTestIngestWithACL is not setting up LoadTestTool correctly.
> {noformat}
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 601.709 sec
> <<< FAILURE!
> testIngest(org.apache.hadoop.hbase.IntegrationTestIngestWithACL) Time
> elapsed: 601.489 sec <<< FAILURE!
> java.lang.AssertionError: Failed to initialize LoadTestTool expected:<0> but
> was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.hbase.IntegrationTestIngest.initTable(IntegrationTestIngest.java:74)
> at
> org.apache.hadoop.hbase.IntegrationTestIngest.setUpCluster(IntegrationTestIngest.java:69)
> at
> org.apache.hadoop.hbase.IntegrationTestIngestWithACL.setUpCluster(IntegrationTestIngestWithACL.java:58)
> at
> org.apache.hadoop.hbase.IntegrationTestBase.setUp(IntegrationTestBase.java:89)
> {noformat}
> Could be related to HBASE-10675?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11036) Online schema change with region merge may cause data loss
[
https://issues.apache.org/jira/browse/HBASE-11036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11036:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> Online schema change with region merge may cause data loss
> ---
>
> Key: HBASE-11036
> URL: https://issues.apache.org/jira/browse/HBASE-11036
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
> Fix For: 0.99.0, 0.98.3
>
>
> We have found out that online schema change and region merges may still cause
> issues about merged regions coming back online, and thus causing data loss.
> Recently ITBLL failed reporting 800K missing rows out of 720M. We've been
> running this test for some extended period of time, and this is the first we
> are seeing it, meaning that it is more rare. But it is still concerning.
> From master's log:
> The merge has happened:
> {code}
> 2014-04-16 18:26:37,247 INFO [AM.ZK.Worker-pool2-t73]
> master.AssignmentManager: Handled MERGED event;
> merged=IntegrationTestBigLinkedList,\xB2\xFE\x03s,1397672795119.80159738a0167e20a2e29fb2c46702f2.,
>
>
> region_a=IntegrationTestBigLinkedList,\xB2\xFE\x03s,1397670978959.0ac116e4d7da87922d3a8f218ca21079.,
>
>
> region_b=IntegrationTestBigLinkedList,\xB8\x03\x94\x15,1397672587456.1265d06478082ced65dd9a0c1c2b63c2.,
> on hor13n03.gq1.ygridcore.net,60020,1397672668647
> {code}
> The region server shows the merge is complete:
> {code}
> 2014-04-16 18:26:37,352 INFO [regionserver60020-merges-1397672794972]
> regionserver.RegionMergeRequest: Regions merged, hbase:meta updated, and
> report to master.
> region_a=IntegrationTestBigLinkedList,\xB2\xFE\x03s,1397670978959.0ac116e4d7da87922d3a8f218ca21079.,
>
> region_b=IntegrationTestBigLinkedList,\xB8\x03\x94\x15,1397672587456.1265d06478082ced65dd9a0c1c2b63c2..
> Region merge took 2sec
> {code}
> The new region was online on the region server for some time:
> {code}
> 2014-04-16 18:31:22,858 DEBUG [RS_OPEN_REGION-hor13n03:60020-1]
> handler.OpenRegionHandler: Opened
> IntegrationTestBigLinkedList,\xB2\xFE\x03s,1397672795119.80159738a0167e20a2e29fb2c46702f2.
> on hor13n03.gq1.ygridcore.net,60020,1397672668647
> {code}
> Then the region server was killed around 2014-04-16 18:31:26,254.
> The master started log splitting etc for the dead RS:
> {code}
> 2014-04-16 18:31:28,942 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
> handler.ServerShutdownHandler: Splitting logs for
> hor13n03.gq1.ygridcore.net,60020,1397672668647 before assignment.
> ..
> 2014-04-16 18:31:40,887 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
> master.RegionStates: Transitioned {80159738a0167e20a2e29fb2c46702f2
> state=OPEN, ts=1397673082874,
> server=hor13n03.gq1.ygridcore.net,60020,1397672668647} to
> {80159738a0167e20a2e29fb
> 2014-04-16 18:31:40,887 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
> master.RegionStates: Offlined 80159738a0167e20a2e29fb2c46702f2 from
> hor13n03.gq1.ygridcore.net,60020,1397672668647
> {code}
> But this region was not assigned again at all. Instead, the master restarted
> shortly after, and reassigned the regions that were merged already:
> {code}
> 2014-04-16 18:34:02,569 INFO [master:hor13n02:6]
> master.ActiveMasterManager: Registered Active
> Master=hor13n02.gq1.ygridcore.net,6,1397673241215
> ...
> 2014-04-16 18:34:10,412 INFO [master:hor13n02:6]
> master.AssignmentManager: Found regions out on cluster or in RIT; presuming
> failover
> 2014-04-16 18:34:10,412 WARN [master:hor13n02:6] master.ServerManager:
> Expiration of hor13n03.gq1.ygridcore.net,60020,1397671753021 but server not
> online
> ..
> 2014-04-16 18:34:10,880 INFO [MASTER_SERVER_OPERATIONS-hor13n02:6-3]
> master.AssignmentManager: Bulk assigning 28 region(s) across 4 server(s),
> round-robin=true
> ..
> 2014-04-16 18:34:10,882 DEBUG
> [hor13n02.gq1.ygridcore.net,6,1397673241215-GeneralBulkAssigner-1]
> zookeeper.ZKAssign: master:6-0x2456a4863640255,
> quorum=hor13n04.gq1.ygridcore.net:2181,hor13n03.gq1.ygridcore.net:2181,hor13n20.gq1.ygridcore.net:2181,
> baseZNode=/hbase Async create of unassigned node 0ac116e4d7da8792..
> ..
> 2014-04-16 18:34:13,077 INFO [AM.ZK.Worker-pool2-t7] master.RegionStates:
> Onlined 0ac116e4d7da87922d3a8f218ca21079 on
> hor13n04.gq1.ygridcore.net,60020,1397672685370
> {code}
> Before the master went down, there were other logs that indicate something
> funky:
> {code}
> 2014-04-16 18:32:42,113 DEBUG
> [hor13n02.gq1.ygridcore.net,6,1397672191204-org.apache.hadoop.hbase.master.BulkReOpen-0]
> master.AssignmentManager: Starting unassign of
> IntegrationTestBigLinkedList,\xB2\xFE\x03s,1397670978959.0ac116e4d7da87922d3a8f218ca
> 2014-04-16 18:32:42,113 INFO
> [hor13n02.gq1.ygridcore.net,600
[jira] [Updated] (HBASE-11048) Support setting custom priority per client RPC
[ https://issues.apache.org/jira/browse/HBASE-11048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11048: --- Fix Version/s: (was: 0.98.2) 0.98.3 > Support setting custom priority per client RPC > -- > > Key: HBASE-11048 > URL: https://issues.apache.org/jira/browse/HBASE-11048 > Project: HBase > Issue Type: Improvement > Components: Client >Affects Versions: 0.99.0, 0.98.2 >Reporter: Jesse Yates >Assignee: Jesse Yates > Labels: Phoenix > Fix For: 0.99.0, 0.98.3 > > Attachments: hbase-11048-trunk-v0.patch > > > Servers have the ability to handle custom rpc priority levels, but currently > we are only using it to differentiate META/ROOT updates from replication and > other 'priority' updates (as specified by annotation tags per RS method). > However, some clients need the ability to create custom handlers (e.g. > PHOENIX-938) which can really only be cleanly tied together to requests by > the request priority. The disconnect is in that there is no way for the > client to overwrite the priority per table - the PayloadCarryingRpcController > will always just set priority per ROOT/META and otherwise just use the > generic priority. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10820) Bulkload: something is going bananas with '_tmp'
[
https://issues.apache.org/jira/browse/HBASE-10820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10820:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> Bulkload: something is going bananas with '_tmp'
>
>
> Key: HBASE-10820
> URL: https://issues.apache.org/jira/browse/HBASE-10820
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Nick Dimiduk
>Priority: Minor
> Fix For: 0.99.0, 0.98.3
>
>
> While working on HBASE-10818, I noted the following in my logs
> {noformat}
> 2014-03-24 15:26:25,283 INFO [RpcServer.handler=24,port=52056]
> regionserver.HStore: Successfully loaded store file
> file:/Users/ndimiduk/repos/hbase/target/test-data/8e203abb-90b4-4284-9816-963ce3461d5f/IntegrationTestBulkLoad-2/L/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/_tmp/IntegrationTestBulkLoad,119.bottom
> into store L (new location:
> file:/var/folders/b8/n5n91drd7xg0rlt5n6fgsjtwgn/T/hbase-ndimiduk/hbase/data/default/IntegrationTestBulkLoad/42e004b8909307e2d983ddbe59276638/L/6443c445f00d453dbdb0b41f925efdb5_SeqId_9_)
> {noformat}
> Something is going overboard with the temp directory path. Worth
> investigating.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11041) HBaseTestingUtil.createMultiRegions deals incorrectly with missing column family
[
https://issues.apache.org/jira/browse/HBASE-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-11041:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> HBaseTestingUtil.createMultiRegions deals incorrectly with missing column
> family
>
>
> Key: HBASE-11041
> URL: https://issues.apache.org/jira/browse/HBASE-11041
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3
>
>
> Just found a test failing like this:
> {code}
> Error Message
> HTableDescriptor is read-only
> Stacktrace
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.UnmodifyableHTableDescriptor.addFamily(UnmodifyableHTableDescriptor.java:64)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:1302)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:1291)
> at
> org.apache.hadoop.hbase.HBaseTestingUtility.createMultiRegions(HBaseTestingUtility.java:1286)
> at
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting.installTable(TestDistributedLogSplitting.java:485)
> at
> org.apache.hadoop.hbase.master.TestDistributedLogSplitting.testMasterStartsUpWithLogSplittingWork(TestDistributedLogSplitting.java:282)
> {code}
> The code that causes this looks like this:
> {code}
> HTableDescriptor htd = table.getTableDescriptor();
> if(!htd.hasFamily(columnFamily)) {
> HColumnDescriptor hcd = new HColumnDescriptor(columnFamily);
> htd.addFamily(hcd);
> }
> {code}
> But note that table.getTableDescriptor() returns an
> UnmodifyableHTableDescriptor, so the add would *always* fail.
> The specific test that failed was
> TestDistributedLogSplitting.testMasterStartsUpWithLogSplittingWork.
> Looks like the HMaster did not have the last table descriptor state, yet.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10761) StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic
[ https://issues.apache.org/jira/browse/HBASE-10761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10761: --- Fix Version/s: (was: 0.98.2) 0.98.3 > StochasticLoadBalancer still uses SimpleLoadBalancer's needBalance logic > > > Key: HBASE-10761 > URL: https://issues.apache.org/jira/browse/HBASE-10761 > Project: HBase > Issue Type: Bug > Components: Balancer >Affects Versions: 0.98.0 >Reporter: Victor Xu > Fix For: 0.99.0, 0.98.3 > > Attachments: HBASE_10761.patch, HBASE_10761_v2.patch > > > StochasticLoadBalancer has become the default balancer since 0.98.0. But its > balanceCluster method still uses the BaseLoadBalancer.needBalance() which is > originally designed for SimpleLoadBalancer. It's all based on the number of > regions on the regionservers. > This can cause such a problem: when the cluster has some Hot Spot Region, the > balance process may not be triggered because the numbers of regions on the > RegionServers are averaged. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10832) IntegrationTestIngestStripeCompactions timed out
[ https://issues.apache.org/jira/browse/HBASE-10832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-10832: --- Fix Version/s: (was: 0.98.2) 0.98.3 > IntegrationTestIngestStripeCompactions timed out > > > Key: HBASE-10832 > URL: https://issues.apache.org/jira/browse/HBASE-10832 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.1 >Reporter: Andrew Purtell >Priority: Minor > Fix For: 0.99.0, 0.98.3 > > > IntegrationTestIngestStripeCompactions timed out when executing in local > mode, failed to shut down cleanly (LoadTestTool worker threads were trying to > finish, master's catalog scanner was trying to scan a meta table that had > gone away, etc.), and became a zombie. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10664) TestImportExport runs too long
[
https://issues.apache.org/jira/browse/HBASE-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10664:
---
Fix Version/s: (was: 0.98.2)
0.98.3
> TestImportExport runs too long
> --
>
> Key: HBASE-10664
> URL: https://issues.apache.org/jira/browse/HBASE-10664
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Andrew Purtell
> Fix For: 0.99.0, 0.98.3
>
>
> Debugging with -Dsurefire.firstPartForkMode=always
> -Dsurefire.secondPartForkMode=always looking for a hanging test.
> 388 seconds.
> {noformat}
> Forking command line: /bin/sh -c cd /data/src/hbase/hbase-server &&
> /usr/lib/jvm/java-1.7.0.45-oracle-amd64/jre/bin/java -enableassertions
> -Xmx1900m -XX:MaxPermSize=100m -Djava.security.egd=file:/dev/./urandom
> -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -jar
> /data/src/hbase/hbase-server/target/surefire/surefirebooter7637958208277391169.jar
> /data/src/hbase/hbase-server/target/surefire/surefire6877889026110956843tmp
> /data/src/hbase/hbase-server/target/surefire/surefire_1907837210788480451831tmp
> Running org.apache.hadoop.hbase.mapreduce.TestImportExport
> Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 388.246 sec
> {noformat}
> Slim down or break it up.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11077: --- Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Status: Resolved (was: Patch Available) What I committed to trunk and 0.98. Thanks for the reviews and feedback all! > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077-0.98.patch, HBASE-11077.patch, > HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-11077: --- Attachment: HBASE-11077-0.98.patch HBASE-11077.patch > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077-0.98.patch, HBASE-11077.patch, > HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10810) LoadTestTool should share the connection and connection pool
[
https://issues.apache.org/jira/browse/HBASE-10810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986255#comment-13986255
]
Enis Soztutar commented on HBASE-10810:
---
Thanks Nick for the review.
bq. What's unclear to me is how the 100 reader threads will contend for the (up
to) 256 threads in the shared pool used by all HTables for the connection
In case this becomes a problem, we can increase the default pool side that the
LTT connection is using.
> LoadTestTool should share the connection and connection pool
>
>
> Key: HBASE-10810
> URL: https://issues.apache.org/jira/browse/HBASE-10810
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: hbase-10070
>
> Attachments: hbase-10810_v1.patch
>
>
> While running the IT test from HBASE-10572, we've noticed that the number of
> threads jumps to 4K's when CM actions are going on.
> Our [~ndimiduk] summarizes the problem quite good:
> MultiThreadedReader creates this pool for each HTable:
> {code}
> ThreadPoolExecutor pool = new ThreadPoolExecutor(1, maxThreads,
> keepAliveTime, TimeUnit.SECONDS,
> new SynchronousQueue(),
> Threads.newDaemonThreadFactory("htable"));
> {code}
> This comes from the HTable creation
> {code}
> public HTable(Configuration conf, final TableName tableName)
> {code}
> As well the javadoc says Recommended.
> This is wrong.
> In this issue we can change the LTT sub classes to use the shared connection
> object and initialize their tables using HConnection.getTable() rather than
> new HTable().
> This is relevant to trunk as well, but there since there is only one
> outstanding RPC per thread, it is not such a big problem.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10661) TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is flaky
[ https://issues.apache.org/jira/browse/HBASE-10661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-10661: -- Attachment: hbase-10661_v6.patch [~devaraj] can you review the latest version as well. The tests have been more stable after this for some time. v6 should make it even more so. > TestStochasticLoadBalancer.testRegionReplicationOnMidClusterWithRacks() is > flaky > > > Key: HBASE-10661 > URL: https://issues.apache.org/jira/browse/HBASE-10661 > Project: HBase > Issue Type: Sub-task >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: hbase-10070 > > Attachments: hbase-10661_v1.patch, hbase-10661_v2.patch, > hbase-10661_v3.patch, hbase-10661_v4.patch, hbase-10661_v5.patch, > hbase-10661_v6.patch > > > One of the tests introduced in HBASE-10351 seems to be flaky. The LB cannot > compute the fully assignment plan in time when there are racks and region > replicas for the test, so it is failing sometimes. > We can reduce the computation amount, and increase the LB runtime to make the > test stable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HBASE-11065) hbase 0.94
[ https://issues.apache.org/jira/browse/HBASE-11065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-11065. --- Resolution: Invalid Fix Version/s: (was: 0.94.20) Thanks [~dougc3]. I just build mvn site successfully on my local machine. The jenkins builds also do not complain. If this problem persists for you, it's usually best to ask on the mailing lists. > hbase 0.94 > -- > > Key: HBASE-11065 > URL: https://issues.apache.org/jira/browse/HBASE-11065 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.94.18 > Environment: centos 6.4 >Reporter: doug chang >Priority: Minor > Attachments: HBASE-11065.patch > > > mvn site fails > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-site-plugin:3.2:site (default-site) on project > hbase: SiteToolException: ArtifactResolutionException: Unable to find skin: > Could not transfer artifact org.apache.maven.skins:maven-stylus-skin:jar:1.5 > from/to central (http//repo.maven.apache.org/maven2/): No connector available > to access repository central (http//repo.maven.apache.org/maven2/) of type > default using the available factories WagonRepositoryConnectorFactory > [ERROR] org.apache.maven.skins:maven-stylus-skin:jar:RELEASE -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[ https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11104: --- Attachment: 11104-v1.txt > IntegrationTestImportTsv#testRunFromOutputCommitter misses credential > initialization > > > Key: HBASE-11104 > URL: https://issues.apache.org/jira/browse/HBASE-11104 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Attachments: 11104-v1.txt > > > IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships > the HBase dependencies. > However, call to TableMapReduceUtil.initCredentials(job) is missing - making > this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[ https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11104: --- Attachment: (was: 11104-v1.txt) > IntegrationTestImportTsv#testRunFromOutputCommitter misses credential > initialization > > > Key: HBASE-11104 > URL: https://issues.apache.org/jira/browse/HBASE-11104 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Attachments: 11104-v1.txt > > > IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships > the HBase dependencies. > However, call to TableMapReduceUtil.initCredentials(job) is missing - making > this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10942) support parallel request cancellation for multi-get
[ https://issues.apache.org/jira/browse/HBASE-10942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986223#comment-13986223 ] Enis Soztutar commented on HBASE-10942: --- [~nkeywal] it seems that this is your area of expertise. > support parallel request cancellation for multi-get > --- > > Key: HBASE-10942 > URL: https://issues.apache.org/jira/browse/HBASE-10942 > Project: HBase > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: hbase-10070 > > Attachments: HBASE-10942.01.patch, HBASE-10942.patch > > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986221#comment-13986221 ] Lars Hofhansl commented on HBASE-10958: --- You 'd man. > [dataloss] Bulk loading with seqids can prevent some log entries from being > replayed > > > Key: HBASE-10958 > URL: https://issues.apache.org/jira/browse/HBASE-10958 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.18 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.99.0, 0.98.2, 0.96.3, 0.94.20 > > Attachments: HBASE-10958-0.94.patch, > HBASE-10958-less-intrusive-hack-0.96.patch, > HBASE-10958-quick-hack-0.96.patch, HBASE-10958-v2.patch, > HBASE-10958-v3.patch, HBASE-10958.patch > > > We found an issue with bulk loads causing data loss when assigning sequence > ids (HBASE-6630) that is triggered when replaying recovered edits. We're > nicknaming this issue *Blindspot*. > The problem is that the sequence id given to a bulk loaded file is higher > than those of the edits in the region's memstore. When replaying recovered > edits, the rule to skip some of them is that they have to be _lower than the > highest sequence id_. In other words, the edits that have a sequence id lower > than the highest one in the store files *should* have also been flushed. This > is not the case with bulk loaded files since we now have an HFile with a > sequence id higher than unflushed edits. > The log recovery code takes this into account by simply skipping the bulk > loaded files, but this "bulk loaded status" is *lost* on compaction. The > edits in the logs that have a sequence id lower than the bulk loaded file > that got compacted are put in a blind spot and are skipped during replay. > Here's the easiest way to recreate this issue: > - Create an empty table > - Put one row in it (let's say it gets seqid 1) > - Bulk load one file (it gets seqid 2). I used ImporTsv and set > hbase.mapreduce.bulkload.assign.sequenceNumbers. > - Bulk load a second file the same way (it gets seqid 3). > - Major compact the table (the new file has seqid 3 and isn't considered > bulk loaded). > - Kill the region server that holds the table's region. > - Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 3 makes us believe that > everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11105) Add bandwidth limit documentation to snapshot export section of refguide
Ted Yu created HBASE-11105: -- Summary: Add bandwidth limit documentation to snapshot export section of refguide Key: HBASE-11105 URL: https://issues.apache.org/jira/browse/HBASE-11105 Project: HBase Issue Type: Task Reporter: Ted Yu Priority: Minor http://hbase.apache.org/book.html#ops.snapshots.export lists command line arguments for ExportSnapshot Parameters for bandwidth limitation (HBASE-11083) should be documented. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-9445) Snapshots should create column family dirs for empty regions
[
https://issues.apache.org/jira/browse/HBASE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-9445:
-
Status: Open (was: Patch Available)
> Snapshots should create column family dirs for empty regions
>
>
> Key: HBASE-9445
> URL: https://issues.apache.org/jira/browse/HBASE-9445
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.99.0, 0.98.2
>
> Attachments: hbase-9445_v1.patch, hbase-9445_v2.patch,
> hbase-9445_v3.patch, hbase-9445_v4.patch
>
>
> Currently, taking a snapshot will not create the family directory under a
> region if the family does not have any files in it.
> Subsequent verification fails because of this. There is some logic in the
> SnapshotTestingUtils.confirmSnapshotValid() to deal with empty family
> directories, but I think we should create the family directories regardless
> of whether there are any hfiles referencing them.
> {code}
> 2013-09-05 11:07:21,566 DEBUG [Thread-208] util.FSUtils(1687): |-data/
> 2013-09-05 11:07:21,567 DEBUG [Thread-208] util.FSUtils(1687): |default/
> 2013-09-05 11:07:21,568 DEBUG [Thread-208] util.FSUtils(1687): |---test/
> 2013-09-05 11:07:21,569 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tabledesc/
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1690):
> |-.tableinfo.01
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tmp/
> 2013-09-05 11:07:21,571 DEBUG [Thread-208] util.FSUtils(1687):
> |--accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1690):
> |-.regioninfo
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1687):
> |-fam/
> 2013-09-05 11:07:21,555 DEBUG [Thread-208] util.FSUtils(1687):
> |-.hbase-snapshot/
> 2013-09-05 11:07:21,556 DEBUG [Thread-208] util.FSUtils(1687): |.tmp/
> 2013-09-05 11:07:21,557 DEBUG [Thread-208] util.FSUtils(1687):
> |offlineTableSnapshot/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |---.snapshotinfo
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1687):
> |---.tabledesc/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |--.tableinfo.01
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687): |---.tmp/
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687):
> |---accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,560 DEBUG [Thread-208] util.FSUtils(1690):
> |--.regioninfo
> {code}
> I think this is important for 0.96.0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-9445) Snapshots should create column family dirs for empty regions
[
https://issues.apache.org/jira/browse/HBASE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-9445:
-
Status: Patch Available (was: Open)
> Snapshots should create column family dirs for empty regions
>
>
> Key: HBASE-9445
> URL: https://issues.apache.org/jira/browse/HBASE-9445
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.99.0, 0.98.2
>
> Attachments: hbase-9445_v1.patch, hbase-9445_v2.patch,
> hbase-9445_v3.patch, hbase-9445_v4.patch
>
>
> Currently, taking a snapshot will not create the family directory under a
> region if the family does not have any files in it.
> Subsequent verification fails because of this. There is some logic in the
> SnapshotTestingUtils.confirmSnapshotValid() to deal with empty family
> directories, but I think we should create the family directories regardless
> of whether there are any hfiles referencing them.
> {code}
> 2013-09-05 11:07:21,566 DEBUG [Thread-208] util.FSUtils(1687): |-data/
> 2013-09-05 11:07:21,567 DEBUG [Thread-208] util.FSUtils(1687): |default/
> 2013-09-05 11:07:21,568 DEBUG [Thread-208] util.FSUtils(1687): |---test/
> 2013-09-05 11:07:21,569 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tabledesc/
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1690):
> |-.tableinfo.01
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tmp/
> 2013-09-05 11:07:21,571 DEBUG [Thread-208] util.FSUtils(1687):
> |--accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1690):
> |-.regioninfo
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1687):
> |-fam/
> 2013-09-05 11:07:21,555 DEBUG [Thread-208] util.FSUtils(1687):
> |-.hbase-snapshot/
> 2013-09-05 11:07:21,556 DEBUG [Thread-208] util.FSUtils(1687): |.tmp/
> 2013-09-05 11:07:21,557 DEBUG [Thread-208] util.FSUtils(1687):
> |offlineTableSnapshot/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |---.snapshotinfo
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1687):
> |---.tabledesc/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |--.tableinfo.01
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687): |---.tmp/
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687):
> |---accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,560 DEBUG [Thread-208] util.FSUtils(1690):
> |--.regioninfo
> {code}
> I think this is important for 0.96.0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-9445) Snapshots should create column family dirs for empty regions
[
https://issues.apache.org/jira/browse/HBASE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-9445:
-
Attachment: hbase-9445_v4.patch
Rebased the patch, and added the test from v1. [~mbertozzi] would you mind
taking a look.
> Snapshots should create column family dirs for empty regions
>
>
> Key: HBASE-9445
> URL: https://issues.apache.org/jira/browse/HBASE-9445
> Project: HBase
> Issue Type: Bug
> Components: snapshots
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 0.99.0, 0.98.2
>
> Attachments: hbase-9445_v1.patch, hbase-9445_v2.patch,
> hbase-9445_v3.patch, hbase-9445_v4.patch
>
>
> Currently, taking a snapshot will not create the family directory under a
> region if the family does not have any files in it.
> Subsequent verification fails because of this. There is some logic in the
> SnapshotTestingUtils.confirmSnapshotValid() to deal with empty family
> directories, but I think we should create the family directories regardless
> of whether there are any hfiles referencing them.
> {code}
> 2013-09-05 11:07:21,566 DEBUG [Thread-208] util.FSUtils(1687): |-data/
> 2013-09-05 11:07:21,567 DEBUG [Thread-208] util.FSUtils(1687): |default/
> 2013-09-05 11:07:21,568 DEBUG [Thread-208] util.FSUtils(1687): |---test/
> 2013-09-05 11:07:21,569 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tabledesc/
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1690):
> |-.tableinfo.01
> 2013-09-05 11:07:21,570 DEBUG [Thread-208] util.FSUtils(1687):
> |--.tmp/
> 2013-09-05 11:07:21,571 DEBUG [Thread-208] util.FSUtils(1687):
> |--accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1690):
> |-.regioninfo
> 2013-09-05 11:07:21,572 DEBUG [Thread-208] util.FSUtils(1687):
> |-fam/
> 2013-09-05 11:07:21,555 DEBUG [Thread-208] util.FSUtils(1687):
> |-.hbase-snapshot/
> 2013-09-05 11:07:21,556 DEBUG [Thread-208] util.FSUtils(1687): |.tmp/
> 2013-09-05 11:07:21,557 DEBUG [Thread-208] util.FSUtils(1687):
> |offlineTableSnapshot/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |---.snapshotinfo
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1687):
> |---.tabledesc/
> 2013-09-05 11:07:21,558 DEBUG [Thread-208] util.FSUtils(1690):
> |--.tableinfo.01
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687): |---.tmp/
> 2013-09-05 11:07:21,559 DEBUG [Thread-208] util.FSUtils(1687):
> |---accd6e55887057888de758df44dacda7/
> 2013-09-05 11:07:21,560 DEBUG [Thread-208] util.FSUtils(1690):
> |--.regioninfo
> {code}
> I think this is important for 0.96.0.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
[ https://issues.apache.org/jira/browse/HBASE-11104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-11104: --- Attachment: 11104-v1.txt > IntegrationTestImportTsv#testRunFromOutputCommitter misses credential > initialization > > > Key: HBASE-11104 > URL: https://issues.apache.org/jira/browse/HBASE-11104 > Project: HBase > Issue Type: Test >Reporter: Ted Yu >Priority: Minor > Attachments: 11104-v1.txt > > > IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships > the HBase dependencies. > However, call to TableMapReduceUtil.initCredentials(job) is missing - making > this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11104) IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization
Ted Yu created HBASE-11104: -- Summary: IntegrationTestImportTsv#testRunFromOutputCommitter misses credential initialization Key: HBASE-11104 URL: https://issues.apache.org/jira/browse/HBASE-11104 Project: HBase Issue Type: Test Reporter: Ted Yu Priority: Minor IntegrationTestImportTsv#testRunFromOutputCommitter a parent job that ships the HBase dependencies. However, call to TableMapReduceUtil.initCredentials(job) is missing - making this test fail on a secure cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986173#comment-13986173 ] Enis Soztutar commented on HBASE-11077: --- bq. I will do that and add a patch here for the 0.98 version unless objection. Sounds good. bq. If there isn't any take up of the per-operation strategy toggle, we could drop it in 0.99+ ? It seems to me that the strategy should be a table level property (a part of the param this patch adds) or baked into the priviledge (like the READ_INVISIBLE) we were talking about. Another argument for defaulting to a behavior for disallowing scans without read perm by default is that, we do not want a user to DOS the table by launching scanners against it although with no read permission. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11102) Document JDK versions supported by each release
[
https://issues.apache.org/jira/browse/HBASE-11102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986165#comment-13986165
]
Andrew Purtell commented on HBASE-11102:
bq. There has been some work to support JDK8. We can also document that.
Running on a Java 8 runtime "just works" with 0.98 today, but I can't say that
I know of any scale testing so consider that.
Building with JDK8 would require the removal of a deprecated method that causes
a compile time error with this version, a trivial change:
{code}
diff --git hbase-client/src/main/java/org/apache/hadoop/hbase/util/PoolMap.java
index 7283831..5326de4 100644
--- hbase-client/src/main/java/org/apache/hadoop/hbase/util/PoolMap.java
+++ hbase-client/src/main/java/org/apache/hadoop/hbase/util/PoolMap.java
@@ -93,14 +93,6 @@ public class PoolMap implements Map {
return null;
}
- /**
- * @deprecated Will be removed for Java 8, use {@link #removeValue} instead
- */
- @Deprecated
- public boolean remove(K key, V value) {
-return removeValue(key, value);
- }
-
public boolean removeValue(K key, V value) {
Pool pool = pools.get(key);
boolean res = false;
{code}
> Document JDK versions supported by each release
> ---
>
> Key: HBASE-11102
> URL: https://issues.apache.org/jira/browse/HBASE-11102
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
> Fix For: 0.99.0
>
>
> We can make use of a JDK version x HBase version matrix to explain which JDK
> version is supported and required.
> 0.94, 0.96, and 0.98 releases all support JDK6 and JDK7. For 1.0, there is a
> discussion thread to decide whether to drop JDK6 support.
> There has been some work to support JDK8. We can also document that.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11008) Align bulk load, flush, and compact to require Action.CREATE
[ https://issues.apache.org/jira/browse/HBASE-11008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-11008: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) And committed to 0.94, thanks everyone. > Align bulk load, flush, and compact to require Action.CREATE > > > Key: HBASE-11008 > URL: https://issues.apache.org/jira/browse/HBASE-11008 > Project: HBase > Issue Type: Improvement > Components: security >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans > Fix For: 0.99.0, 0.98.2, 0.96.3, 0.94.20 > > Attachments: HBASE-11008-0.94.patch, HBASE-11008-v2.patch, > HBASE-11008-v3.patch, HBASE-11008.patch > > > Over in HBASE-10958 we noticed that it might make sense to require > Action.CREATE for bulk load, flush, and compact since it is also required for > things like enable and disable. > This means the following changes: > - preBulkLoadHFile goes from WRITE to CREATE > - compact/flush go from ADMIN to ADMIN or CREATE -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Daniel Cryans updated HBASE-10958: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Now committed to 0.94 (took some time because I wanted to double check the tests I modified were all green). Thanks everyone. > [dataloss] Bulk loading with seqids can prevent some log entries from being > replayed > > > Key: HBASE-10958 > URL: https://issues.apache.org/jira/browse/HBASE-10958 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.18 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.99.0, 0.98.2, 0.96.3, 0.94.20 > > Attachments: HBASE-10958-0.94.patch, > HBASE-10958-less-intrusive-hack-0.96.patch, > HBASE-10958-quick-hack-0.96.patch, HBASE-10958-v2.patch, > HBASE-10958-v3.patch, HBASE-10958.patch > > > We found an issue with bulk loads causing data loss when assigning sequence > ids (HBASE-6630) that is triggered when replaying recovered edits. We're > nicknaming this issue *Blindspot*. > The problem is that the sequence id given to a bulk loaded file is higher > than those of the edits in the region's memstore. When replaying recovered > edits, the rule to skip some of them is that they have to be _lower than the > highest sequence id_. In other words, the edits that have a sequence id lower > than the highest one in the store files *should* have also been flushed. This > is not the case with bulk loaded files since we now have an HFile with a > sequence id higher than unflushed edits. > The log recovery code takes this into account by simply skipping the bulk > loaded files, but this "bulk loaded status" is *lost* on compaction. The > edits in the logs that have a sequence id lower than the bulk loaded file > that got compacted are put in a blind spot and are skipped during replay. > Here's the easiest way to recreate this issue: > - Create an empty table > - Put one row in it (let's say it gets seqid 1) > - Bulk load one file (it gets seqid 2). I used ImporTsv and set > hbase.mapreduce.bulkload.assign.sequenceNumbers. > - Bulk load a second file the same way (it gets seqid 3). > - Major compact the table (the new file has seqid 3 and isn't considered > bulk loaded). > - Kill the region server that holds the table's region. > - Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 3 makes us believe that > everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986151#comment-13986151 ] Andrew Purtell commented on HBASE-11077: bq. Maybe default to false on 0.98, but true on trunk. Ok. It's a discontinuity on the one hand, but 0.98 is explicitly a laboratory for this stuff. I will do that and add a patch here for the 0.98 version unless objection. bq. Can we reduce that to at least two? If there isn't any take up of the per-operation strategy toggle, we could drop it in 0.99+ ? > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986148#comment-13986148 ] Andrew Purtell commented on HBASE-11077: I added another subtask to the parent for capturing the configuration space and nuances (with use case examples) into the security section of the manual as HBASE-11103 > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986147#comment-13986147 ] Enis Soztutar commented on HBASE-11077: --- bq. Then we break compatibility from 0.98.1 to 0.98.2, in that default behavior prior to 0.98.2 in the 0.98 release line is quite different Maybe default to false on 0.98, but true on trunk. bq. unfortunately cell ACLs would become largely useless, unless the admin research the feature and flip the attribute to "false", because when we early out at CF checks to retain pre-0.98 behavior the cell ACLs that would otherwise grant exceptional access won't be visited, unless using the cell-first strategy Surely we do not want to make the model complex, but at the same time allow both of the use cases. If we have table privs + config option + per-operation cell-first strategy it is already three dimensions. Can we reduce that to at least two? Can we get away with per-operation strategy or the config option? > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11103) [AccessController] Enhance the cell ACL documentation in the security section of the online manual
Andrew Purtell created HBASE-11103: -- Summary: [AccessController] Enhance the cell ACL documentation in the security section of the online manual Key: HBASE-11103 URL: https://issues.apache.org/jira/browse/HBASE-11103 Project: HBase Issue Type: Sub-task Reporter: Andrew Purtell Fix For: 0.99.0, 0.98.3 We should capture the configuration space and the nuances involved wrt cell level ACLs, with use case examples, into the security section of the manual. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986124#comment-13986124 ] Andrew Purtell edited comment on HBASE-11077 at 4/30/14 9:53 PM: - bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2, in that default behavior prior to 0.98.2 in the 0.98 release line is quite different. And, unfortunately cell ACLs would become largely useless, unless the admin research the feature and flip the attribute to "false", because when we early out at CF checks to retain pre-0.98 behavior the cell ACLs that would otherwise grant exceptional access won't be visited, unless using the cell-first strategy, which has the drawback of requiring the cell grant access. bq. Did you stop pursuing the READ_INVISIBLE priv aproach? Check out the other subtask and let's figure out what makes sense for 0.99+. Edits: Clarity, sorry for the multiple changes. was (Author: apurtell): bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. And, cell ACLs become largely useless, since we early out at CF checks, they can't grant exceptional access unless using the cell-first strategy, which has the drawback of requiring the cell grant access. bq. Did you stop pursuing the READ_INVISIBLE priv aproach? Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986131#comment-13986131 ] Andrew Purtell commented on HBASE-11077: There was a reason we changed the behavior when introducing cell ACLs. The parent mentions the issues involved, and the release notes on this issue also. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986124#comment-13986124 ] Andrew Purtell edited comment on HBASE-11077 at 4/30/14 9:43 PM: - bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. And, cell ACLs become largely useless, since we early out at CF checks, they can't grant exceptional access unless using the cell-first strategy, which has the drawback of requiring the cell grant access. bq. Did you stop pursuing the READ_INVISIBLE priv aproach? Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. was (Author: apurtell): bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. And, cell ACLs become largely useless, since we early out at CF checks, they can't grant exception access unless using the cell-first strategy, which has the drawback of requiring the cell grant access. bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986124#comment-13986124 ] Andrew Purtell edited comment on HBASE-11077 at 4/30/14 9:42 PM: - bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. And, cell ACLs become largely useless, since we early out at CF checks, they can't grant exception access unless using the cell-first strategy, which has the drawback of requiring the cell grant access. bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. was (Author: apurtell): bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986124#comment-13986124 ] Andrew Purtell commented on HBASE-11077: bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Then we break compatibility from 0.98.1 to 0.98.2. bq. I think early_out should be set to true by default, so that it is least surprise to the admin. Check out the other subtask and let's figure out what makes sense for 0.99+. It's not a trivial discussion. > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[
https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986112#comment-13986112
]
Enis Soztutar commented on HBASE-11077:
---
Sorry i missed this, was following the parent jira.
bq. This setting is AccessControlConstants.CF_ATTRIBUTE_EARLY_OUT
("hbase.security.access.early_out"), a boolean. Set to "true" for backwards
compatible behavior.
I think early_out should be set to true by default, so that it is least
surprise to the admin. Otherwise, she has to know about this particular
configuration option, and turn this on. In the other case, where you would not
have perm to table, but some specific cells, then it is fine to require and
explicit parameter. It should not be a common use case.
Did you stop pursuing the READ_INVISIBLE priv aproach?
> [AccessController] Restore compatible early-out access denial
> -
>
> Key: HBASE-11077
> URL: https://issues.apache.org/jira/browse/HBASE-11077
> Project: HBase
> Issue Type: Sub-task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Critical
> Fix For: 0.99.0, 0.98.2
>
> Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch,
> HBASE-11077.patch
>
>
> See parent for the whole story.
> For 0.98, to start, just put back the early out that was removed in 0.98.0
> and allow it to be overridden with a table attribute.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-10993) Deprioritize long-running scanners
[ https://issues.apache.org/jira/browse/HBASE-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-10993: Attachment: HBASE-10993-v4.patch > Deprioritize long-running scanners > -- > > Key: HBASE-10993 > URL: https://issues.apache.org/jira/browse/HBASE-10993 > Project: HBase > Issue Type: Sub-task >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.99.0 > > Attachments: HBASE-10993-v0.patch, HBASE-10993-v1.patch, > HBASE-10993-v2.patch, HBASE-10993-v3.patch, HBASE-10993-v4.patch > > > Currently we have a single call queue that serves all the "normal user" > requests, and the requests are executed in FIFO. > When running map-reduce jobs and user-queries on the same machine, we want to > prioritize the user-queries. > Without changing too much code, and not having the user giving hints, we can > add a “vtime” field to the scanner, to keep track from how long is running. > And we can replace the callQueue with a priorityQueue. In this way we can > deprioritize long-running scans, the longer a scan request lives the less > priority it gets. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11077) [AccessController] Restore compatible early-out access denial
[ https://issues.apache.org/jira/browse/HBASE-11077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986074#comment-13986074 ] Andrew Purtell commented on HBASE-11077: Going to commit in a couple of hours > [AccessController] Restore compatible early-out access denial > - > > Key: HBASE-11077 > URL: https://issues.apache.org/jira/browse/HBASE-11077 > Project: HBase > Issue Type: Sub-task >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Critical > Fix For: 0.99.0, 0.98.2 > > Attachments: HBASE-11077.patch, HBASE-11077.patch, HBASE-11077.patch, > HBASE-11077.patch > > > See parent for the whole story. > For 0.98, to start, just put back the early out that was removed in 0.98.0 > and allow it to be overridden with a table attribute. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-8332) Add truncate as HMaster method
[ https://issues.apache.org/jira/browse/HBASE-8332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-8332: --- Attachment: HBASE-8332-v4.patch > Add truncate as HMaster method > -- > > Key: HBASE-8332 > URL: https://issues.apache.org/jira/browse/HBASE-8332 > Project: HBase > Issue Type: Improvement > Components: master >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Attachments: HBASE-8332-v0.patch, HBASE-8332-v2.patch, > HBASE-8332-v3.patch, HBASE-8332-v4.patch, HBASE-8332.draft.patch > > > Currently truncate and truncate_preserve are only shell functions, and > implemented as deleteTable() + createTable(). > Using ACLs the user running truncate, must have rights to create a table and > only "global granted" users can create tables. > Add truncate() and truncatePreserve() to HBaseAdmin/HMaster with its own ACL > check. > https://reviews.apache.org/r/15835/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HBASE-11102) Document JDK versions supported by each release
Enis Soztutar created HBASE-11102: - Summary: Document JDK versions supported by each release Key: HBASE-11102 URL: https://issues.apache.org/jira/browse/HBASE-11102 Project: HBase Issue Type: Sub-task Reporter: Enis Soztutar Fix For: 0.99.0 We can make use of a JDK version x HBase version matrix to explain which JDK version is supported and required. 0.94, 0.96, and 0.98 releases all support JDK6 and JDK7. For 1.0, there is a discussion thread to decide whether to drop JDK6 support. There has been some work to support JDK8. We can also document that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986053#comment-13986053 ] Jean-Daniel Cryans commented on HBASE-10958: Sorry, went to do something else, lemme get that in (with HBASE-11008 first). > [dataloss] Bulk loading with seqids can prevent some log entries from being > replayed > > > Key: HBASE-10958 > URL: https://issues.apache.org/jira/browse/HBASE-10958 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.18 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.99.0, 0.98.2, 0.96.3, 0.94.20 > > Attachments: HBASE-10958-0.94.patch, > HBASE-10958-less-intrusive-hack-0.96.patch, > HBASE-10958-quick-hack-0.96.patch, HBASE-10958-v2.patch, > HBASE-10958-v3.patch, HBASE-10958.patch > > > We found an issue with bulk loads causing data loss when assigning sequence > ids (HBASE-6630) that is triggered when replaying recovered edits. We're > nicknaming this issue *Blindspot*. > The problem is that the sequence id given to a bulk loaded file is higher > than those of the edits in the region's memstore. When replaying recovered > edits, the rule to skip some of them is that they have to be _lower than the > highest sequence id_. In other words, the edits that have a sequence id lower > than the highest one in the store files *should* have also been flushed. This > is not the case with bulk loaded files since we now have an HFile with a > sequence id higher than unflushed edits. > The log recovery code takes this into account by simply skipping the bulk > loaded files, but this "bulk loaded status" is *lost* on compaction. The > edits in the logs that have a sequence id lower than the bulk loaded file > that got compacted are put in a blind spot and are skipped during replay. > Here's the easiest way to recreate this issue: > - Create an empty table > - Put one row in it (let's say it gets seqid 1) > - Bulk load one file (it gets seqid 2). I used ImporTsv and set > hbase.mapreduce.bulkload.assign.sequenceNumbers. > - Bulk load a second file the same way (it gets seqid 3). > - Major compact the table (the new file has seqid 3 and isn't considered > bulk loaded). > - Kill the region server that holds the table's region. > - Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 3 makes us believe that > everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10958) [dataloss] Bulk loading with seqids can prevent some log entries from being replayed
[ https://issues.apache.org/jira/browse/HBASE-10958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986050#comment-13986050 ] Lars Hofhansl commented on HBASE-10958: --- Are you waiting for me to commit [~jdcryans]? Just making sure we're not mutually waiting. > [dataloss] Bulk loading with seqids can prevent some log entries from being > replayed > > > Key: HBASE-10958 > URL: https://issues.apache.org/jira/browse/HBASE-10958 > Project: HBase > Issue Type: Bug >Affects Versions: 0.96.2, 0.98.1, 0.94.18 >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans >Priority: Blocker > Fix For: 0.99.0, 0.98.2, 0.96.3, 0.94.20 > > Attachments: HBASE-10958-0.94.patch, > HBASE-10958-less-intrusive-hack-0.96.patch, > HBASE-10958-quick-hack-0.96.patch, HBASE-10958-v2.patch, > HBASE-10958-v3.patch, HBASE-10958.patch > > > We found an issue with bulk loads causing data loss when assigning sequence > ids (HBASE-6630) that is triggered when replaying recovered edits. We're > nicknaming this issue *Blindspot*. > The problem is that the sequence id given to a bulk loaded file is higher > than those of the edits in the region's memstore. When replaying recovered > edits, the rule to skip some of them is that they have to be _lower than the > highest sequence id_. In other words, the edits that have a sequence id lower > than the highest one in the store files *should* have also been flushed. This > is not the case with bulk loaded files since we now have an HFile with a > sequence id higher than unflushed edits. > The log recovery code takes this into account by simply skipping the bulk > loaded files, but this "bulk loaded status" is *lost* on compaction. The > edits in the logs that have a sequence id lower than the bulk loaded file > that got compacted are put in a blind spot and are skipped during replay. > Here's the easiest way to recreate this issue: > - Create an empty table > - Put one row in it (let's say it gets seqid 1) > - Bulk load one file (it gets seqid 2). I used ImporTsv and set > hbase.mapreduce.bulkload.assign.sequenceNumbers. > - Bulk load a second file the same way (it gets seqid 3). > - Major compact the table (the new file has seqid 3 and isn't considered > bulk loaded). > - Kill the region server that holds the table's region. > - Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 3 makes us believe that > everything that came before it was flushed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10993) Deprioritize long-running scanners
[
https://issues.apache.org/jira/browse/HBASE-10993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986045#comment-13986045
]
Hadoop QA commented on HBASE-10993:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642688/HBASE-10993-v3.patch
against trunk revision .
ATTACHMENT ID: 12642688
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 8 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestBoundedPriorityBlockingQueue
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9436//console
This message is automatically generated.
> Deprioritize long-running scanners
> --
>
> Key: HBASE-10993
> URL: https://issues.apache.org/jira/browse/HBASE-10993
> Project: HBase
> Issue Type: Sub-task
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
>Priority: Minor
> Fix For: 0.99.0
>
> Attachments: HBASE-10993-v0.patch, HBASE-10993-v1.patch,
> HBASE-10993-v2.patch, HBASE-10993-v3.patch
>
>
> Currently we have a single call queue that serves all the "normal user"
> requests, and the requests are executed in FIFO.
> When running map-reduce jobs and user-queries on the same machine, we want to
> prioritize the user-queries.
> Without changing too much code, and not having the user giving hints, we can
> add a “vtime” field to the scanner, to keep track from how long is running.
> And we can replace the callQueue with a priorityQueue. In this way we can
> deprioritize long-running scans, the longer a scan request lives the less
> priority it gets.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13986006#comment-13986006
]
stack commented on HBASE-11099:
---
Sorry, missed your comments lads. Thanks for clarification. Let me try and
make a test to catch the hole so I know I'm putting plug in right place (smile).
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7987) Snapshot Manifest file instead of multiple empty files
[
https://issues.apache.org/jira/browse/HBASE-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985991#comment-13985991
]
Hadoop QA commented on HBASE-7987:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12642669/HBASE-7987-v5.patch
against trunk revision .
ATTACHMENT ID: 12642669
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 40 new
or modified tests.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9434//console
This message is automatically generated.
> Snapshot Manifest file instead of multiple empty files
> --
>
> Key: HBASE-7987
> URL: https://issues.apache.org/jira/browse/HBASE-7987
> Project: HBase
> Issue Type: Improvement
> Components: snapshots
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.99.0
>
> Attachments: HBASE-7987-v0.patch, HBASE-7987-v1.patch,
> HBASE-7987-v2.patch, HBASE-7987-v2.sketch, HBASE-7987-v3.patch,
> HBASE-7987-v4.patch, HBASE-7987-v5.patch, HBASE-7987.sketch
>
>
> Currently taking a snapshot means creating one empty file for each file in
> the source table directory, plus copying the .regioninfo file for each
> region, the table descriptor file and a snapshotInfo file.
> during the restore or snapshot verification we traverse the filesystem
> (fs.listStatus()) to find the snapshot files, and we open the .regioninfo
> files to get the information.
> to avoid hammering the NameNode and having lots of empty files, we can use a
> manifest file that contains the list of files and information that we need.
> To keep the RS parallelism that we have, each RS can write its own manifest.
> {code}
> message SnapshotDescriptor {
> required string name;
> optional string table;
> optional int64 creationTime;
> optional Type type;
> optional int32 version;
> }
> message SnapshotRegionManifest {
> optional int32 version;
> required RegionInfo regionInfo;
> repeated FamilyFiles familyFiles;
> message StoreFile {
> required string name;
> optional Reference reference;
> }
> message FamilyFiles {
> required bytes familyName;
> repeated StoreFile storeFiles;
> }
> }
> {code}
> {code}
> /hbase/.snapshot/
> /hbase/.snapshot//snapshotInfo
> /hbase/.snapshot//
> /hbase/.snapshot///tableInfo
> /hbase/.snapshot///regionManifest(.n)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-10926) Use global procedure to flush table memstore cache
[ https://issues.apache.org/jira/browse/HBASE-10926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985966#comment-13985966 ] stack commented on HBASE-10926: --- bq. since you have already done the patch, and all the code seems to be isolated in procedure/flush i guess is fine if you just use acquireBarrier() to do the flush I'm w/ [~mbertozzi] here. Can file an issue to refactor once the 'state change' notification system goes in. > Use global procedure to flush table memstore cache > -- > > Key: HBASE-10926 > URL: https://issues.apache.org/jira/browse/HBASE-10926 > Project: HBase > Issue Type: Improvement > Components: Admin >Affects Versions: 0.96.2, 0.98.1 >Reporter: Jerry He >Assignee: Jerry He > Fix For: 0.99.0 > > Attachments: HBASE-10926-trunk-v1.patch > > > Currently, user can trigger table flush through hbase shell or HBaseAdmin > API. To flush the table cache, each region server hosting the regions is > contacted and flushed sequentially, which is less efficient. > In HBase snapshot global procedure is used to coordinate and flush the > regions in a distributed way. > Let's provide a distributed table flush for general use. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11099) Two situations where we could open a region with smaller sequence number
[
https://issues.apache.org/jira/browse/HBASE-11099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13985961#comment-13985961
]
stack commented on HBASE-11099:
---
Calling a sync while writes are blocked on this store would not be the end of
the world. When the sync comes out the otherside, I know all of my region
edits have been pushed through ahead of it. Syncs happen at a rate of many per
ms -- in last measure over in hbase-10156 it had us syncing over a 15 minute
period at almost 4 times a ms -- so would be a bit of a speed bump but probably
not too bad.
> Two situations where we could open a region with smaller sequence number
>
>
> Key: HBASE-11099
> URL: https://issues.apache.org/jira/browse/HBASE-11099
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.99.0
>Reporter: Jeffrey Zhong
> Fix For: 0.99.0
>
>
> Recently I happened to run into code where we potentially could open region
> with smaller sequence number:
> 1) Inside function: HRegion#internalFlushcache. This is due to we change the
> way WAL Sync where we use late binding(assign sequence number right before
> wal sync).
> The flushSeqId may less than the change sequence number included in the flush
> which may cause later region opening code to use a smaller than expected
> sequence number when we reopen the region.
> {code}
> flushSeqId = this.sequenceId.incrementAndGet();
> ...
> mvcc.waitForRead(w);
> {code}
> 2) HRegion#replayRecoveredEdits where we have following code:
> {code}
> ...
> if (coprocessorHost != null) {
> status.setStatus("Running pre-WAL-restore hook in coprocessors");
> if (coprocessorHost.preWALRestore(this.getRegionInfo(), key,
> val)) {
> // if bypass this log entry, ignore it ...
> continue;
> }
> }
> ...
> currentEditSeqId = key.getLogSeqNum();
> {code}
> If coprocessor skip some tail WALEdits, then the function will return smaller
> currentEditSeqId. In the end, a region may also open with a smaller sequence
> number. This may cause data loss because Master may record a larger flushed
> sequence Id and some WALEdits maybe skipped during recovery if the region
> fail again.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
