[jira] [Commented] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387214#comment-15387214 ] Anoop Sam John commented on HBASE-14921: [~anastas] So you got the comment abt the heap sizing for the CellArrayEntry ? I mean to say here just consider the extra heap space need for adding an item to this CellArray structure. Do not consider the heap need for that object as such. That can be done within that object's heap size calc. Finally when we have to find whole heap size of a given CellArray structure we can + for(Each entry in CellArray) CellArray Entry overhead + Cell object heap > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387209#comment-15387209
]
Anoop Sam John commented on HBASE-16257:
You mean you want to just remove this config and go with a fixed path? After
this change, we will append this config value under the hbase root path and
make the full path. So that is infact a behave change no?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387187#comment-15387187
]
ramkrishna.s.vasudevan commented on HBASE-15638:
+1. Thanks Stack. So the only thing pending is the Endpoint break?
Also once we are moving to say a newer version of protobuf and the proto file
have to be regenerated still we break src compatibility but wire compatibility
is fine.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387188#comment-15387188
]
Jerry He commented on HBASE-16257:
--
We will deprecate 'hbase.bulkload.staging.dir'. I was thinking just ignoring
it in the new server, maybe print a message.
This will not cause any BC problem on its own because the directory is only
relevant internally to the region server. Right?
But we still need a deprecation cycle?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16205) When Cells are not copied to MSLAB, deep clone it while adding to Memstore
[ https://issues.apache.org/jira/browse/HBASE-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387157#comment-15387157 ] Anoop Sam John commented on HBASE-16205: The upsert code path rely on Object equality check to make sure we wont remove the just added cell. That was broken by v2 patch and V3 corrects that. Also added back the old ShareableMemory interface. We will create Cell impls of this type from the Codec ByteBufferedKeyValueDecoder. Here all Cells backed by same buffer chunk where the RPC requests are read into. In such case only we need this deep copy (if MSLAB copy not happened). We are not blind doing this deep copy now (as per V3 patch). > When Cells are not copied to MSLAB, deep clone it while adding to Memstore > -- > > Key: HBASE-16205 > URL: https://issues.apache.org/jira/browse/HBASE-16205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16205.patch, HBASE-16205_V2.patch, > HBASE-16205_V3.patch > > > This is imp after HBASE-15180 optimization. After that we the cells flowing > in write path will be backed by the same byte[] where the RPC read the > request into. By default we have MSLAB On and so we have a copy operation > while adding Cells to memstore. This copy might not be there if > 1. MSLAB is turned OFF > 2. Cell size is more than a configurable max size. This defaults to 256 KB > 3. If the operation is Append/Increment. > In such cases, we should just clone the Cell into a new byte[] and then add > to memstore. Or else we keep referring to the bigger byte[] chunk for longer > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16205) When Cells are not copied to MSLAB, deep clone it while adding to Memstore
[ https://issues.apache.org/jira/browse/HBASE-16205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16205: --- Attachment: HBASE-16205_V3.patch > When Cells are not copied to MSLAB, deep clone it while adding to Memstore > -- > > Key: HBASE-16205 > URL: https://issues.apache.org/jira/browse/HBASE-16205 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Critical > Fix For: 2.0.0 > > Attachments: HBASE-16205.patch, HBASE-16205_V2.patch, > HBASE-16205_V3.patch > > > This is imp after HBASE-15180 optimization. After that we the cells flowing > in write path will be backed by the same byte[] where the RPC read the > request into. By default we have MSLAB On and so we have a copy operation > while adding Cells to memstore. This copy might not be there if > 1. MSLAB is turned OFF > 2. Cell size is more than a configurable max size. This defaults to 256 KB > 3. If the operation is Append/Increment. > In such cases, we should just clone the Cell into a new byte[] and then add > to memstore. Or else we keep referring to the bigger byte[] chunk for longer > time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387143#comment-15387143
]
stack commented on HBASE-15638:
---
Uploaded patch that changes all pb references from com.google.protobuf to
org.apache.hbase.shaded.com.google.protobuf. Includes "HBASE-16263 Move all to
do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol"
so we compile still.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15638:
--
Attachment: HBASE-15638.master.001.patch
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, HBASE-15638.master.001.patch,
> as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v7.txt Patch v7 fixes the two hashCode() warnings and ambiguous getConf() call. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v2.txt, 14123-master.v3.txt, > 14123-master.v5.txt, 14123-master.v6.txt, 14123-master.v7.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387119#comment-15387119
]
Anoop Sam John commented on HBASE-16257:
So now this config 'hbase.bulkload.staging.dir' expects a full path and you
want to change the meaning of that also? When we say move it under hbase root
dir, it has to be always under that. So BC issue?
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387113#comment-15387113 ] Ted Yu commented on HBASE-14123: w.r.t. BackupInfo defines equals and uses Object.hashCode() , there're 15 non-transient fields in BackupInfo, including a Map. Not sure if it is worth it to override hashCode() for BackupInfo. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v2.txt, 14123-master.v3.txt, > 14123-master.v5.txt, 14123-master.v6.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16257) Move staging dir to be under hbase root dir
[
https://issues.apache.org/jira/browse/HBASE-16257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387103#comment-15387103
]
Anoop Sam John commented on HBASE-16257:
+1 for doing this.
> Move staging dir to be under hbase root dir
> ---
>
> Key: HBASE-16257
> URL: https://issues.apache.org/jira/browse/HBASE-16257
> Project: HBase
> Issue Type: Sub-task
>Reporter: Jerry He
>Assignee: Jerry He
>Priority: Minor
> Fix For: 2.0.0
>
>
> The hbase.bulkload.staging.dir defaults to hbase.fs.tmp.dir which then
> defaults to
> {code}
> public static final String DEFAULT_TEMPORARY_HDFS_DIRECTORY = "/user/"
> + System.getProperty("user.name") + "/hbase-staging";
> {code}
> This default would have problem on local file system standalone case.
> We can move the staging dir to be under hbase.rootdir. We are bringing
> secure bulkload to the core. It makes sense to bring it under core control as
> well, instead of an optional property.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang updated HBASE-16261: - Status: Patch Available (was: Open) > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16261-V1.patch > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang updated HBASE-16261: - Attachment: HBASE-16261-V1.patch > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16261-V1.patch > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15227) HBase Backup Phase 3: Fault tolerance (client/server) support
[ https://issues.apache.org/jira/browse/HBASE-15227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-15227: -- Attachment: HBASE-15277-v1.patch patch v1. requires HBASE-16181 > HBase Backup Phase 3: Fault tolerance (client/server) support > - > > Key: HBASE-15227 > URL: https://issues.apache.org/jira/browse/HBASE-15227 > Project: HBase > Issue Type: Task >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Labels: backup > Fix For: 2.0.0 > > Attachments: HBASE-15277-v1.patch > > > System must be tolerant to faults. Backup operations MUST be atomic (no > partial completion state in system table) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14921) Memory optimizations
[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387050#comment-15387050
]
Anoop Sam John commented on HBASE-14921:
I got your argument abt dynamic decision making for compaction vs flatten only.
Worry was how costly it will be to do another round of scan. It involve SQM
and a Heap with many compares. It is not so cheap. As of now you are not
adding the CellChunkMap based flattening. Things will be much worse, when we
have that. We get rid of Cell objects as such in that flattened form. But then
this scan need Cells to be created again. Means lots of garbage.
May be in a use case where duplicates are possible, chances are there that
there are not enough #duplicate records so that we get some real benefit out of
compaction. So then flatten would be enough. So an extra scan may make sense
there.
But when the use case is like some thing of time series data, where we really
dont expect duplicates/updates, it might be better to turn off compaction and
do only flatten.
Again flatten to CellChunkMap would be ideal as that will release and reduce
heap memory footprint for this memstore considerably. CellArrayMap, yes it
reduces but not much. In your usecase, the max adv you get because of the
compaction as many cells will get removed.
My another concern is regarding the fact that in this memstore only the tail of
the pipeline getting flushed to disk when a flush request comes. In 1st
version it was like always the compaction happens. So all chances that the tail
of pipeline is much bigger sized and so that much data gets flushed. Now when
compaction is not at all happening and we do have many small sized segments in
pipeline, it would have been better to flush all the segments to disk that
making small sized flushes. I raised this concern at first step also. But then
the counter was that the compaction happens always but now it is not the case.
Ya Ram will come up with al perf analysis.
bq.We are now holding more in the memory and thus having more possibility to
let a cell "die" in memory.
JFYI.. There is a periodic memstore flush checking. If we accumulate more than
30 million edits in memstore, we will flush
{code}
if (this.maxFlushedSeqId > 0
&& (this.maxFlushedSeqId + this.flushPerChanges <
this.mvcc.getReadPoint())) {
whyFlush.append("more than max edits, " + this.flushPerChanges + ", since
last flush");
return true;
}
{code}
This flushPerChanges is configurable btw.
The second check here is time based. If we have not flushed memstore for quite
some time, we will make a flush. This time def to 1 hr.
Just saying for your consideration.
> Memory optimizations
>
>
> Key: HBASE-14921
> URL: https://issues.apache.org/jira/browse/HBASE-14921
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Eshcar Hillel
>Assignee: Anastasia Braginsky
> Attachments: CellBlocksSegmentInMemStore.pdf,
> CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch,
> HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch,
> HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch,
> HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf,
> IntroductiontoNewFlatandCompactMemStore.pdf
>
>
> Memory optimizations including compressed format representation and offheap
> allocations
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387041#comment-15387041
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 1s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.1/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 45 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 51s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 54s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 13m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 4m
12s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 3s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 43s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 19m 55s
{color} | {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 2
new + 4 unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 19m 55s
{color} | {color:red} root-jdk1.7.0_80 with JDK v1.7.0_80 generated 2 new + 30
unchanged - 2 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 3m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
2s {color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 621 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 18s
[jira] [Commented] (HBASE-14921) Memory optimizations
[
https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387034#comment-15387034
]
Hadoop QA commented on HBASE-14921:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 2s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 2s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 3s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 44s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 44s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 2s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 13s
{color} | {color:green} hbase-shell in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 188m 34s {color}
| {color:black}
[jira] [Commented] (HBASE-16265) BulkLoadSuite#Wide Row Bulk Load fails in master branch
[
https://issues.apache.org/jira/browse/HBASE-16265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387014#comment-15387014
]
Jerry He commented on HBASE-16265:
--
This may be related to the commit by HBASE-13701.
But the root cause is HBASE-15291. See my comment on that JIRA.
The refactoring makes it happen more often in the tests since secure bulk load
is the default now.
I think we need to revert HBASE-15291
See:
https://builds.apache.org/job/PreCommit-HBASE-Build/2437/testReport/org.apache.hadoop.hbase.mapreduce/TestLoadIncrementalHFilesSplitRecovery/testSplitTmpFileCleanUp/
Towards the end of the output:
{noformat}
2016-06-30 07:48:23,454 ERROR
[RpcServer.FifoWFPBQ.default.handler=3,queue=0,port=56734]
regionserver.HRegion(5508): There was a partial failure due to IO when
attempting to load family_0004 :
hdfs://localhost:52668/user/jenkins/test-data/34d5b872-06e9-4844-a90b-4635a736994d/splitTmpFileCleanUp/splitTmpFileCleanUp2/family_0004/.tmp/2f32f7a8e89f4dbcbbe38b095e06352e.top
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
at org.apache.hadoop.hdfs.DFSClient.getFileInfo(DFSClient.java:2113)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1305)
at
org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
at
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.isFile(SecureBulkLoadManager.java:410)
at
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$SecureBulkLoadListener.prepareBulkLoad(SecureBulkLoadManager.java:347)
at
org.apache.hadoop.hbase.regionserver.HRegion.bulkLoadHFiles(HRegion.java:5479)
at
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:249)
at
org.apache.hadoop.hbase.regionserver.SecureBulkLoadManager$1.run(SecureBulkLoadManager.java:234)
{noformat}
> BulkLoadSuite#Wide Row Bulk Load fails in master branch
> ---
>
> Key: HBASE-16265
> URL: https://issues.apache.org/jira/browse/HBASE-16265
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>
> See
> https://builds.apache.org/job/HBase-TRUNK_matrix/1265/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.spark/BulkLoadSuite/Wide_Row_Bulk_Load__Test_multi_family_and_multi_column_tests_with_one_column_family_with_custom_configs_plus_multi_region/
> :
> {code}
> File does not exist:
> /tmp/junit811617169708528768/junit7419829633245108670/f1/b934f7ee0d414de5ab3457d057b9df64
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at
> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16262) Update RegionsInTransition UI for branch-1
[
https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387000#comment-15387000
]
Hadoop QA commented on HBASE-16262:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
7s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 37s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} branch-1 passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
17m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 6s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 117m 39s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.TestMasterStatusServlet |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12819211/HBASE-16262-branch-1.patch
|
| JIRA Issue | HBASE-16262 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | branch-1 / d86f8a4 |
|
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386944#comment-15386944 ] Ted Yu commented on HBASE-14123: TestAssignmentListener is marked flaky test in trunk build. Let me check the other test failures. > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v2.txt, 14123-master.v3.txt, > 14123-master.v5.txt, 14123-master.v6.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16221) Thrift server drops connection on long scans
[ https://issues.apache.org/jira/browse/HBASE-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph reassigned HBASE-16221: -- Assignee: Joseph > Thrift server drops connection on long scans > > > Key: HBASE-16221 > URL: https://issues.apache.org/jira/browse/HBASE-16221 > Project: HBase > Issue Type: Bug > Components: Thrift >Affects Versions: 2.0.0, 1.2.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Joseph > > Thrift servers use connection cache and we drop connections after > hbase.thrift.connection.max-idletime milliseconds from the last time a > connection object was accessed. However, we never update this last accessed > time on scan path. > By default, this will cause scanners to fail after 10 minutes, if the > underlying connection object is not being used along other operation paths > (like put). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16265) BulkLoadSuite#Wide Row Bulk Load fails in master branch
Ted Yu created HBASE-16265:
--
Summary: BulkLoadSuite#Wide Row Bulk Load fails in master branch
Key: HBASE-16265
URL: https://issues.apache.org/jira/browse/HBASE-16265
Project: HBase
Issue Type: Bug
Reporter: Ted Yu
See
https://builds.apache.org/job/HBase-TRUNK_matrix/1265/jdk=latest1.8,label=yahoo-not-h2/testReport/org.apache.hadoop.hbase.spark/BulkLoadSuite/Wide_Row_Bulk_Load__Test_multi_family_and_multi_column_tests_with_one_column_family_with_custom_configs_plus_multi_region/
:
{code}
File does not exist:
/tmp/junit811617169708528768/junit7419829633245108670/f1/b934f7ee0d414de5ab3457d057b9df64
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:71)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:61)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1828)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1712)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:587)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:365)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049) at
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:422) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16264) Figure how to deal with endpoints and shaded pb
stack created HBASE-16264: - Summary: Figure how to deal with endpoints and shaded pb Key: HBASE-16264 URL: https://issues.apache.org/jira/browse/HBASE-16264 Project: HBase Issue Type: Sub-task Components: Coprocessors, Protobufs Reporter: stack Assignee: stack Come up w/ a migration plan for coprocessor endpoints when our pb is shaded. Would be sweet if could make it so all just worked. At worst, come up w/ a prescription for how to migrate existing CPs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-15638:
--
Assignee: stack
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Assignee: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
[ https://issues.apache.org/jira/browse/HBASE-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386926#comment-15386926 ] stack commented on HBASE-16263: --- nvm. Its a PITA. The hadoop plugin that runs the protoc is primitive. It allows multiple imports but not output dirs or multiple executions. We could make a test profile. Would have to have one profile for usual proto files and another for tests. You'd have to invoke them distinctly. Then would have to doc it. I think it too much of a PITA. [~eclark]? > Move all to do w/ protobuf -- *.proto files and generated classes -- under > hbase-protocol > - > > Key: HBASE-16263 > URL: https://issues.apache.org/jira/browse/HBASE-16263 > Project: HBase > Issue Type: Sub-task > Components: Protobufs >Reporter: stack >Assignee: stack > Attachments: HBASE-16263.master.001.patch > > > To better contain our protobuf exposure, keep all to do w/ protobuffing > inside the hbase-protocol module. > This task moves rest, examples, and server proto files that do testing and CP > endpoints etc., under hbase-protocol. The patch is big but it is all just > moving files. > interesting to see the copy-paste being done everywhere of proto generation. > For this reason alone, it is better to have all the protos in the one place > beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386899#comment-15386899 ] Ted Yu commented on HBASE-16181: w.r.t. the new test, on Mac it took 113.497 sec. Can the test be simplified ? Meaning, without the fix in ClientSnapshotDescriptionUtils.java, the snapshot of hbase:backup would trigger exception. Otherwise the snapshot would be successful, resulting in assertions to pass. So the test just verifies that snapshot on hbase:backup is allowed. > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16142) Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir
[
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386867#comment-15386867
]
Hadoop QA commented on HBASE-16142:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
20s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 32s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 13s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 129m 47s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestFromClientSide |
| | org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor |
| | org.apache.hadoop.hbase.util.TestRegionMover |
| | org.apache.hadoop.hbase.util.TestMiniClusterLoadParallel |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12819041/HBASE-16142.master.003.patch
|
| JIRA Issue | HBASE-16142 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386857#comment-15386857
]
stack commented on HBASE-15638:
---
Thank you [~ghelmling] That helps. I think best thing for me to do is try an
endpoint and see how it breaks.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386851#comment-15386851
]
Mikhail Antonov commented on HBASE-16232:
-
no didn't see anything funny about it
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386849#comment-15386849 ] Vladimir Rodionov commented on HBASE-16181: --- Ok. > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
[ https://issues.apache.org/jira/browse/HBASE-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386847#comment-15386847 ] stack commented on HBASE-16263: --- Let me do that. > Move all to do w/ protobuf -- *.proto files and generated classes -- under > hbase-protocol > - > > Key: HBASE-16263 > URL: https://issues.apache.org/jira/browse/HBASE-16263 > Project: HBase > Issue Type: Sub-task > Components: Protobufs >Reporter: stack >Assignee: stack > Attachments: HBASE-16263.master.001.patch > > > To better contain our protobuf exposure, keep all to do w/ protobuffing > inside the hbase-protocol module. > This task moves rest, examples, and server proto files that do testing and CP > endpoints etc., under hbase-protocol. The patch is big but it is all just > moving files. > interesting to see the copy-paste being done everywhere of proto generation. > For this reason alone, it is better to have all the protos in the one place > beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16217) Identify calling user in ObserverContext
[
https://issues.apache.org/jira/browse/HBASE-16217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386837#comment-15386837
]
Hadoop QA commented on HBASE-16217:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 52s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 24s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 26s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 102m 50s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
1s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 188m 30s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| Timed out junit tests |
org.apache.hadoop.hbase.util.TestMiniClusterLoadSequential |
| | org.apache.hadoop.hbase.util.TestFSUtils |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386819#comment-15386819
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.2.1/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 45 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 23s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 19s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m
48s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
0s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 3m
4s {color} | {color:green} the patch passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped branch modules with no Java source: . {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 25s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 7s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 26s
{color} | {color:red} hbase-server-jdk1.7.0_80 with JDK v1.7.0_80 generated 2
new + 4 unchanged - 2 fixed = 6 total (was 6) {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 26s
{color} | {color:red} root-jdk1.7.0_80 with JDK v1.7.0_80 generated 2 new + 30
unchanged - 2 fixed = 32 total (was 32) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 12m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
2s {color} | {color:green} There were no new shelldocs issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 621 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 14s
{color}
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386820#comment-15386820
]
Gary Helmling commented on HBASE-15638:
---
[~saint@gmail.com], I'm not sure how to make this work with coprocessor
endpoints.
For endpoints, the user will have:
# Created a .proto file with the service definition for their endpoint
# Run their own version of protoc to do the code gen for the service
# The generated PB messages for request/response, and, as you note, the
RpcController and RpcCallback in the service method signatures will be in the
com.google.protobuf package
# The endpoint jar is loaded in HBase
# The ServerRpcController we pass through for the endpoint invocation will
extend our shaded RpcController
So the only way I see this working is if we force endpoint implementors to run
a preprocessor step to shade the endpoint jar prior to step #4. Maybe that
would work, but it seems pretty cumbersome. Or else we somehow avoid shading
ServerRpcController.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
[ https://issues.apache.org/jira/browse/HBASE-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386803#comment-15386803 ] Elliott Clark commented on HBASE-16263: --- Should the protobuf files for test be moved into src/test/protobuf ? > Move all to do w/ protobuf -- *.proto files and generated classes -- under > hbase-protocol > - > > Key: HBASE-16263 > URL: https://issues.apache.org/jira/browse/HBASE-16263 > Project: HBase > Issue Type: Sub-task > Components: Protobufs >Reporter: stack >Assignee: stack > Attachments: HBASE-16263.master.001.patch > > > To better contain our protobuf exposure, keep all to do w/ protobuffing > inside the hbase-protocol module. > This task moves rest, examples, and server proto files that do testing and CP > endpoints etc., under hbase-protocol. The patch is big but it is all just > moving files. > interesting to see the copy-paste being done everywhere of proto generation. > For this reason alone, it is better to have all the protos in the one place > beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
[ https://issues.apache.org/jira/browse/HBASE-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386800#comment-15386800 ] stack commented on HBASE-16263: --- Any chance of a +1 [~busbey]? It is just moving files apart from a bit of pom messing and updated doc. > Move all to do w/ protobuf -- *.proto files and generated classes -- under > hbase-protocol > - > > Key: HBASE-16263 > URL: https://issues.apache.org/jira/browse/HBASE-16263 > Project: HBase > Issue Type: Sub-task > Components: Protobufs >Reporter: stack >Assignee: stack > Attachments: HBASE-16263.master.001.patch > > > To better contain our protobuf exposure, keep all to do w/ protobuffing > inside the hbase-protocol module. > This task moves rest, examples, and server proto files that do testing and CP > endpoints etc., under hbase-protocol. The patch is big but it is all just > moving files. > interesting to see the copy-paste being done everywhere of proto generation. > For this reason alone, it is better to have all the protos in the one place > beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
[ https://issues.apache.org/jira/browse/HBASE-16263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16263: -- Attachment: HBASE-16263.master.001.patch > Move all to do w/ protobuf -- *.proto files and generated classes -- under > hbase-protocol > - > > Key: HBASE-16263 > URL: https://issues.apache.org/jira/browse/HBASE-16263 > Project: HBase > Issue Type: Sub-task > Components: Protobufs >Reporter: stack >Assignee: stack > Attachments: HBASE-16263.master.001.patch > > > To better contain our protobuf exposure, keep all to do w/ protobuffing > inside the hbase-protocol module. > This task moves rest, examples, and server proto files that do testing and CP > endpoints etc., under hbase-protocol. The patch is big but it is all just > moving files. > interesting to see the copy-paste being done everywhere of proto generation. > For this reason alone, it is better to have all the protos in the one place > beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16263) Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol
stack created HBASE-16263: - Summary: Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol Key: HBASE-16263 URL: https://issues.apache.org/jira/browse/HBASE-16263 Project: HBase Issue Type: Sub-task Components: Protobufs Reporter: stack Assignee: stack To better contain our protobuf exposure, keep all to do w/ protobuffing inside the hbase-protocol module. This task moves rest, examples, and server proto files that do testing and CP endpoints etc., under hbase-protocol. The patch is big but it is all just moving files. interesting to see the copy-paste being done everywhere of proto generation. For this reason alone, it is better to have all the protos in the one place beside doc on how-to, etc. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15638) Shade protobuf
[
https://issues.apache.org/jira/browse/HBASE-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386794#comment-15386794
]
stack commented on HBASE-15638:
---
Made a subtask out of moving all protos under hbase-protocol.
Chatted w/ [~busbey] on approach. He noted that an outcome is that our build
should not have a protobuf.jar in it at all.
> Shade protobuf
> --
>
> Key: HBASE-15638
> URL: https://issues.apache.org/jira/browse/HBASE-15638
> Project: HBase
> Issue Type: Bug
> Components: Protobufs
>Reporter: stack
>Priority: Critical
> Attachments: 15638v2.patch, as.far.as.server.patch
>
>
> Shade protobufs so we can move to a different version without breaking the
> world. We want to get up on pb3 because it has unsafe methods that allow us
> save on copies; it also has some means of dealing with BBs so we can pass it
> offheap DBBs. We'll probably want to change PB3 to open it up some more too
> so we can stay offheap as we traverse PB. This issue comes of [~anoop.hbase]
> and [~ram_krish]'s offheaping of the readpath work.
> This change is mostly straight-forward but there are some tricky bits:
> # How to interface with HDFS? It wants its ByteStrings. Here in particular
> in FanOutOneBlockAsyncDFSOutputSaslHelper:
> {code}
> if (payload != null) {
> builder.setPayload(ByteString.copyFrom(payload));
> }
> {code}
> # [~busbey] also points out that we need to take care of endpoints done as
> pb. Test at least.
> Let me raise this one on the dev list too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386790#comment-15386790 ] Ted Yu commented on HBASE-16181: The license header in TestSystemTableBackup is not formatted in the same way as other files. Vlad: Mind fixing it ? Thanks > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386784#comment-15386784 ] Ted Yu commented on HBASE-14123: The initial mega patch was based on: git diff 7c54525c89bbbe0c66401813433bfb957e461eac..e8e51418101d41faf3f95fba03bfbe1271e4dbdb > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v2.txt, 14123-master.v3.txt, > 14123-master.v5.txt, 14123-master.v6.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16258) Wrong compaction and requests metrices on webUI
[ https://issues.apache.org/jira/browse/HBASE-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16258: - Summary: Wrong compaction and requests metrices on webUI (was: Some issues about metrices on webUI) > Wrong compaction and requests metrices on webUI > --- > > Key: HBASE-16258 > URL: https://issues.apache.org/jira/browse/HBASE-16258 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.17 >Reporter: Heng Chen > Attachments: B14D88EB-4793-4D0D-B2F3-60629C48D82E.png, > blockcache_off.png > > > 1. the first pic shows bad number about compaction metrics. > 2. blockcache_off.png shows when blockcache off, there is no requests > number on master UI, but the table have requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386724#comment-15386724
]
stack commented on HBASE-16232:
---
If you search he NN log, does it say anything interesting about the missing
file?
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16217) Identify calling user in ObserverContext
[
https://issues.apache.org/jira/browse/HBASE-16217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386718#comment-15386718
]
Hadoop QA commented on HBASE-16217:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 4s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
35m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 129m 50s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
45s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 198m 3s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| |
hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpointNoMaster
|
| |
[jira] [Commented] (HBASE-16095) Add priority to TableDescriptor and priority region open thread pool
[ https://issues.apache.org/jira/browse/HBASE-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386715#comment-15386715 ] stack commented on HBASE-16095: --- Thinking on this more, I think the change itself innocuous as long as it clear that the table marking is a 'guidance' only and not to be depended upon. If anything, it would be better if it were done more thoroughly with priority done on all admin functions. So, I think it fine going into 2.0. Suggest a release note that this 'guidance' only, that it can't be depended upon being honored given this a distributed system (see Gary comments). > Add priority to TableDescriptor and priority region open thread pool > > > Key: HBASE-16095 > URL: https://issues.apache.org/jira/browse/HBASE-16095 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16095-0.98.patch, HBASE-16095-0.98.patch, > hbase-16095_v0.patch, hbase-16095_v1.patch, hbase-16095_v2.patch, > hbase-16095_v3.patch > > > This is in the similar area with HBASE-15816, and also required with the > current secondary indexing for Phoenix. > The problem with P secondary indexes is that data table regions depend on > index regions to be able to make progress. Possible distributed deadlocks can > be prevented via custom RpcScheduler + RpcController configuration via > HBASE-11048 and PHOENIX-938. However, region opening also has the same > deadlock situation, because data region open has to replay the WAL edits to > the index regions. There is only 1 thread pool to open regions with 3 workers > by default. So if the cluster is recovering / restarting from scratch, the > deadlock happens because some index regions cannot be opened due to them > being in the same queue waiting for data regions to open (which waits for > RPC'ing to index regions which is not open). This is reproduced in almost all > Phoenix secondary index clusters (mutable table w/o transactions) that we > see. > The proposal is to have a "high priority" region opening thread pool, and > have the HTD carry the relative priority of a table. This maybe useful for > other "framework" level tables from Phoenix, Tephra, Trafodian, etc if they > want some specific tables to become online faster. > As a follow up patch, we can also take a look at how this priority > information can be used by the rpc scheduler on the server side or rpc > controller on the client side, so that we do not have to set priorities > manually per-operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386710#comment-15386710 ] Ted Yu edited comment on HBASE-16181 at 7/20/16 10:08 PM: -- As mentioned above, should black list be introduced so that more system tables are allowed to be snapshotted ? was (Author: yuzhih...@gmail.com): As mentioned above, should black list be introduced so that it is easier to add more system tables to the list in the future ? > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Status: Patch Available (was: Open) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: HBASE-16262-branch-1.patch > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: (was: HBASE-16262-branch-1.patch) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Status: Open (was: Patch Available) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386710#comment-15386710 ] Ted Yu commented on HBASE-16181: As mentioned above, should black list be introduced so that it is easier to add more system tables to the list in the future ? > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: HBASE-16262-branch-1.patch > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: (was: HBASE-16262-branch-1.patch) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Status: Open (was: Patch Available) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: (was: HBASE-16262.patch) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: HBASE-16262-branch-1.patch > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Status: Patch Available (was: Open) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262-branch-1.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Status: Patch Available (was: Open) Tested locally on Branch-1 by failing openRegion commands and checking out the UI. Appears to have the same behavior as Master's RegionInTransition UI. > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Attachment: HBASE-16262.patch > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > Attachments: HBASE-16262.patch > > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386695#comment-15386695 ] Matteo Bertozzi commented on HBASE-16181: - +1 > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[
https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386691#comment-15386691
]
Hadoop QA commented on HBASE-16181:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 8m 15s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
37s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 20s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hbase-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 36s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0. {color}
|
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s {color}
| {color:red} hbase-client in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 36s {color}
| {color:red} hbase-server in the patch failed with JDK v1.8.0. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 14s
{color} | {color:red} hbase-client in the patch failed with JDK v1.7.0_80.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 34s
{color} | {color:red} hbase-server in the patch failed with JDK v1.7.0_80.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 14s {color}
| {color:red} hbase-client in the patch failed with JDK v1.7.0_80. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 34s {color}
| {color:red} hbase-server in the patch failed with JDK v1.7.0_80. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 6 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 0m 41s
{color} | {color:red} Patch causes 18 errors with Hadoop v2.4.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 20s
{color} | {color:red} Patch causes 18 errors with Hadoop v2.4.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 2s
{color} | {color:red} Patch causes 18 errors with Hadoop v2.5.0. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 42s
{color} | {color:red} Patch causes 18 errors with Hadoop v2.5.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 21s
{color} | {color:red} Patch causes 18 errors with Hadoop v2.5.2. {color} |
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386690#comment-15386690
]
Mikhail Antonov commented on HBASE-16232:
-
The file 81580c0a6d8f41cdb83a1e9abf6491df was in the list of files that
comprised "failed to compliete archive of" list in the second snippet, I just
left only first file in the list, sorry.
WARN backup.HFileArchiver: Failed to complete archive of: {list of files}, this
file that wasn't found is in this list.
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Summary: Update RegionsInTransition UI for branch-1 (was: Update RegionsInTransition UI for branch-1. Also have Failed_Open regions appear first for RegionsInTransition) > Update RegionsInTransition UI for branch-1 > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16262) Update RegionsInTransition UI for branch-1. Also have Failed_Open regions appear first for RegionsInTransition
[ https://issues.apache.org/jira/browse/HBASE-16262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph updated HBASE-16262: --- Summary: Update RegionsInTransition UI for branch-1. Also have Failed_Open regions appear first for RegionsInTransition (was: Update RegionsInTransition UI for branch-1) > Update RegionsInTransition UI for branch-1. Also have Failed_Open regions > appear first for RegionsInTransition > -- > > Key: HBASE-16262 > URL: https://issues.apache.org/jira/browse/HBASE-16262 > Project: HBase > Issue Type: Improvement >Reporter: Joseph >Assignee: Joseph >Priority: Minor > > Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16262) Update RegionsInTransition UI for branch-1
Joseph created HBASE-16262: -- Summary: Update RegionsInTransition UI for branch-1 Key: HBASE-16262 URL: https://issues.apache.org/jira/browse/HBASE-16262 Project: HBase Issue Type: Improvement Reporter: Joseph Assignee: Joseph Priority: Minor Backport HBASE-13839 to Branch 1. Also make a few style changes in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386684#comment-15386684
]
stack commented on HBASE-16232:
---
The original file name that could not be found is not same as those that follow.
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386673#comment-15386673 ] yi liang commented on HBASE-16261: -- will provide patch shortly > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang updated HBASE-16261: - Description: MultiHFileOutputFormat follow HFileOutputFormat2 (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. was: aurelius (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > > MultiHFileOutputFormat follow HFileOutputFormat2 > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16261) MultiHFileOutputFormat Enhancement
yi liang created HBASE-16261: Summary: MultiHFileOutputFormat Enhancement Key: HBASE-16261 URL: https://issues.apache.org/jira/browse/HBASE-16261 Project: HBase Issue Type: Sub-task Components: hbase Affects Versions: 2.0.0 Reporter: yi liang Assignee: yi liang Priority: Minor (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang updated HBASE-16261: - Description: (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. was: (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16261) MultiHFileOutputFormat Enhancement
[ https://issues.apache.org/jira/browse/HBASE-16261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yi liang updated HBASE-16261: - Description: aurelius (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. was: (1) HFileOutputFormat2 can read one table's region split keys. and then output multiple hfiles for one family, and each hfile map to one region. We can add partitioner in MultiHFileOutputFormat to make it support this feature. (2) HFileOutputFormat2 support Customized Compression algorithm for column family and BloomFilter, also support customized DataBlockEncoding for the output hfiles. We can also make MultiHFileOutputFormat to support these features. > MultiHFileOutputFormat Enhancement > > > Key: HBASE-16261 > URL: https://issues.apache.org/jira/browse/HBASE-16261 > Project: HBase > Issue Type: Sub-task > Components: hbase >Affects Versions: 2.0.0 >Reporter: yi liang >Assignee: yi liang >Priority: Minor > Fix For: 2.0.0 > > > aurelius > (1) HFileOutputFormat2 can read one table's region split keys. and then > output multiple hfiles for one family, and each hfile map to one region. We > can add partitioner in MultiHFileOutputFormat to make it support this feature. > (2) HFileOutputFormat2 support Customized Compression algorithm for column > family and BloomFilter, also support customized DataBlockEncoding for the > output hfiles. We can also make MultiHFileOutputFormat to support these > features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16260) Audit dependencies for Category-X
[ https://issues.apache.org/jira/browse/HBASE-16260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386668#comment-15386668 ] Jonathan Hsieh commented on HBASE-16260: xom as added as part of esapi jar additions in HBASE-15122. > Audit dependencies for Category-X > - > > Key: HBASE-16260 > URL: https://issues.apache.org/jira/browse/HBASE-16260 > Project: HBase > Issue Type: Task > Components: community, dependencies >Affects Versions: 2.0.0, 1.2.0, 1.3.0, 1.2.1, 1.1.4, 1.0.4, 1.1.5, 1.2.2 >Reporter: Sean Busbey >Assignee: Sean Busbey >Priority: Blocker > Fix For: 2.0.0, 1.1.6, 1.2.3 > > > Make sure we do not have category x dependencies. > right now we atleast have an LGPL for xom:xom (thanks to PHOENIX-3103 for the > catch) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386652#comment-15386652
]
Mikhail Antonov commented on HBASE-16232:
-
So one observation I have so far, maybe completely irrelevant.
During the time of generator run which led to lost keys I'm seeing this in the
logs, looking at one region, and I didn't see those on other runs:
{code}
WARN backup.HFileArchiver: Failed to archive class
org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile,
file:/data/default/IntegrationTestBigLinkedList.5/ee81b123a9d48bf086d1a237c35c563b/tiny/81580c0a6d8f41cdb83a1e9abf6491df
on try #0
java.io.FileNotFoundException: File/Directory
/data/default/IntegrationTestBigLinkedList.5/ee81b123a9d48bf086d1a237c35c563b/tiny/81580c0a6d8f41cdb83a1e9abf6491df
does not exist.
at
org.apache.hadoop.hdfs.server.namenode.FSDirAttrOp.setTimes(FSDirAttrOp.java:121)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.setTimes(FSNamesystem.java:1907)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.setTimes(NameNodeRpcServer.java:1222)
{code}
I'm then getting :
{code}
WARN backup.HFileArchiver: Failed to complete archive of:
[class org.apache.hadoop.hbase.backup.HFileArchiver$FileableStoreFile,
/data/default/IntegrationTestBigLinkedList.5/ee81b123a9d48bf086d1a237c35c563b/tiny/32029c46f36e481b99a7af10900e533a,
<8 more files here>]
Those files are still in the original location, and they may slow down reads.
{code}
and
{code}
FATAL regionserver.HRegionServer: ABORTING region server :
Unrecoverable exception while closing region
IntegrationTestBigLinkedList.5,$rA\xC0gzW\x14,1468971262948.ee81b123a9d48bf086d1a237c35c563b.,
still finishing close
java.io.IOException: java.io.IOException: Failed to archive/delete all the
files for
region:IntegrationTestBigLinkedList.5,$rA?gzW,1468971262948.ee81b123a9d48bf086d1a237c35c563b.,
family:tiny into
/archive/data/default/IntegrationTestBigLinkedList.5/ee81b123a9d48bf086d1a237c35c563b/tiny.
Something is probably awry on the filesystem.
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1516)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1378)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:138)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to archive/delete all the files for
region:IntegrationTestBigLinkedList.5,$rA?gzW,1468971262948.ee81b123a9d48bf086d1a237c35c563b.,
family:tiny into
/archive/data/default/IntegrationTestBigLinkedList.5/ee81b123a9d48bf086d1a237c35c563b/tiny.
Something is probably awry on the filesystem.
at
org.apache.hadoop.hbase.backup.HFileArchiver.archiveStoreFiles(HFileArchiver.java:233)
at
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.removeStoreFiles(HRegionFileSystem.java:424)
at
org.apache.hadoop.hbase.regionserver.HStore.removeCompactedfiles(HStore.java:2699)
at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:835)
at org.apache.hadoop.hbase.regionserver.HStore.close(HStore.java:116)
at
org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1498)
at
org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1494)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
{code}
and then
ERROR executor.EventHandler: Caught throwable while processing event
M_RS_CLOSE_REGION
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
>
[jira] [Updated] (HBASE-14921) Memory optimizations
[ https://issues.apache.org/jira/browse/HBASE-14921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anastasia Braginsky updated HBASE-14921: Attachment: HBASE-14921-V06-CAO.patch > Memory optimizations > > > Key: HBASE-14921 > URL: https://issues.apache.org/jira/browse/HBASE-14921 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Eshcar Hillel >Assignee: Anastasia Braginsky > Attachments: CellBlocksSegmentInMemStore.pdf, > CellBlocksSegmentinthecontextofMemStore(1).pdf, HBASE-14921-V01.patch, > HBASE-14921-V02.patch, HBASE-14921-V03.patch, HBASE-14921-V04-CA-V02.patch, > HBASE-14921-V04-CA.patch, HBASE-14921-V05-CAO.patch, > HBASE-14921-V06-CAO.patch, InitialCellArrayMapEvaluation.pdf, > IntroductiontoNewFlatandCompactMemStore.pdf > > > Memory optimizations including compressed format representation and offheap > allocations -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13701) Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load
[ https://issues.apache.org/jira/browse/HBASE-13701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jerry He updated HBASE-13701: - Attachment: HBASE-13701-v4-addendum.patch > Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load > --- > > Key: HBASE-13701 > URL: https://issues.apache.org/jira/browse/HBASE-13701 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-13701-v1.patch, HBASE-13701-v2.patch, > HBASE-13701-v3.patch, HBASE-13701-v4-addendum.patch, HBASE-13701-v4.patch > > > HBASE-12052 makes SecureBulkLoadEndpoint work in a non-secure env to solve > HDFS permission issues. > We have encountered some of the permission issues and have to use this > SecureBulkLoadEndpoint to workaround issues. > We should probably consolidate SecureBulkLoadEndpoint into HBase core as > default for bulk load since it is able to handle both secure Kerberos and > non-secure cases. > Maintaining two versions of bulk load implementation is also a cause of > confusion, and having to explicitly set it is also inconvenient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386587#comment-15386587 ] huaxiang sun commented on HBASE-15143: -- I'd like to help on this one, thanks. > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-15143) Procedure v2 - Web UI displaying queues
[ https://issues.apache.org/jira/browse/HBASE-15143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun reassigned HBASE-15143: Assignee: huaxiang sun > Procedure v2 - Web UI displaying queues > --- > > Key: HBASE-15143 > URL: https://issues.apache.org/jira/browse/HBASE-15143 > Project: HBase > Issue Type: Sub-task > Components: proc-v2, UI >Reporter: Matteo Bertozzi >Assignee: huaxiang sun >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > > We can query MasterProcedureScheduler to display the various procedures and > who is holding table/region locks. > Each procedure is in a TableQueue or ServerQueue, so it is easy to display > the procedures in its own group. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13701) Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load
[ https://issues.apache.org/jira/browse/HBASE-13701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386575#comment-15386575 ] Jerry He commented on HBASE-13701: -- Ok, I pushed a trivial addendum to fix a trivial issue and to show the JIRA number. > Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load > --- > > Key: HBASE-13701 > URL: https://issues.apache.org/jira/browse/HBASE-13701 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-13701-v1.patch, HBASE-13701-v2.patch, > HBASE-13701-v3.patch, HBASE-13701-v4.patch > > > HBASE-12052 makes SecureBulkLoadEndpoint work in a non-secure env to solve > HDFS permission issues. > We have encountered some of the permission issues and have to use this > SecureBulkLoadEndpoint to workaround issues. > We should probably consolidate SecureBulkLoadEndpoint into HBase core as > default for bulk load since it is able to handle both secure Kerberos and > non-secure cases. > Maintaining two versions of bulk load implementation is also a cause of > confusion, and having to explicitly set it is also inconvenient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16260) Audit dependencies for Category-X
Sean Busbey created HBASE-16260: --- Summary: Audit dependencies for Category-X Key: HBASE-16260 URL: https://issues.apache.org/jira/browse/HBASE-16260 Project: HBase Issue Type: Task Components: community, dependencies Affects Versions: 1.2.2, 1.1.5, 1.1.4, 1.2.1, 1.2.0, 2.0.0, 1.3.0, 1.0.4 Reporter: Sean Busbey Assignee: Sean Busbey Priority: Blocker Fix For: 2.0.0, 1.1.6, 1.2.3 Make sure we do not have category x dependencies. right now we atleast have an LGPL for xom:xom (thanks to PHOENIX-3103 for the catch) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16259) Full backup only deployments
Vladimir Rodionov created HBASE-16259:
-
Summary: Full backup only deployments
Key: HBASE-16259
URL: https://issues.apache.org/jira/browse/HBASE-16259
Project: HBase
Issue Type: New Feature
Reporter: Vladimir Rodionov
[~enis]:
{quote}
We keep the WALs which can accumulate a lot if the use case is to only do
backups infrequently. This will definitely cause issues since HDFS space
will get filled up. That is why we may need an option for having
incremental backups not used, and WAL references being deleted.
{quote}
In deployments with infrequent full backups only, there is no need to keep WAL
files in archive for incremental backups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386560#comment-15386560 ] Sean Busbey commented on HBASE-16179: - I suspect we need a dev@ discussion about what versions of Spark we want to support in what versions of HBase. I still expect that spark support will get in to HBase branch-1, and I'd expect that we should work with both Spark 1.6 and Spark 2.0. I'm less sure about Scala versions. > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16210) Add Timestamp class to the hbase-common
[ https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Teja Ranuva updated HBASE-16210: Status: Open (was: Patch Available) > Add Timestamp class to the hbase-common > --- > > Key: HBASE-16210 > URL: https://issues.apache.org/jira/browse/HBASE-16210 > Project: HBase > Issue Type: Sub-task >Reporter: Sai Teja Ranuva >Assignee: Sai Teja Ranuva >Priority: Minor > Labels: patch, testing > Attachments: HBASE-16210.master.1.patch, HBASE-16210.master.2.patch, > HBASE-16210.master.3.patch, HBASE-16210.master.4.patch, > HBASE-16210.master.5.patch, HBASE-16210.master.6.patch, > HBASE-16210.master.7.patch, HBASE-16210.master.8.1.patch, > HBASE-16210.master.8.2.patch, HBASE-16210.master.8.3.patch, > HBASE-16210.master.8.patch > > > This is a sub-issue of > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070]. This JIRA is > a small step towards completely adding Hybrid Logical Clocks(HLC) to HBase. > The main idea of HLC is described in > [HBase-14070|https://issues.apache.org/jira/browse/HBASE-14070] along with > the motivation of adding it to HBase. > This patch in this issue takes the code from the patch in the parent. > The parent patch is pretty big to review at once. So, plan is to get code > reviewed in smaller patches and > in the process take suggestions and change things if necessary. > What is this patch/issue about ? > This issue attempts to add a timestamp class to hbase-common and timestamp > type to HTable. > This is a part of the attempt to get HLC into HBase. This patch does not > interfere with the current working of HBase. > Why Timestamp Class ? > Timestamp class can be as an abstraction to represent time in Hbase in 64 > bits. > It is just used for manipulating with the 64 bits of the timestamp and is not > concerned about the actual time. > There are three types of timestamps. System time, Custom and HLC. Each one of > it has methods to manipulate the 64 bits of timestamp. > HTable changes: Added a timestamp type property to HTable. This will help > HBase exist in conjunction with old type of timestamp and also the HLC which > will be introduced. The default is set to custom timestamp(current way of > usage of timestamp). default unset timestamp is also custom timestamp as it > should be so. The default timestamp will be changed to HLC when HLC feature > is introduced completely in HBase. > Check HBASE-16210.master.6.patch. > Update: Based on the suggestions, made timestamp enum. Here is the > description of the new changes. > Check the HBASE-16210.master.8.1.patch > 1. Changed the Timestamp Implementation to Enum. > 2. Changed the Timestamp semantics. Instead of HLC, System monotonic and > custom, we now have Hybrid and Physical. System monotonic clock and Custom > clocks can map their timestamps to Physical. HLC clock can map its timestamp > to Hybrid. > 3. The HTableDescriptor will contain clock type(not implemented yet) instead > of timestamp type. As clocks convey the semantics of monotonic increasing and > non decreasing etc. TimestampType doesn't have those semantics enforced, it > just knows what to do with given 64 bits. Therefore, I removed the timestamp > type field in the HTableDescriptor. > Open for suggestions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16142) Trigger JFR session when under duress -- e.g. backed-up request queue count -- and dump the recording to log dir
[
https://issues.apache.org/jira/browse/HBASE-16142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386555#comment-15386555
]
Hadoop QA commented on HBASE-16142:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
27s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 34s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 101m 43s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 144m 38s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12819041/HBASE-16142.master.003.patch
|
| JIRA Issue | HBASE-16142 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 70f330d |
| Default Java | 1.7.0_80 |
| Multi-JDK versions | /home/jenkins/tools/java/jdk1.8.0:1.8.0
/home/jenkins/jenkins-slave/tools/hudson.model.JDK/JDK_1.7_latest_:1.7.0_80 |
| findbugs | v3.0.0 |
| unit |
[jira] [Commented] (HBASE-16095) Add priority to TableDescriptor and priority region open thread pool
[ https://issues.apache.org/jira/browse/HBASE-16095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386548#comment-15386548 ] Andrew Purtell commented on HBASE-16095: bq. I'd be ok w/ this in branch-1 as a hack to get over support burden imposed by a deployed dodgy secondary index implementation. I don't think we should commit to branch-2 though. That should be time enough to get off this misbehavior. This is reasonable IMHO bq. On some user tables being able to be opened before others, won't this always be sloppy when regionserver-determined Yes > Add priority to TableDescriptor and priority region open thread pool > > > Key: HBASE-16095 > URL: https://issues.apache.org/jira/browse/HBASE-16095 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.3.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16095-0.98.patch, HBASE-16095-0.98.patch, > hbase-16095_v0.patch, hbase-16095_v1.patch, hbase-16095_v2.patch, > hbase-16095_v3.patch > > > This is in the similar area with HBASE-15816, and also required with the > current secondary indexing for Phoenix. > The problem with P secondary indexes is that data table regions depend on > index regions to be able to make progress. Possible distributed deadlocks can > be prevented via custom RpcScheduler + RpcController configuration via > HBASE-11048 and PHOENIX-938. However, region opening also has the same > deadlock situation, because data region open has to replay the WAL edits to > the index regions. There is only 1 thread pool to open regions with 3 workers > by default. So if the cluster is recovering / restarting from scratch, the > deadlock happens because some index regions cannot be opened due to them > being in the same queue waiting for data regions to open (which waits for > RPC'ing to index regions which is not open). This is reproduced in almost all > Phoenix secondary index clusters (mutable table w/o transactions) that we > see. > The proposal is to have a "high priority" region opening thread pool, and > have the HTD carry the relative priority of a table. This maybe useful for > other "framework" level tables from Phoenix, Tephra, Trafodian, etc if they > want some specific tables to become online faster. > As a follow up patch, we can also take a look at how this priority > information can be used by the rpc scheduler on the server side or rpc > controller on the client side, so that we do not have to set priorities > manually per-operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16052) Improve HBaseFsck Scalability
[ https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386507#comment-15386507 ] Ted Yu commented on HBASE-16052: Integrated the addendum to 0.98 branch. Thanks, Ben. > Improve HBaseFsck Scalability > - > > Key: HBASE-16052 > URL: https://issues.apache.org/jira/browse/HBASE-16052 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16052-0.98.v3-amendment.patch, > HBASE-16052-0.98.v3.patch, HBASE-16052-master.patch, > HBASE-16052-v3-0.98.patch, HBASE-16052-v3-branch-1.patch, > HBASE-16052-v3-master.patch > > > There are some problems with HBaseFsck that make it unnecessarily slow > especially for large tables or clusters with many regions. > This patch tries to fix the biggest bottlenecks and also include a couple of > bug fixes for some of the race conditions caused by gathering and holding > state about a live cluster that is no longer true by the time you use that > state in Fsck processing. These race conditions cause Fsck to crash and > become unusable on large clusters with lots of region splits/merges. > Here are some scalability/performance problems in HBaseFsck and the changes > the patch makes: > - Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and > then discarding everything but the Paths, then passing the Paths to a > PathFilter, and then having the filter look up the (previously discarded) > FileStatuses of the paths again. This is actually worse than double I/O > because the first lookup obtains a batch of FileStatuses while all the other > lookups are individual RPCs performed sequentially. > -- Avoid this by adding a FileStatusFilter so that filtering can happen > directly on FileStatuses > -- This performance bug affects more than Fsck, but also to some extent > things like snapshots, hfile archival, etc. I didn't have time to look too > deep into other things affected and didn't want to increase the scope of this > ticket so I focus mostly on Fsck and make only a few improvements to other > codepaths. The changes in this patch though should make it fairly easy to > fix other code paths in later jiras if we feel there are some other features > strongly impacted by this problem. > - OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of > Fsck runtime) and the running time scales with the number of store files, yet > the function is completely serial > -- Make offlineReferenceFileRepair multithreaded > - LoadHdfsRegionDirs() uses table-level concurrency, which is a big > bottleneck if you have 1 large cluster with 1 very large table that has > nearly all the regions > -- Change loadHdfsRegionDirs() to region-level parallelism instead of > table-level parallelism for operations. > The changes benefit all clusters but are especially noticeable for large > clusters with a few very large tables. On our version of 0.98 with the > original patch we had a moderately sized production cluster with 2 (user) > tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[
https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386512#comment-15386512
]
Hadoop QA commented on HBASE-16210:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
24s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
39s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 19s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 8s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 58s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} |
[jira] [Comment Edited] (HBASE-16052) Improve HBaseFsck Scalability
[ https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386493#comment-15386493 ] Ben Lau edited comment on HBASE-16052 at 7/20/16 7:42 PM: -- Seems it should just be this method call. [~tedyu] can you take a look at the amendment patch I just attached? was (Author: benlau): Seems it should just be this method call. [~tedyu] can you take a look? > Improve HBaseFsck Scalability > - > > Key: HBASE-16052 > URL: https://issues.apache.org/jira/browse/HBASE-16052 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16052-0.98.v3-amendment.patch, > HBASE-16052-0.98.v3.patch, HBASE-16052-master.patch, > HBASE-16052-v3-0.98.patch, HBASE-16052-v3-branch-1.patch, > HBASE-16052-v3-master.patch > > > There are some problems with HBaseFsck that make it unnecessarily slow > especially for large tables or clusters with many regions. > This patch tries to fix the biggest bottlenecks and also include a couple of > bug fixes for some of the race conditions caused by gathering and holding > state about a live cluster that is no longer true by the time you use that > state in Fsck processing. These race conditions cause Fsck to crash and > become unusable on large clusters with lots of region splits/merges. > Here are some scalability/performance problems in HBaseFsck and the changes > the patch makes: > - Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and > then discarding everything but the Paths, then passing the Paths to a > PathFilter, and then having the filter look up the (previously discarded) > FileStatuses of the paths again. This is actually worse than double I/O > because the first lookup obtains a batch of FileStatuses while all the other > lookups are individual RPCs performed sequentially. > -- Avoid this by adding a FileStatusFilter so that filtering can happen > directly on FileStatuses > -- This performance bug affects more than Fsck, but also to some extent > things like snapshots, hfile archival, etc. I didn't have time to look too > deep into other things affected and didn't want to increase the scope of this > ticket so I focus mostly on Fsck and make only a few improvements to other > codepaths. The changes in this patch though should make it fairly easy to > fix other code paths in later jiras if we feel there are some other features > strongly impacted by this problem. > - OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of > Fsck runtime) and the running time scales with the number of store files, yet > the function is completely serial > -- Make offlineReferenceFileRepair multithreaded > - LoadHdfsRegionDirs() uses table-level concurrency, which is a big > bottleneck if you have 1 large cluster with 1 very large table that has > nearly all the regions > -- Change loadHdfsRegionDirs() to region-level parallelism instead of > table-level parallelism for operations. > The changes benefit all clusters but are especially noticeable for large > clusters with a few very large tables. On our version of 0.98 with the > original patch we had a moderately sized production cluster with 2 (user) > tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16052) Improve HBaseFsck Scalability
[ https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ben Lau updated HBASE-16052: Attachment: HBASE-16052-0.98.v3-amendment.patch Seems it should just be this method call. [~tedyu] can you take a look? > Improve HBaseFsck Scalability > - > > Key: HBASE-16052 > URL: https://issues.apache.org/jira/browse/HBASE-16052 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16052-0.98.v3-amendment.patch, > HBASE-16052-0.98.v3.patch, HBASE-16052-master.patch, > HBASE-16052-v3-0.98.patch, HBASE-16052-v3-branch-1.patch, > HBASE-16052-v3-master.patch > > > There are some problems with HBaseFsck that make it unnecessarily slow > especially for large tables or clusters with many regions. > This patch tries to fix the biggest bottlenecks and also include a couple of > bug fixes for some of the race conditions caused by gathering and holding > state about a live cluster that is no longer true by the time you use that > state in Fsck processing. These race conditions cause Fsck to crash and > become unusable on large clusters with lots of region splits/merges. > Here are some scalability/performance problems in HBaseFsck and the changes > the patch makes: > - Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and > then discarding everything but the Paths, then passing the Paths to a > PathFilter, and then having the filter look up the (previously discarded) > FileStatuses of the paths again. This is actually worse than double I/O > because the first lookup obtains a batch of FileStatuses while all the other > lookups are individual RPCs performed sequentially. > -- Avoid this by adding a FileStatusFilter so that filtering can happen > directly on FileStatuses > -- This performance bug affects more than Fsck, but also to some extent > things like snapshots, hfile archival, etc. I didn't have time to look too > deep into other things affected and didn't want to increase the scope of this > ticket so I focus mostly on Fsck and make only a few improvements to other > codepaths. The changes in this patch though should make it fairly easy to > fix other code paths in later jiras if we feel there are some other features > strongly impacted by this problem. > - OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of > Fsck runtime) and the running time scales with the number of store files, yet > the function is completely serial > -- Make offlineReferenceFileRepair multithreaded > - LoadHdfsRegionDirs() uses table-level concurrency, which is a big > bottleneck if you have 1 large cluster with 1 very large table that has > nearly all the regions > -- Change loadHdfsRegionDirs() to region-level parallelism instead of > table-level parallelism for operations. > The changes benefit all clusters but are especially noticeable for large > clusters with a few very large tables. On our version of 0.98 with the > original patch we had a moderately sized production cluster with 2 (user) > tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16256: Issue Type: Improvement (was: Wish) > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0 > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16256: Labels: beginner clarification (was: clarification) > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Improvement > Components: documentation, regionserver >Reporter: Sai Teja Ranuva >Priority: Trivial > Labels: beginner, clarification > Fix For: 2.0.0 > > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13701) Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load
[ https://issues.apache.org/jira/browse/HBASE-13701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386439#comment-15386439 ] Jerry He commented on HBASE-13701: -- Thanks, [~ghelmling]. My bad. Did it again. is there a way to amend the commit message without reverting and commit again? > Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load > --- > > Key: HBASE-13701 > URL: https://issues.apache.org/jira/browse/HBASE-13701 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-13701-v1.patch, HBASE-13701-v2.patch, > HBASE-13701-v3.patch, HBASE-13701-v4.patch > > > HBASE-12052 makes SecureBulkLoadEndpoint work in a non-secure env to solve > HDFS permission issues. > We have encountered some of the permission issues and have to use this > SecureBulkLoadEndpoint to workaround issues. > We should probably consolidate SecureBulkLoadEndpoint into HBase core as > default for bulk load since it is able to handle both secure Kerberos and > non-secure cases. > Maintaining two versions of bulk load implementation is also a cause of > confusion, and having to explicitly set it is also inconvenient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386431#comment-15386431 ] Matteo Bertozzi commented on HBASE-16181: - the fix on the snapshot side is easy. just put a blacklist instead of excluding all sys-tables. but can you elaborate more on the above? why do you have to snapshot the full backup table when you are taking a new backup? don't you just need to keep track of which rows are you adding to the backup table and remove those in case of rollback? > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16210) Add Timestamp class to the hbase-common
[
https://issues.apache.org/jira/browse/HBASE-16210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386421#comment-15386421
]
Hadoop QA commented on HBASE-16210:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 5s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
56s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 29s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} master passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scalac {color} | {color:green} 1m 24s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 30s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed with JDK v1.8.0 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} scaladoc {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 46s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 9s
{color} | {color:green} hbase-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} |
[jira] [Commented] (HBASE-16181) Backup of hbase:backup table
[ https://issues.apache.org/jira/browse/HBASE-16181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386417#comment-15386417 ] Vladimir Rodionov commented on HBASE-16181: --- [~mbertozzi], we need snapshots for hbase:backup table to implement fault tolerance during backup procedure. Before backup starts we snapshot hbase:backup. If backup fails, we clean up all files/dirs and restore hbase:backup from snapshot. > Backup of hbase:backup table > > > Key: HBASE-16181 > URL: https://issues.apache.org/jira/browse/HBASE-16181 > Project: HBase > Issue Type: Task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16181-v1.patch > > > Snapshot of HBase system tables is not supported, we need either move > hbase:backup into different name space or fix snapshots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386392#comment-15386392
]
stack commented on HBASE-16232:
---
Per RS. I'd be interested in Search not finding keys...
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16232) ITBLL fails on branch-1.3, now loosing actual keys
[
https://issues.apache.org/jira/browse/HBASE-16232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386390#comment-15386390
]
Mikhail Antonov commented on HBASE-16232:
-
48 total or per rs? I'd say try upping that by order of magnitude..
(digging in; search tool doesn't find any of the lost keys in oldWALs, though
the oldWALs for time range in question are there)
> ITBLL fails on branch-1.3, now loosing actual keys
> --
>
> Key: HBASE-16232
> URL: https://issues.apache.org/jira/browse/HBASE-16232
> Project: HBase
> Issue Type: Bug
> Components: dataloss, integration tests
>Affects Versions: 1.3.0
>Reporter: Mikhail Antonov
>Assignee: Mikhail Antonov
>Priority: Blocker
>
> So I'm running ITBLL off branch-1.3 on recent commit (after [~stack]'s fix
> for fake keys showing up in the scans) with increased number of regions per
> regionserver and seeing the following.
> {quote}
> $Verify$Counts
> REFERENCED0 4,999,999,994 4,999,999,994
> UNDEFINED 0 3 3
> UNREFERENCED 0 3 3
> {quote}
> So we're loosing some keys. This time those aren't fake:
> {quote}
> undef
> \x89\x10\xE0\xBBx\xF1\xC4\xBAY`\xC4\xD77\x87\x84\x0F 0 1 1
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE30 1 1
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 0 1 1
> unref
> \x15\x1F*f\x92i6\x86\x1D\x8E\xB7\xE1\xC1=\x96\xEF 0 1 1
> \xF4G\xC6E\xD6\xF1\xAB\xB7\xDB\xC0\x94\xF2\xE7mN\xEC 0 1 1
> U\x0F'\x88\x106\x19\x1C\x87Y"\xF3\xE6\xC1\xC8\x15
> {quote}
> Re-running verify step with CM off still shows this issue. Search tool
> reports:
> {quote}
> Total
> \x89\x11\x0F\xBA@\x0D8^\xAE \xB1\xCAh\xEB&\xE35 0 5
> \x89\x16waxv;\xB1\xE3Z\xE6"|\xFC\xBE\x9A 4 0 4
> CELL_WITH_MISSING_ROW 15 0 15
> {quote}
> Will post more as I dig into.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16256) Purpose of EnvironmentEdge, EnvironmentEdgeManager
[ https://issues.apache.org/jira/browse/HBASE-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386376#comment-15386376 ] Sai Teja Ranuva commented on HBASE-16256: - Yes, it helped. Thank you [~stack]. > Purpose of EnvironmentEdge, EnvironmentEdgeManager > -- > > Key: HBASE-16256 > URL: https://issues.apache.org/jira/browse/HBASE-16256 > Project: HBase > Issue Type: Wish >Reporter: Sai Teja Ranuva >Priority: Trivial > Labels: clarification > > I am trying to figure out the purpose of EnvironmentEdge in the HBase code. > The java doc says it has something to do with environment, I feel it is > vague. It looks like there is a bigger picture for such a design. Currently > only concrete method that is available in it is currentTime(). > Can anyone summarize the rationale behind having > EnvironmentEdge/EnvironmentEdgeManager ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16052) Improve HBaseFsck Scalability
[ https://issues.apache.org/jira/browse/HBASE-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386372#comment-15386372 ] Ben Lau commented on HBASE-16052: - Looks like some of the methods don't exist on Hadoop 1.1. When I ran tests locally it was against Hadoop 2.X. Will fix. > Improve HBaseFsck Scalability > - > > Key: HBASE-16052 > URL: https://issues.apache.org/jira/browse/HBASE-16052 > Project: HBase > Issue Type: Improvement > Components: hbck >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.4.0, 0.98.21 > > Attachments: HBASE-16052-0.98.v3.patch, HBASE-16052-master.patch, > HBASE-16052-v3-0.98.patch, HBASE-16052-v3-branch-1.patch, > HBASE-16052-v3-master.patch > > > There are some problems with HBaseFsck that make it unnecessarily slow > especially for large tables or clusters with many regions. > This patch tries to fix the biggest bottlenecks and also include a couple of > bug fixes for some of the race conditions caused by gathering and holding > state about a live cluster that is no longer true by the time you use that > state in Fsck processing. These race conditions cause Fsck to crash and > become unusable on large clusters with lots of region splits/merges. > Here are some scalability/performance problems in HBaseFsck and the changes > the patch makes: > - Unnecessary I/O and RPCs caused by fetching an array of FileStatuses and > then discarding everything but the Paths, then passing the Paths to a > PathFilter, and then having the filter look up the (previously discarded) > FileStatuses of the paths again. This is actually worse than double I/O > because the first lookup obtains a batch of FileStatuses while all the other > lookups are individual RPCs performed sequentially. > -- Avoid this by adding a FileStatusFilter so that filtering can happen > directly on FileStatuses > -- This performance bug affects more than Fsck, but also to some extent > things like snapshots, hfile archival, etc. I didn't have time to look too > deep into other things affected and didn't want to increase the scope of this > ticket so I focus mostly on Fsck and make only a few improvements to other > codepaths. The changes in this patch though should make it fairly easy to > fix other code paths in later jiras if we feel there are some other features > strongly impacted by this problem. > - OfflineReferenceFileRepair is the most expensive part of Fsck (often 50% of > Fsck runtime) and the running time scales with the number of store files, yet > the function is completely serial > -- Make offlineReferenceFileRepair multithreaded > - LoadHdfsRegionDirs() uses table-level concurrency, which is a big > bottleneck if you have 1 large cluster with 1 very large table that has > nearly all the regions > -- Change loadHdfsRegionDirs() to region-level parallelism instead of > table-level parallelism for operations. > The changes benefit all clusters but are especially noticeable for large > clusters with a few very large tables. On our version of 0.98 with the > original patch we had a moderately sized production cluster with 2 (user) > tables and ~160k regions where HBaseFsck went from taking 18 min to 5 minutes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-13701) Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load
[ https://issues.apache.org/jira/browse/HBASE-13701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15386343#comment-15386343 ] Gary Helmling commented on HBASE-13701: --- [~jerryhe] please include the JIRA number in the commit message in the future. This went in without it, which makes it hard to trace back from the commit log. > Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load > --- > > Key: HBASE-13701 > URL: https://issues.apache.org/jira/browse/HBASE-13701 > Project: HBase > Issue Type: Improvement >Reporter: Jerry He >Assignee: Jerry He > Fix For: 2.0.0 > > Attachments: HBASE-13701-v1.patch, HBASE-13701-v2.patch, > HBASE-13701-v3.patch, HBASE-13701-v4.patch > > > HBASE-12052 makes SecureBulkLoadEndpoint work in a non-secure env to solve > HDFS permission issues. > We have encountered some of the permission issues and have to use this > SecureBulkLoadEndpoint to workaround issues. > We should probably consolidate SecureBulkLoadEndpoint into HBase core as > default for bulk load since it is able to handle both secure Kerberos and > non-secure cases. > Maintaining two versions of bulk load implementation is also a cause of > confusion, and having to explicitly set it is also inconvenient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
