[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594277#comment-15594277 ] Devaraj Das commented on HBASE-16414: - Np [~ram_krish]. Fine by me to commit the patch if everyone else is okay with it. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch, > HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch, > HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594258#comment-15594258
]
Anoop Sam John commented on HBASE-15968:
One Q. So in this test u did a major compaction before doing the reads. In old
semantic we would be having 0 as seqId in each of the cell. That is just one
byte. (Correct? I did not check in detail when we will be doing the seqId reset
to 0). In new semantic u mentioned some where else that we can not set seqId to
0. So even after the major compaction also it is a non zero value? How many
bytes per Cell? In read path this matters when we do seeks. (Get op has to do a
linear seek within an HFile block) As we save seqId as vlong, we need to read
every byte by byte of seqId part and decode to know are we done with it for
this cell.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594248#comment-15594248
]
Anoop Sam John commented on HBASE-15968:
bq.Use JMH to get more direct result for SQM.
Good. That might give better insights.
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-15968-v1.patch, HBASE-15968-v2.patch,
> HBASE-15968-v3.patch, HBASE-15968-v4.patch
>
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13408: --- Fix Version/s: (was: 2.0.0) > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10713) A MemStore implementation with in memory flushes to CellBlocks
[ https://issues.apache.org/jira/browse/HBASE-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-10713: --- Fix Version/s: (was: 2.0.0) > A MemStore implementation with in memory flushes to CellBlocks > -- > > Key: HBASE-10713 > URL: https://issues.apache.org/jira/browse/HBASE-10713 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John > Attachments: HBASE-10713_WIP.patch > > > After HBASE-10648 we can plugin any implementation for MemStore. This issue > aims at coming up with an implementation, in which we will have in between in > memory flushes. This will reduce the need to keep lots of KVs in heap as well > as in CSLM. CSLM perform poor when no# items in it increases. We can keep > create CellBlocks (contigous byte[] like HFile block) out of KVs and keep it > as one object rather than many KVs. At some point in time, MemStore might > have N CellBlocks and one CSLM. > These in memory CellBlocks can be compacted to one bigger block in between. > We can target that in follow on tasks once the basic code is ready. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-13408) HBase In-Memory Memstore Compaction
[ https://issues.apache.org/jira/browse/HBASE-13408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-13408: --- Resolution: Duplicate Assignee: (was: Eshcar Hillel) Status: Resolved (was: Patch Available) Dup of HBASE-14918. > HBase In-Memory Memstore Compaction > --- > > Key: HBASE-13408 > URL: https://issues.apache.org/jira/browse/HBASE-13408 > Project: HBase > Issue Type: New Feature >Reporter: Eshcar Hillel > Fix For: 2.0.0 > > Attachments: HBASE-13408-trunk-v01.patch, > HBASE-13408-trunk-v02.patch, HBASE-13408-trunk-v03.patch, > HBASE-13408-trunk-v04.patch, HBASE-13408-trunk-v05.patch, > HBASE-13408-trunk-v06.patch, HBASE-13408-trunk-v07.patch, > HBASE-13408-trunk-v08.patch, HBASE-13408-trunk-v09.patch, > HBASE-13408-trunk-v10.patch, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver02.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver03.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument-ver04.pdf, > HBaseIn-MemoryMemstoreCompactionDesignDocument.pdf, > InMemoryMemstoreCompactionEvaluationResults.pdf, > InMemoryMemstoreCompactionMasterEvaluationResults.pdf, > InMemoryMemstoreCompactionScansEvaluationResults.pdf, > StoreSegmentandStoreSegmentScannerClassHierarchies.pdf > > > A store unit holds a column family in a region, where the memstore is its > in-memory component. The memstore absorbs all updates to the store; from time > to time these updates are flushed to a file on disk, where they are > compacted. Unlike disk components, the memstore is not compacted until it is > written to the filesystem and optionally to block-cache. This may result in > underutilization of the memory due to duplicate entries per row, for example, > when hot data is continuously updated. > Generally, the faster the data is accumulated in memory, more flushes are > triggered, the data sinks to disk more frequently, slowing down retrieval of > data, even if very recent. > In high-churn workloads, compacting the memstore can help maintain the data > in memory, and thereby speed up data retrieval. > We suggest a new compacted memstore with the following principles: > 1.The data is kept in memory for as long as possible > 2.Memstore data is either compacted or in process of being compacted > 3.Allow a panic mode, which may interrupt an in-progress compaction and > force a flush of part of the memstore. > We suggest applying this optimization only to in-memory column families. > A design document is attached. > This feature was previously discussed in HBASE-5311. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-10713) A MemStore implementation with in memory flushes to CellBlocks
[ https://issues.apache.org/jira/browse/HBASE-10713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-10713. Resolution: Duplicate Assignee: (was: Anoop Sam John) Closing as a dup of HBASE-14918. There we are doing conceptually the same.. Instead of CellBlock as flat byte[], we have Segments there which can be cell array or cell chunk with index. HBASE-14918 is almost getting to its closure. > A MemStore implementation with in memory flushes to CellBlocks > -- > > Key: HBASE-10713 > URL: https://issues.apache.org/jira/browse/HBASE-10713 > Project: HBase > Issue Type: New Feature >Reporter: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-10713_WIP.patch > > > After HBASE-10648 we can plugin any implementation for MemStore. This issue > aims at coming up with an implementation, in which we will have in between in > memory flushes. This will reduce the need to keep lots of KVs in heap as well > as in CSLM. CSLM perform poor when no# items in it increases. We can keep > create CellBlocks (contigous byte[] like HFile block) out of KVs and keep it > as one object rather than many KVs. At some point in time, MemStore might > have N CellBlocks and one CSLM. > These in memory CellBlocks can be compacted to one bigger block in between. > We can target that in follow on tasks once the basic code is ready. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594213#comment-15594213 ] Anoop Sam John commented on HBASE-16608: Fixed ur comments in the _Final.patch Thanks > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-V01.patch, > HBASE-16608-V03.patch, HBASE-16608-V04.patch, HBASE-16608-V08.patch, > HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15905) Makefile build env incorrectly links in tests
[ https://issues.apache.org/jira/browse/HBASE-15905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-15905: -- Attachment: HBASE-15905.HBASE-14850.v2.patch This patch has the foll fixes:- 1) Unit tests removed from shared libraary. 2) Fixed issue where running make fails on clean checkout. 3) Fixed issue where protobuf objects were not included in shared library, resulting in unresovled references. -- Thanks > Makefile build env incorrectly links in tests > - > > Key: HBASE-15905 > URL: https://issues.apache.org/jira/browse/HBASE-15905 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Priyadharshini karthikeyan > Attachments: HBASE-15905.HBASE-14850.v1.patch, > HBASE-15905.HBASE-14850.v2.patch > > > Right now the makefile build system doesn't seem to do so well. > * Tests are included in the lib > * Documentation includes the protobuf dir > * just running make on a clean check out fails. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16608) Introducing the ability to merge ImmutableSegments without copy-compaction or SQM usage
[ https://issues.apache.org/jira/browse/HBASE-16608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16608: --- Attachment: HBASE-16608-Final.patch > Introducing the ability to merge ImmutableSegments without copy-compaction or > SQM usage > --- > > Key: HBASE-16608 > URL: https://issues.apache.org/jira/browse/HBASE-16608 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Anastasia Braginsky >Assignee: Anastasia Braginsky > Fix For: 2.0.0 > > Attachments: HBASE-16417-V02.patch, HBASE-16417-V04.patch, > HBASE-16417-V06.patch, HBASE-16417-V07.patch, HBASE-16417-V08.patch, > HBASE-16417-V10.patch, HBASE-16608-Final.patch, HBASE-16608-V01.patch, > HBASE-16608-V03.patch, HBASE-16608-V04.patch, HBASE-16608-V08.patch, > HBASE-16608-V09.patch, HBASE-16608-V09.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15893) Get object
[ https://issues.apache.org/jira/browse/HBASE-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sudeep Sunthankar updated HBASE-15893: -- Attachment: HBASE-15893.HBASE-14850.v5.patch Hi , Fixed the issue where unit tests where failing -- Thanks > Get object > -- > > Key: HBASE-15893 > URL: https://issues.apache.org/jira/browse/HBASE-15893 > Project: HBase > Issue Type: Sub-task >Reporter: Sudeep Sunthankar >Assignee: Sudeep Sunthankar > Attachments: HBASE-15893.HBASE-14850.v1.patch, > HBASE-15893.HBASE-14850.v2.patch, HBASE-15893.HBASE-14850.v3.patch, > HBASE-15893.HBASE-14850.v4.patch, HBASE-15893.HBASE-14850.v5.patch > > > Patch for creating Get objects. Get objects can be passed to the Table > implementation to fetch results for a given row. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[ https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594172#comment-15594172 ] Hudson commented on HBASE-16889: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #55 (See [https://builds.apache.org/job/HBase-1.3-JDK8/55/]) HBASE-16889 Proc-V2: verifyTables in the (syuanjiangdev: rev cd29ea79587ae58711ac33e0f3c60e8368d9686d) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestDDLMasterFailover.java > Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each > table DDL is incorrect > - > > Key: HBASE-16889 > URL: https://issues.apache.org/jira/browse/HBASE-16889 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 2.0.0, 1.1.7 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-16889.v1-branch-1.1.patch, > HBASE-16889.v1-master.patch > > > In the IntegrationTestDDLMasterFailover test, verifyTables is called after > each table DDL. It iterates 3 lists of tables in ConcurrentHashMap > (enabledTables, disabledTables, deletedTables) and tries to do some > verification. This is incorrect, eg. a table in enabledTables map could be > picked up by DeleteTableAction and is disabled, while the verification tries > to check whether table is enabled. This leads to false assertion. > The same for verifyNamespaces(). > The proposed fix is to verify maps only at the end of tests (while no active > DDL operation is going on). During test run, we only verify the target table > before putting into map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594171#comment-15594171
]
Hadoop QA commented on HBASE-16870:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
36s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} branch-1.1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 44s
{color} | {color:red} hbase-server in branch-1.1 has 80 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 34s
{color} | {color:red} hbase-server in branch-1.1 failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
11m 58s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed with JDK v1.8.0_101.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 38s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.handler.TestEnableTableHandler |
| | hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures |
| Timed out junit tests |
org.apache.hadoop.hbase.replication.regionserver.TestRegionReplicaReplicationEndpoint
|
| |
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager
|
| | org.apache.hadoop.hbase.replication.TestMultiSlaveReplication |
| | org.apache.hadoop.hbase.mapred.TestTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:35e2245 |
| JIRA Patch URL |
https:/
[jira] [Commented] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[ https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594149#comment-15594149 ] Hudson commented on HBASE-16889: SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #44 (See [https://builds.apache.org/job/HBase-1.2-JDK8/44/]) HBASE-16889 Proc-V2: verifyTables in the (syuanjiangdev: rev 04ff0afb5b949c7eb3517e35d2ca792ff0c45fd6) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestDDLMasterFailover.java > Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each > table DDL is incorrect > - > > Key: HBASE-16889 > URL: https://issues.apache.org/jira/browse/HBASE-16889 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 2.0.0, 1.1.7 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-16889.v1-branch-1.1.patch, > HBASE-16889.v1-master.patch > > > In the IntegrationTestDDLMasterFailover test, verifyTables is called after > each table DDL. It iterates 3 lists of tables in ConcurrentHashMap > (enabledTables, disabledTables, deletedTables) and tries to do some > verification. This is incorrect, eg. a table in enabledTables map could be > picked up by DeleteTableAction and is disabled, while the verification tries > to check whether table is enabled. This leads to false assertion. > The same for verifyNamespaces(). > The proposed fix is to verify maps only at the end of tests (while no active > DDL operation is going on). During test run, we only verify the target table > before putting into map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15594097#comment-15594097 ] Anoop Sam John commented on HBASE-16438: Thanks Ram.. Explains things clearly. There is a jira for CellChunkMap work. Can u post this same 3 issues there as well.. I think #1 and #2 were discussed already. But #3 am not sure.. Ya let us discuss on this and then only go forward. > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16890) Analyze the performance of AsyncWAL and fix the same
[ https://issues.apache.org/jira/browse/HBASE-16890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-16890: --- Attachment: contention.png contention_defaultWAL.png When we try to load 15G of data but with 50 threads and we do a profile for 1 min. We could see that in AsyncWAL case we have contention in the 'waitingConsumePayloads' ArrayDeque. But in the default case there is not much contention as you can see in the snapshots attached. > Analyze the performance of AsyncWAL and fix the same > > > Key: HBASE-16890 > URL: https://issues.apache.org/jira/browse/HBASE-16890 > Project: HBase > Issue Type: Sub-task > Components: wal >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0 > > Attachments: contention.png, contention_defaultWAL.png > > > Tests reveal that AsyncWAL under load in single node cluster performs slower > than the Default WAL. This task is to analyze and see if we could fix it. > See some discussions in the tail of JIRA HBASE-15536. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huzheng updated HBASE-16886: Labels: patch (was: ) Status: Patch Available (was: Open) > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Labels: patch > Attachments: HBASE-16886.v0.patch, TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16438) Create a cell type so that chunk id is embedded in it
[ https://issues.apache.org/jira/browse/HBASE-16438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593960#comment-15593960 ] ramkrishna.s.vasudevan commented on HBASE-16438: [~saint@gmail.com] The idea was actually to create an index to the cells such that we avoid heap overhead. Suppose we have MSLAB and Chunk pool enabled. We do add every cell to a chunk. Now this is in case of the active segment. Once the segment is either MERGED (as in HBASE-16608 or we move them to the pipeline) we can actually create an indexed version of the cell such that they occupy less over head. If suppose we know to which chunk the cell was copied to, during index creation we could add the chunkid, length and offset of every cell in to the index array and we only need to maintain the index array and for any cell retrieval (we do binary search) on the index. Remember that if we use ChunkPool then we are going to have a fixed number of chunks and we would be reusing it. bq.Sequenceid could be used to identify a Cell uniquely. Could we use this to go indirectly to a chunkid? A cell can definitely be identified uniquely but how will we know where is the cell located now? We need a pointer to that to retrive it. But some points to note before we even do this full fledged is - -> ChunkPool if not enabled how will we retrieve the chunk? -> Currently we have a limitation on cell size. -> Append/Increment does not use MSLAB at all for upsert - so we are at a risk here? > Create a cell type so that chunk id is embedded in it > - > > Key: HBASE-16438 > URL: https://issues.apache.org/jira/browse/HBASE-16438 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > For CellChunkMap we may need a cell such that the chunk out of which it was > created, the id of the chunk be embedded in it so that when doing flattening > we can use the chunk id as a meta data. More details will follow once the > initial tasks are completed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16886) hbase-client: scanner with reversed=true and small=true get no result.
[ https://issues.apache.org/jira/browse/HBASE-16886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huzheng updated HBASE-16886: Attachment: HBASE-16886.v0.patch > hbase-client: scanner with reversed=true and small=true get no result. > --- > > Key: HBASE-16886 > URL: https://issues.apache.org/jira/browse/HBASE-16886 > Project: HBase > Issue Type: Bug >Reporter: huzheng > Attachments: HBASE-16886.v0.patch, TestReversedSmallScan.java > > > Assume HBase have four regions (-oo, b), [b, c), [c, d), [d,+oo) , and all > rowKeys are located in region [d, +oo). using a Reversed Small Scanner will > get no result. > Attached file show this failed test case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593911#comment-15593911
]
Hadoop QA commented on HBASE-16071:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
31m 29s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 87m 0s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 130m 52s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicaWithCluster
|
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12813468/HBASE-16071-v3.patch |
| JIRA Issue | HBASE-16071 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 51fbe7c2a388 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / d3decaa |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4130/artifact/patchprocess/branch-findbugs-hbase-server-warnings.html
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/4130/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/4130/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/4130/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593899#comment-15593899 ] ramkrishna.s.vasudevan commented on HBASE-16414: [~devaraj] Sorry that I did not ask for your review yesterday specifically. My bad. Good that you had some concerns before we commit. There is SocketTimeoutException newly added which was previously not there. Do you think it is agreeable? If so we can go ahead with the commit. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch, > HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch, > HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov resolved HBASE-16701. - Resolution: Fixed Fix Version/s: (was: 1.3.1) 1.3.0 > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593863#comment-15593863 ] Mikhail Antonov commented on HBASE-16701: - [~appy] yeah, opened HBASE-16909 and linked to this, and closing this one. Thanks! > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16909) Fix TestHRegion and TestHRegionWithInMemoryFlush timing out (second iteration)
Mikhail Antonov created HBASE-16909: --- Summary: Fix TestHRegion and TestHRegionWithInMemoryFlush timing out (second iteration) Key: HBASE-16909 URL: https://issues.apache.org/jira/browse/HBASE-16909 Project: HBase Issue Type: Bug Components: regionserver, test Affects Versions: 1.3.0, 1.2.5 Reporter: Mikhail Antonov Fix For: 2.0.0, 1.4.0, 1.3.1 See linked jira for comments. Change name as seems appropriate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[ https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593857#comment-15593857 ] Hudson commented on HBASE-16889: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1824 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1824/]) HBASE-16889 Proc-V2: verifyTables in the (syuanjiangdev: rev d3decaab8e13c5cdeb7bafa95ac13e91be71fc9c) * (edit) hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestDDLMasterFailover.java > Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each > table DDL is incorrect > - > > Key: HBASE-16889 > URL: https://issues.apache.org/jira/browse/HBASE-16889 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 2.0.0, 1.1.7 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-16889.v1-branch-1.1.patch, > HBASE-16889.v1-master.patch > > > In the IntegrationTestDDLMasterFailover test, verifyTables is called after > each table DDL. It iterates 3 lists of tables in ConcurrentHashMap > (enabledTables, disabledTables, deletedTables) and tries to do some > verification. This is incorrect, eg. a table in enabledTables map could be > picked up by DeleteTableAction and is disabled, while the verification tries > to check whether table is enabled. This leads to false assertion. > The same for verifyNamespaces(). > The proposed fix is to verify maps only at the end of tests (while no active > DDL operation is going on). During test run, we only verify the target table > before putting into map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593849#comment-15593849 ] Appy commented on HBASE-16701: -- Sounds right to me. Mind creating new one when you close this? (i thought about it when i saw the version change, but then left it to your good judgement given my inexperience in releases). > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-branch-1.1-v1.patch
Reattach v1 patch for branch-1.1 to trigger Hadoop QA.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM-v1.patch,
> HBASE-16870-ADDENDUM.patch, HBASE-16870-branch-1-v1.patch,
> HBASE-16870-branch-1-v1.patch, HBASE-16870-branch-1.1-v1.patch,
> HBASE-16870-branch-1.1-v1.patch, HBASE-16870-branch-1.1.patch,
> HBASE-16870-branch-1.3-v1.patch, HBASE-16870-branch-1.3-v1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593823#comment-15593823 ] Mikhail Antonov commented on HBASE-16701: - [~appy] on the second thought from maintenance perspective it kinds of creates a mess in history if the patch was committed, not reverted afterwards, but jora is marked as re-opened? Should we file a separate jira for that? > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy reopened HBASE-16701: -- > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593820#comment-15593820 ] Mikhail Antonov commented on HBASE-16701: - [~appy] all right - moving it out of 1.3.0 then > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16701: Fix Version/s: (was: 1.3.0) 1.3.1 > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[
https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593817#comment-15593817
]
Appy edited comment on HBASE-16701 at 10/21/16 3:25 AM:
I think [~busbey] left it unresolved because increasing the timeout didn't push
it out of [flaky
list|https://builds.apache.org/job/HBase-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html].
{noformat}
TestHRegion 6.7% (2 / 30)
TestHRegionWithInMemoryFlush6.7% (2 / 30)
{noformat}
Reopening.
bq. If the HBASE-16429 problem reproduced there will be a deadlock and it will
still timeout even we increase the timeout to 10min, so this change won't break
the test case.
[~carp84] Since the test still times out, does that mean the problem you
mention persists? (i haven't looked at that jira)
was (Author: appy):
I think [~busbey] left it unresolved because increasing the timeout didn't push
it out of flaky list.
{noformat}
TestHRegion 6.7% (2 / 30)
TestHRegionWithInMemoryFlush6.7% (2 / 30)
{noformat}
Reopening.
> TestHRegion and TestHRegionWithInMemoryFlush timing out
> ---
>
> Key: HBASE-16701
> URL: https://issues.apache.org/jira/browse/HBASE-16701
> Project: HBase
> Issue Type: Bug
> Components: regionserver, test
>Reporter: Appy
>Assignee: Sean Busbey
> Fix For: 2.0.0, 1.4.0, 1.3.1, 1.2.5, 1.1.8
>
> Attachments: HBASE-16701.1.patch
>
>
> These two tests are marked LargeTests and thus have a timeout of 10 minutes,
> but it's not sufficient since they have like more than 100 testcases.
> Ideal fix would be splitting the tests, but that'll be much more involved
> effort.
> For now, increasing the timeout to 15min.
> Current stats:
> On apache infra:
> TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0
> TestHRegion 9.3% (7 / 75) 7 / 5 / 0
> On GCE infra:
> TestHRegion 89.7% (87 / 97) 87 / 87 / 0
> TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[
https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593817#comment-15593817
]
Appy commented on HBASE-16701:
--
I think [~busbey] left it unresolved because increasing the timeout didn't push
it out of flaky list.
{noformat}
TestHRegion 6.7% (2 / 30)
TestHRegionWithInMemoryFlush6.7% (2 / 30)
{noformat}
Reopening.
> TestHRegion and TestHRegionWithInMemoryFlush timing out
> ---
>
> Key: HBASE-16701
> URL: https://issues.apache.org/jira/browse/HBASE-16701
> Project: HBase
> Issue Type: Bug
> Components: regionserver, test
>Reporter: Appy
>Assignee: Sean Busbey
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8
>
> Attachments: HBASE-16701.1.patch
>
>
> These two tests are marked LargeTests and thus have a timeout of 10 minutes,
> but it's not sufficient since they have like more than 100 testcases.
> Ideal fix would be splitting the tests, but that'll be much more involved
> effort.
> For now, increasing the timeout to 15min.
> Current stats:
> On apache infra:
> TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0
> TestHRegion 9.3% (7 / 75) 7 / 5 / 0
> On GCE infra:
> TestHRegion 89.7% (87 / 97) 87 / 87 / 0
> TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593815#comment-15593815 ] Mikhail Antonov commented on HBASE-15291: - moved to 1.3.1. any progress here? > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.2.1, 0.98.18, 1.4.0, 1.3.1 > > Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, > HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, > HBASE-15291.addendum, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15291) FileSystem not closed in secure bulkLoad
[ https://issues.apache.org/jira/browse/HBASE-15291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-15291: Fix Version/s: (was: 1.3.0) 1.3.1 > FileSystem not closed in secure bulkLoad > > > Key: HBASE-15291 > URL: https://issues.apache.org/jira/browse/HBASE-15291 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.2, 0.98.16.1 >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.2.1, 0.98.18, 1.4.0, 1.3.1 > > Attachments: HBASE-15291-revert-master.patch, HBASE-15291.001.patch, > HBASE-15291.002.patch, HBASE-15291.003.patch, HBASE-15291.004.patch, > HBASE-15291.addendum, patch > > > FileSystem not closed in secure bulkLoad after bulkLoad finish, it will > cause memory used more and more if too many bulkLoad . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16239) Better logging for RPC related exceptions
[
https://issues.apache.org/jira/browse/HBASE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593807#comment-15593807
]
Hadoop QA commented on HBASE-16239:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16239 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12821948/hbase-16239_v2.patch |
| JIRA Issue | HBASE-16239 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4131/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Better logging for RPC related exceptions
> -
>
> Key: HBASE-16239
> URL: https://issues.apache.org/jira/browse/HBASE-16239
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: hbase-16239_v1.patch, hbase-16239_v2.patch,
> hbase-16239_v2.patch, hbase-16239_v2.patch
>
>
> On many occasions, we have to debug RPC related issues, but it is hard in AP
> + RetryingRpcCaller since we mask the stack traces until all retries have
> been exhausted (which takes 10 minutes by default).
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16239) Better logging for RPC related exceptions
[ https://issues.apache.org/jira/browse/HBASE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593800#comment-15593800 ] Mikhail Antonov commented on HBASE-16239: - Moving out of 1.3. [~enis] you're planning to get this one in? anything to fix here? > Better logging for RPC related exceptions > - > > Key: HBASE-16239 > URL: https://issues.apache.org/jira/browse/HBASE-16239 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: hbase-16239_v1.patch, hbase-16239_v2.patch, > hbase-16239_v2.patch, hbase-16239_v2.patch > > > On many occasions, we have to debug RPC related issues, but it is hard in AP > + RetryingRpcCaller since we mask the stack traces until all retries have > been exhausted (which takes 10 minutes by default). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16239) Better logging for RPC related exceptions
[ https://issues.apache.org/jira/browse/HBASE-16239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16239: Fix Version/s: (was: 1.3.0) 1.3.1 > Better logging for RPC related exceptions > - > > Key: HBASE-16239 > URL: https://issues.apache.org/jira/browse/HBASE-16239 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 1.4.0, 1.3.1 > > Attachments: hbase-16239_v1.patch, hbase-16239_v2.patch, > hbase-16239_v2.patch, hbase-16239_v2.patch > > > On many occasions, we have to debug RPC related issues, but it is hard in AP > + RetryingRpcCaller since we mask the stack traces until all retries have > been exhausted (which takes 10 minutes by default). > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16701) TestHRegion and TestHRegionWithInMemoryFlush timing out
[ https://issues.apache.org/jira/browse/HBASE-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov resolved HBASE-16701. - Resolution: Fixed resolving as committed to relevant branches > TestHRegion and TestHRegionWithInMemoryFlush timing out > --- > > Key: HBASE-16701 > URL: https://issues.apache.org/jira/browse/HBASE-16701 > Project: HBase > Issue Type: Bug > Components: regionserver, test >Reporter: Appy >Assignee: Sean Busbey > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.5, 1.1.8 > > Attachments: HBASE-16701.1.patch > > > These two tests are marked LargeTests and thus have a timeout of 10 minutes, > but it's not sufficient since they have like more than 100 testcases. > Ideal fix would be splitting the tests, but that'll be much more involved > effort. > For now, increasing the timeout to 15min. > Current stats: > On apache infra: > TestHRegionWithInMemoryFlush 20.0% (15 / 75) 15 / 12 / 0 > TestHRegion 9.3% (7 / 75) 7 / 5 / 0 > On GCE infra: > TestHRegion 89.7% (87 / 97) 87 / 87 / 0 > TestHRegionWithInMemoryFlush 71.1% (69 / 97) 69 / 69 / 0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593787#comment-15593787
]
Yu Li commented on HBASE-16698:
---
Thanks for the confirmation [~mantonov]
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593766#comment-15593766
]
Yu Li commented on HBASE-16698:
---
Thanks for the question [~enis], the above numbers are against branch-1 codes,
not master.
For master branch, the issue description only talks in theory and no real
testing done. Let me add the testing and will upload data later.
It's confirmed we won't pull this in for branch-1.3 until 1.3.0 got released
and 1.3.1 comes out. For branch-1.2, [~stack] could you help confirm sir?
Thanks.
Will resolve this JIRA if perf data against master branch is good as expected
and decision for branch-1.2 is made. :-)
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593769#comment-15593769
]
Hadoop QA commented on HBASE-16870:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
52s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 26s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 4s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 39s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 99m 36s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| Timed out junit tests | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.io.encoding.TestDataBlockEncoders |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.tool.TestCanaryTool |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:463e832 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834585/HBASE-16870-branch-1.3-v1.patch
|
| JIRA Issue | HBASE-16870 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti che
[jira] [Commented] (HBASE-15532) core favored nodes enhancements
[ https://issues.apache.org/jira/browse/HBASE-15532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593764#comment-15593764 ] Devaraj Das commented on HBASE-15532: - Thanks for the patch [~thiruvel]. I started reviewing it but realized that it's somewhat difficult to keep track of the various changes and why something is done the way it is. Can you please help here. I'd like the patch to be broken up in chunks for review. 1. The split and merge code paths 2. The Balancer related enhancements 3. The various tools (and here also it should be broken up). > core favored nodes enhancements > --- > > Key: HBASE-15532 > URL: https://issues.apache.org/jira/browse/HBASE-15532 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu >Assignee: Thiruvel Thirumoolan > Fix For: 2.0.0 > > Attachments: HBASE-15532.master.000.patch, > HBASE-15532.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593720#comment-15593720 ] Colin Ma commented on HBASE-16414: -- [~devaraj], for the different configuration in server/client, the unit test is added in TestSecureIPC.testDifferentConfWithCryptoAES(). For the different version of server/client, actually, it's about the compatibility of protobuf, I also tested it in my local. The following table is the description for these situations: ||||Server side ||Client side || Action|| |Case1 | disable AES(without the feature) | enable AES(with the feature) | Client will send the request to Server for connection header negotiation, but Server won't response. Client will throw the exception for timeout at last| |Case2 | enable AES(with the feature) | disable AES(without the feature) | Client won't send the request to Server for connection header negotiation, no negotiation happen| > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HBASE-16414.003.patch, HBASE-16414.004.patch, HBASE-16414.005.patch, > HBASE-16414.006.patch, HBASE-16414.007.patch, HBASE-16414.008.patch, > HBASE-16414.009.patch, HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593677#comment-15593677 ] Hudson commented on HBASE-16752: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK7 #48 (See [https://builds.apache.org/job/HBase-1.3-JDK7/48/]) HBASE-16752 addendum. Do not retry large request for client versions (garyh: rev 3efa3b911294fe0a42440e2aef3865898d91acc0) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16852) TestDefaultCompactSelection failed on branch-1.3
[ https://issues.apache.org/jira/browse/HBASE-16852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-16852: Fix Version/s: (was: 1.3.0) 1.3.1 > TestDefaultCompactSelection failed on branch-1.3 > > > Key: HBASE-16852 > URL: https://issues.apache.org/jira/browse/HBASE-16852 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 1.3.0 > Environment: asf jenkins >Reporter: Mikhail Antonov > Fix For: 1.3.1 > > > Regression > org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection.testCompactionRatio > Failing for the past 1 build (Since Failed#48 ) > Took 0.1 sec. > Error Message > expected:<[[50, 25, 12, 12]]> but was:<[[]]> > Stacktrace > org.junit.ComparisonFailure: expected:<[[50, 25, 12, 12]]> but was:<[[]]> > at org.junit.Assert.assertEquals(Assert.java:115) at > org.junit.Assert.assertEquals(Assert.java:144) at > org.apache.hadoop.hbase.regionserver.TestCompactionPolicy.compactEquals(TestCompactionPolicy.java:204) >at > org.apache.hadoop.hbase.regionserver.TestCompactionPolicy.compactEquals(TestCompactionPolicy.java:185) >at > org.apache.hadoop.hbase.regionserver.TestDefaultCompactSelection.testCompactionRatio(TestDefaultCompactSelection.java:95) > https://builds.apache.org/view/All/job/HBase-1.3-JDK8/org.apache.hbase$hbase-server/48/testReport/junit/org.apache.hadoop.hbase.regionserver/TestDefaultCompactSelection/testCompactionRatio/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593672#comment-15593672 ] Mikhail Antonov commented on HBASE-11819: - bumped to 1.3.1 > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: beginner > Fix For: 2.0.0, 1.3.1 > > Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 > (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, > HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, > HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch, > HBASE-11819v6-branch-1.patch, HBASE-11819v6-master.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11819: Fix Version/s: (was: 1.3.0) 1.3.1 > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: beginner > Fix For: 2.0.0, 1.3.1 > > Attachments: HBASE-11819v4-master.patch, HBASE-11819v5-0.98 > (1).patch, HBASE-11819v5-0.98.patch, HBASE-11819v5-master (1).patch, > HBASE-11819v5-master.patch, HBASE-11819v5-master.patch, > HBASE-11819v5-v0.98.patch, HBASE-11819v5-v1.0.patch, > HBASE-11819v6-branch-1.patch, HBASE-11819v6-master.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16071:
Fix Version/s: (was: 1.3.0)
1.3.1
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.4.0, 1.3.1
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16071) The VisibilityLabelFilter and AccessControlFilter should not count the "delete cell"
[
https://issues.apache.org/jira/browse/HBASE-16071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593664#comment-15593664
]
Mikhail Antonov commented on HBASE-16071:
-
bumped to 1.3.1
> The VisibilityLabelFilter and AccessControlFilter should not count the
> "delete cell"
>
>
> Key: HBASE-16071
> URL: https://issues.apache.org/jira/browse/HBASE-16071
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.0.0
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16071-v1.patch, HBASE-16071-v2.patch,
> HBASE-16071-v3.patch
>
>
> The VisibilityLabelFilter will see and count the "delete cell" if the
> scan.isRaw() returns true, so the (put) cell will be skipped if it has lower
> version than "delete cell"
> The critical code is shown below:
> {code:title=VisibilityLabelFilter.java|borderStyle=solid}
> public ReturnCode filterKeyValue(Cell cell) throws IOException {
> if (curFamily.getBytes() == null
> || !(CellUtil.matchingFamily(cell, curFamily.getBytes(),
> curFamily.getOffset(),
> curFamily.getLength( {
> curFamily.set(cell.getFamilyArray(), cell.getFamilyOffset(),
> cell.getFamilyLength());
> // For this family, all the columns can have max of
> curFamilyMaxVersions versions. No need to
> // consider the older versions for visibility label check.
> // Ideally this should have been done at a lower layer by HBase (?)
> curFamilyMaxVersions = cfVsMaxVersions.get(curFamily);
> // Family is changed. Just unset curQualifier.
> curQualifier.unset();
> }
> if (curQualifier.getBytes() == null
> || !(CellUtil.matchingQualifier(cell, curQualifier.getBytes(),
> curQualifier.getOffset(),
> curQualifier.getLength( {
> curQualifier.set(cell.getQualifierArray(), cell.getQualifierOffset(),
> cell.getQualifierLength());
> curQualMetVersions = 0;
> }
> curQualMetVersions++;
> if (curQualMetVersions > curFamilyMaxVersions) {
> return ReturnCode.SKIP;
> }
> return this.expEvaluator.evaluate(cell) ? ReturnCode.INCLUDE :
> ReturnCode.SKIP;
> }
> {code}
> [VisibilityLabelFilter.java|https://github.com/apache/hbase/blob/d7a4499dfc8b3936a0eca867589fc2b23b597866/hbase-server/src/main/java/org/apache/hadoop/hbase/security/visibility/VisibilityLabelFilter.java]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16754) Regions failing compaction due to referencing non-existent store file
[
https://issues.apache.org/jira/browse/HBASE-16754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mikhail Antonov updated HBASE-16754:
Fix Version/s: 1.4.0
2.0.0
> Regions failing compaction due to referencing non-existent store file
> -
>
> Key: HBASE-16754
> URL: https://issues.apache.org/jira/browse/HBASE-16754
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.2.3
>Reporter: Gary Helmling
>Assignee: Gary Helmling
>Priority: Blocker
> Fix For: 2.0.0, 1.3.0, 1.4.0
>
> Attachments: HBASE-16754.001.patch, HBASE-16754.branch-1.001.patch,
> HBASE-16754.branch-1.2.001.patch
>
>
> Running a mixed read write workload on a recent build off branch-1.3, we are
> seeing compactions occasionally fail with errors like the following (actual
> filenames replaced with placeholders):
> {noformat}
> 16/09/27 16:57:28 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 116
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompactionPolicy.shouldPerformMajorCompaction(RatioBasedCompactionPolicy.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.SortedCompactionPolicy.selectCompaction(SortedCompactionPolicy.java:82)
>
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.select(DefaultStoreEngine.java:107)
>
> at
> org.apache.hadoop.hbase.regionserver.HStore.requestCompaction(HStore.java:1644)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.selectCompaction(CompactSplitThread.java:373)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.access$100(CompactSplitThread.java:59)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:498)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:568)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 16/09/27 17:01:31 ERROR regionserver.CompactSplitThread: Compaction selection
> failed Store = XXX, pri = 115
> java.io.FileNotFoundException: File does not exist:
> hdfs://.../hbase/data/ns/table/region/cf/XXfilenameXX
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1301)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1317)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getReferencedFileStatus(StoreFileInfo.java:342)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getFileStatus(StoreFileInfo.java:355)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFileInfo.getModificationTime(StoreFileInfo.java:360)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile.getModificationTimeStamp(StoreFile.java:321)
>
> at
> org.apache.hadoop.hbase.regionserver.StoreUtils.getLowestTimestamp(StoreUtils.java:63)
> at
> org.apache.hadoop.hbase.regionserver.compactions.RatioBasedCompacti
[jira] [Commented] (HBASE-16744) Procedure V2 - Lock procedures to allow clients to acquire locks on tables/namespaces/regions
[
https://issues.apache.org/jira/browse/HBASE-16744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593640#comment-15593640
]
Hadoop QA commented on HBASE-16744:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 7m 43s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 15 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 7s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 23s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 10m
31s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
20s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-protocol-shaded in master has 22 extant Findbugs
warnings. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 54s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 3s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 9m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 44 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
47m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 26s
{color} | {color:red} hbase-server generated 1 new + 1 unchanged - 0 fixed = 2
total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 35s
{color} | {color:green} hbase-protocol-shaded in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 8s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s
{color} | {color:green} hbase-rsgroup in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 149m 5s {color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {
[jira] [Updated] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[ https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16889: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.1.8 1.2.4 1.4.0 1.3.0 2.0.0 Status: Resolved (was: Patch Available) > Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each > table DDL is incorrect > - > > Key: HBASE-16889 > URL: https://issues.apache.org/jira/browse/HBASE-16889 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 2.0.0, 1.1.7 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4, 1.1.8 > > Attachments: HBASE-16889.v1-branch-1.1.patch, > HBASE-16889.v1-master.patch > > > In the IntegrationTestDDLMasterFailover test, verifyTables is called after > each table DDL. It iterates 3 lists of tables in ConcurrentHashMap > (enabledTables, disabledTables, deletedTables) and tries to do some > verification. This is incorrect, eg. a table in enabledTables map could be > picked up by DeleteTableAction and is disabled, while the verification tries > to check whether table is enabled. This leads to false assertion. > The same for verifyNamespaces(). > The proposed fix is to verify maps only at the end of tests (while no active > DDL operation is going on). During test run, we only verify the target table > before putting into map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16653) Backport HBASE-11393 to all branches which support namespace
[ https://issues.apache.org/jira/browse/HBASE-16653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593631#comment-15593631 ] Guanghao Zhang commented on HBASE-16653: Agreed. Really looking forward to 1.3 release . :-) > Backport HBASE-11393 to all branches which support namespace > > > Key: HBASE-16653 > URL: https://issues.apache.org/jira/browse/HBASE-16653 > Project: HBase > Issue Type: Bug >Affects Versions: 1.4.0, 1.0.5, 1.3.1, 0.98.22, 1.1.7, 1.2.4 >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang > Fix For: 1.4.0 > > Attachments: HBASE-16653-branch-1-v1.patch, > HBASE-16653-branch-1-v2.patch, HBASE-16653-branch-1-v3.patch, > HBASE-16653-branch-1-v4.patch, HBASE-16653-branch-1-v4.patch, > HBASE-16653-branch-1-v5.patch > > > As HBASE-11386 mentioned, the parse code about replication table-cfs config > will be wrong when table name contains namespace and we can only config the > default namespace's tables in the peer. It is a bug for all branches which > support namespace. HBASE-11393 resolved this by use a pb object but it was > only merged to master branch. Other branches still have this problem. I > thought we should fix this bug in all branches which support namespace. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16887) Allow setting different hadoopcheck versions in precommit for different branches
[ https://issues.apache.org/jira/browse/HBASE-16887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593620#comment-15593620 ] Duo Zhang commented on HBASE-16887: --- Fine. Let me put the question there. Thanks. > Allow setting different hadoopcheck versions in precommit for different > branches > > > Key: HBASE-16887 > URL: https://issues.apache.org/jira/browse/HBASE-16887 > Project: HBase > Issue Type: Bug > Components: build >Reporter: Duo Zhang > > http://hbase.apache.org/book.html#hadoop > The supportted hadoop versions are different for different HBase versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16870:
---
Attachment: HBASE-16870-branch-1.3-v1.patch
Reattach v1 patch for branch-1.3 to trigger Hadoop QA.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM-v1.patch,
> HBASE-16870-ADDENDUM.patch, HBASE-16870-branch-1-v1.patch,
> HBASE-16870-branch-1-v1.patch, HBASE-16870-branch-1.1-v1.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3-v1.patch,
> HBASE-16870-branch-1.3-v1.patch, HBASE-16870-branch-1.3.patch,
> HBASE-16870-branch-1.patch, HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16870) Add the metrics of replication sources which were transformed from other dead rs to ReplicationLoad
[
https://issues.apache.org/jira/browse/HBASE-16870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593588#comment-15593588
]
Guanghao Zhang commented on HBASE-16870:
Take a look about the test result of branch-1's Hadoop QA result. There are no
failed replication unit test.
> Add the metrics of replication sources which were transformed from other dead
> rs to ReplicationLoad
> ---
>
> Key: HBASE-16870
> URL: https://issues.apache.org/jira/browse/HBASE-16870
> Project: HBase
> Issue Type: Bug
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16870-ADDENDUM-v1.patch,
> HBASE-16870-ADDENDUM.patch, HBASE-16870-branch-1-v1.patch,
> HBASE-16870-branch-1-v1.patch, HBASE-16870-branch-1.1-v1.patch,
> HBASE-16870-branch-1.1.patch, HBASE-16870-branch-1.3-v1.patch,
> HBASE-16870-branch-1.3.patch, HBASE-16870-branch-1.patch,
> HBASE-16870-v1.patch, HBASE-16870.patch
>
>
> {code}
> private void buildReplicationLoad() {
> // get source
> List sources =
> this.replicationManager.getSources();
> List sourceMetricsList = new ArrayList();
> for (ReplicationSourceInterface source : sources) {
> if (source instanceof ReplicationSource) {
> sourceMetricsList.add(((ReplicationSource)
> source).getSourceMetrics());
> }
> }
> // get sink
> MetricsSink sinkMetrics = this.replicationSink.getSinkMetrics();
> this.replicationLoad.buildReplicationLoad(sourceMetricsList, sinkMetrics);
> }
> {code}
> The buildReplicationLoad method in o.a.h.h.r.r.Replication didn't consider
> the replication source which were transformed from other died rs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16827) Backup.proto refactoring
[
https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593576#comment-15593576
]
Hadoop QA commented on HBASE-16827:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16827 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834581/HBASE-16827-v3.patch |
| JIRA Issue | HBASE-16827 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/4128/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup.proto refactoring
>
>
> Key: HBASE-16827
> URL: https://issues.apache.org/jira/browse/HBASE-16827
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch,
> HBASE-16827-v3.patch,
> org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt
>
>
> Make server_name type ServerName (from HBase.proto)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16892) Use TableName instead of String in SnapshotDescription
[
https://issues.apache.org/jira/browse/HBASE-16892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593570#comment-15593570
]
Hadoop QA commented on HBASE-16892:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 42s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
29m 59s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 54s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 81m 4s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 128m 23s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestDeleteTableProcedure |
| | org.apache.hadoop.hbase.master.TestGetLastFlushedSequenceId |
| | org.apache.hadoop.hbase.master.procedure.TestMasterProcedureEvents |
| | org.apache.hadoop.hbase.master.procedure.TestCloneSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834565/HBASE-16892-v1.patch |
| JIRA Issue | HBASE-16892 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 620d4b2e3159 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f1691e3 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs
[jira] [Updated] (HBASE-16827) Backup.proto refactoring
[ https://issues.apache.org/jira/browse/HBASE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16827: -- Attachment: HBASE-16827-v3.patch v3. This is maximum we can get from .poto optimization w/o significant code refactoring. > Backup.proto refactoring > > > Key: HBASE-16827 > URL: https://issues.apache.org/jira/browse/HBASE-16827 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16827-v1.patch, HBASE-16827-v2.patch, > HBASE-16827-v3.patch, > org.apache.hadoop.hbase.backup.TestIncrementalBackup-output.txt > > > Make server_name type ServerName (from HBase.proto) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593396#comment-15593396 ] Hudson commented on HBASE-16752: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1823 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1823/]) HBASE-16752 addendum. Do not retry large request for client versions (garyh: rev f1691e3d0d0b650d569540f4711f1f0ba2b5701e) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16892) Use TableName instead of String in SnapshotDescription
[ https://issues.apache.org/jira/browse/HBASE-16892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16892: Attachment: HBASE-16892-v1.patch > Use TableName instead of String in SnapshotDescription > -- > > Key: HBASE-16892 > URL: https://issues.apache.org/jira/browse/HBASE-16892 > Project: HBase > Issue Type: Sub-task > Components: snapshots >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Trivial > Fix For: 2.0.0 > > Attachments: HBASE-16892-v0.patch, HBASE-16892-v1.patch > > > mostly find & replace work: > deprecate the SnapshotDescription constructors with the String argument in > favor of the TableName ones. > Replace the TableName.valueOf() around with the new getTableName() > Replace the TableName.getNameAsString() by just passing the TableName -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[
https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593254#comment-15593254
]
Hadoop QA commented on HBASE-16889:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} branch-1.1 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} branch-1.1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} branch-1.1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
12m 30s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 9s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 24s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:35e2245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834473/HBASE-16889.v1-branch-1.1.patch
|
| JIRA Issue | HBASE-16889 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 670cae641061 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/hbase.sh |
| git revision | branch-1.1 / 4e304b3 |
| Default Java | 1.7.0_80 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8
[jira] [Resolved] (HBASE-16248) Remove direct use of Hadoop Path/FileSysem
[ https://issues.apache.org/jira/browse/HBASE-16248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey resolved HBASE-16248. - Resolution: Duplicate Assignee: (was: Sean Busbey) Fix Version/s: (was: 2.0.0) obviated by HBASE-14439 > Remove direct use of Hadoop Path/FileSysem > -- > > Key: HBASE-16248 > URL: https://issues.apache.org/jira/browse/HBASE-16248 > Project: HBase > Issue Type: Improvement > Components: Filesystem Integration >Reporter: Sean Busbey > > review current direct use of Hadoop's FileSystem / Path and move under > improved abstraction. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16250) Remove direct use of Hadoop Path/FileSysem 2/5
[
https://issues.apache.org/jira/browse/HBASE-16250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-16250.
-
Resolution: Duplicate
Assignee: (was: Sean Busbey)
Fix Version/s: (was: 2.0.0)
obviated by sub-tasks of HBASE-14439
> Remove direct use of Hadoop Path/FileSysem 2/5
> --
>
> Key: HBASE-16250
> URL: https://issues.apache.org/jira/browse/HBASE-16250
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/io/hfile/
> org/apache/hadoop/hbase/mapreduce/
> org/apache/hadoop/hbase/mob/mapreduce/
> org/apache/hadoop/hbase/master/
> org/apache/hadoop/hbase/master/cleaner/
> org/apache/hadoop/hbase/master/procedure/
> org/apache/hadoop/hbase/master/snapshot/
> org/apache/hadoop/hbase/mob/
> org/apache/hadoop/hbase/mob/compactions/
> org/apache/hadoop/hbase/namespace/
> org/apache/hadoop/hbase/procedure2/
> org/apache/hadoop/hbase/procedure2/store/wal/
> org/apache/hadoop/hbase/protobuf/
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16251) Remove direct use of Hadoop Path/FileSysem 3/5
[
https://issues.apache.org/jira/browse/HBASE-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-16251.
-
Resolution: Duplicate
Assignee: (was: Sean Busbey)
Fix Version/s: (was: 2.0.0)
obviated by subtasks of HBASE-14439
> Remove direct use of Hadoop Path/FileSysem 3/5
> --
>
> Key: HBASE-16251
> URL: https://issues.apache.org/jira/browse/HBASE-16251
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/regionserver/
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593219#comment-15593219
]
Enis Soztutar commented on HBASE-16894:
---
bq. or should we pass a custom split list?
I think the users will generally not know these splits points. It should be
hbase which keeps track of region boundaries and internally split points within
the regions. As a user, the API should be like create a split for every 5GB, or
there should be a total of X map tasks.
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593210#comment-15593210 ] Hudson commented on HBASE-16752: FAILURE: Integrated in Jenkins build HBase-1.4 #483 (See [https://builds.apache.org/job/HBase-1.4/483/]) HBASE-16752 addendum. Do not retry large request for client versions (garyh: rev 0117ed9d78820abecf066d742782de1dd49f309e) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16253) Remove direct use of Hadoop Path/FileSysem 5/5
[
https://issues.apache.org/jira/browse/HBASE-16253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-16253.
-
Resolution: Duplicate
Assignee: (was: Sean Busbey)
Fix Version/s: (was: 2.0.0)
obviated by subtask of HBASE-14439
> Remove direct use of Hadoop Path/FileSysem 5/5
> --
>
> Key: HBASE-16253
> URL: https://issues.apache.org/jira/browse/HBASE-16253
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/snapshot/
> org/apache/hadoop/hbase/util/
> org/apache/hadoop/hbase/util/hbck/
> org/apache/hadoop/hbase/wal/
> org/apache/hadoop/hbase/zookeeper/
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16252) Remove direct use of Hadoop Path/FileSysem 4/5
[
https://issues.apache.org/jira/browse/HBASE-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-16252.
-
Resolution: Duplicate
Assignee: (was: Sean Busbey)
Fix Version/s: (was: 2.0.0)
obviated by subtask of HBASE-14439
> Remove direct use of Hadoop Path/FileSysem 4/5
> --
>
> Key: HBASE-16252
> URL: https://issues.apache.org/jira/browse/HBASE-16252
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/regionserver/compactions/
> org/apache/hadoop/hbase/regionserver/wal/
> org/apache/hadoop/hbase/replication/
> org/apache/hadoop/hbase/replication/regionserver/
> org/apache/hadoop/hbase/rest/
> org/apache/hadoop/hbase/security/
> org/apache/hadoop/hbase/security/access/
> org/apache/hadoop/hbase/security/token/
> org/apache/hadoop/hbase/test/
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HBASE-16249) Remove direct use of Hadoop Path/FileSysem 1/5
[
https://issues.apache.org/jira/browse/HBASE-16249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Busbey resolved HBASE-16249.
-
Resolution: Duplicate
Assignee: (was: Sean Busbey)
Fix Version/s: (was: 2.0.0)
obviated by subtask of HBASE-14439
> Remove direct use of Hadoop Path/FileSysem 1/5
> --
>
> Key: HBASE-16249
> URL: https://issues.apache.org/jira/browse/HBASE-16249
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Sean Busbey
>
> Remove or document exceptions for direct use of Hadoop FileSystem APIs in
> {code}
> org/apache/hadoop/hbase/
> org/apache/hadoop/hbase/backup/
> org/apache/hadoop/hbase/backup/example/
> org/apache/hadoop/hbase/client/
> org/apache/hadoop/hbase/client/coprocessor/
> org/apache/hadoop/hbase/coordination/
> org/apache/hadoop/hbase/coprocessor/
> org/apache/hadoop/hbase/fs/
> org/apache/hadoop/hbase/io/
> org/apache/hadoop/hbase/io/asyncfs/
> org/apache/hadoop/hbase/io/encoding/
> org/apache/hadoop/hbase/mapred/
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16889) Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each table DDL is incorrect
[ https://issues.apache.org/jira/browse/HBASE-16889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-16889: --- Status: Patch Available (was: Open) > Proc-V2: verifyTables in the IntegrationTestDDLMasterFailover test after each > table DDL is incorrect > - > > Key: HBASE-16889 > URL: https://issues.apache.org/jira/browse/HBASE-16889 > Project: HBase > Issue Type: Bug > Components: proc-v2 >Affects Versions: 1.1.7, 2.0.0 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Attachments: HBASE-16889.v1-branch-1.1.patch, > HBASE-16889.v1-master.patch > > > In the IntegrationTestDDLMasterFailover test, verifyTables is called after > each table DDL. It iterates 3 lists of tables in ConcurrentHashMap > (enabledTables, disabledTables, deletedTables) and tries to do some > verification. This is incorrect, eg. a table in enabledTables map could be > picked up by DeleteTableAction and is disabled, while the verification tries > to check whether table is enabled. This leads to false assertion. > The same for verifyNamespaces(). > The proposed fix is to verify maps only at the end of tests (while no active > DDL operation is going on). During test run, we only verify the target table > before putting into map. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16752) Upgrading from 1.2 to 1.3 can lead to replication failures due to difference in RPC size limit
[ https://issues.apache.org/jira/browse/HBASE-16752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593168#comment-15593168 ] Hudson commented on HBASE-16752: SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #54 (See [https://builds.apache.org/job/HBase-1.3-JDK8/54/]) HBASE-16752 addendum. Do not retry large request for client versions (garyh: rev 3efa3b911294fe0a42440e2aef3865898d91acc0) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/RequestTooBigException.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcServer.java > Upgrading from 1.2 to 1.3 can lead to replication failures due to difference > in RPC size limit > -- > > Key: HBASE-16752 > URL: https://issues.apache.org/jira/browse/HBASE-16752 > Project: HBase > Issue Type: Bug > Components: Replication, rpc >Affects Versions: 2.0.0, 1.3.0 >Reporter: Ashu Pachauri >Assignee: Ashu Pachauri > Fix For: 2.0.0, 1.3.0, 1.4.0 > > Attachments: HBASE-16752.V1.patch, HBASE-16752.V2.patch, > HBASE-16752.addendum.patch > > > In HBase 1.2, we don't limit size of a single RPC but in 1.3 we limit it by > default to 256 MB. This means that during upgrade scenarios (or when source > is 1.2 peer is already on 1.3), it's possible to encounter a situation where > we try to send an rpc with size greater than 256 MB because we never unroll a > WALEdit while sending replication traffic. > RpcServer throws the underlying exception locally, but closes the connection > with returning the underlying error to the client, and client only sees a > "Broken pipe" error. > I am not sure what is the proper fix here (or if one is needed) to make sure > this does not happen, but we should return the underlying exception to the > RpcClient, because without it, it can be difficult to diagnose the problem, > especially for someone new to HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16905) fold FSUtil classes into fs integration package
[ https://issues.apache.org/jira/browse/HBASE-16905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16905: Description: pull the FSUtils classes into the fs integration package. ideally make it package private to ensure others aren't using it directly. hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/fs/FSUtilsWithRetries.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java was: pull the FSUtils classes into the fs integration package. ideally make it package private to ensure others aren't using it directly. hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/fs/FSUtilsWithRetries.java > fold FSUtil classes into fs integration package > --- > > Key: HBASE-16905 > URL: https://issues.apache.org/jira/browse/HBASE-16905 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Sean Busbey > > pull the FSUtils classes into the fs integration package. ideally make it > package private to ensure others aren't using it directly. > hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java > hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java > hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java > hbase-server/src/main/java/org/apache/hadoop/hbase/fs/FSUtilsWithRetries.java > hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16908) investigate flakey TestQuotaThrottle
huaxiang sun created HBASE-16908: Summary: investigate flakey TestQuotaThrottle Key: HBASE-16908 URL: https://issues.apache.org/jira/browse/HBASE-16908 Project: HBase Issue Type: Bug Components: hbase Affects Versions: 2.0.0 Reporter: huaxiang sun Assignee: huaxiang sun Priority: Minor find out the root cause for TestQuotaThrottle failures. https://builds.apache.org/job/HBASE-Find-Flaky-Tests/lastSuccessfulBuild/artifact/dashboard.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16907) ensure merge operations don't reference filesystem or legacy implementation directly
Sean Busbey created HBASE-16907: --- Summary: ensure merge operations don't reference filesystem or legacy implementation directly Key: HBASE-16907 URL: https://issues.apache.org/jira/browse/HBASE-16907 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey ensure region merging can be done without directly referencing backing filesystem. update masterstorage / regionstorage apis as needed. hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/Merge.java also changes in HRegion, HStore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16906) ensure split operation doesn't directly reference fs / legacy integrations
[ https://issues.apache.org/jira/browse/HBASE-16906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16906: Description: ensure that we can do region splitting without referencing backing filesystem hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java also changes in HRegion, HStore was: ensure that we can do region splitting without referencing backing filesystem hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java also changes in HRegion > ensure split operation doesn't directly reference fs / legacy integrations > -- > > Key: HBASE-16906 > URL: https://issues.apache.org/jira/browse/HBASE-16906 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Sean Busbey > > ensure that we can do region splitting without referencing backing filesystem > hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java > hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java > also changes in HRegion, HStore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16906) ensure split operation doesn't directly reference fs / legacy integrations
Sean Busbey created HBASE-16906: --- Summary: ensure split operation doesn't directly reference fs / legacy integrations Key: HBASE-16906 URL: https://issues.apache.org/jira/browse/HBASE-16906 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey ensure that we can do region splitting without referencing backing filesystem hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransactionImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/RegionSplitter.java also changes in HRegion -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16905) fold FSUtil classes into fs integration package
Sean Busbey created HBASE-16905: --- Summary: fold FSUtil classes into fs integration package Key: HBASE-16905 URL: https://issues.apache.org/jira/browse/HBASE-16905 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey pull the FSUtils classes into the fs integration package. ideally make it package private to ensure others aren't using it directly. hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSHDFSUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSMapRUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/fs/FSUtilsWithRetries.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16902) Remove directory layout/ filesystem references form hbck tool
[
https://issues.apache.org/jira/browse/HBASE-16902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Umesh Agashe updated HBASE-16902:
-
Description:
Remove directory layout/ filesystem references form hbck tool. List of files:
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java
{code}
was:Remove directory layout/ filesystem references form hbck tool
> Remove directory layout/ filesystem references form hbck tool
> -
>
> Key: HBASE-16902
> URL: https://issues.apache.org/jira/browse/HBASE-16902
> Project: HBase
> Issue Type: Sub-task
> Components: Filesystem Integration
>Reporter: Umesh Agashe
>
> Remove directory layout/ filesystem references form hbck tool. List of files:
> {code}
> hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/HFileCorruptionChecker.java
> hbase-server/src/main/java/org/apache/hadoop/hbase/util/hbck/OfflineMetaRepair.java
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16904) remove directory layout / fs references from snapshots
Sean Busbey created HBASE-16904: --- Summary: remove directory layout / fs references from snapshots Key: HBASE-16904 URL: https://issues.apache.org/jira/browse/HBASE-16904 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey ensure snapshot code works through the MasterStorage / RegionStorage APIs and not directly on backing filesystem. hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16902) Remove directory layout/ filesystem references form hbck tool
Umesh Agashe created HBASE-16902: Summary: Remove directory layout/ filesystem references form hbck tool Key: HBASE-16902 URL: https://issues.apache.org/jira/browse/HBASE-16902 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Umesh Agashe Remove directory layout/ filesystem references form hbck tool -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16903) ensure WAL code no longer presumes colocation with region storage
Sean Busbey created HBASE-16903: --- Summary: ensure WAL code no longer presumes colocation with region storage Key: HBASE-16903 URL: https://issues.apache.org/jira/browse/HBASE-16903 Project: HBase Issue Type: Sub-task Components: Filesystem Integration, wal Reporter: Sean Busbey make sure that places where the WAL is using per-region storage do so without breaking the RegionStorage abstraction. e.g. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java Probably more in HRegion or SplitLogManager too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16901) remove directory layout / fs references from bulkload code
Sean Busbey created HBASE-16901: --- Summary: remove directory layout / fs references from bulkload code Key: HBASE-16901 URL: https://issues.apache.org/jira/browse/HBASE-16901 Project: HBase Issue Type: Sub-task Reporter: Sean Busbey ensure bulk load implementation relies on MasterStorage / RegionStorage apis and make changes needed to support that. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SecureBulkLoadManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java also probably changes in HRegion -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16900) decouple Replication from backing files of WAL
Sean Busbey created HBASE-16900: --- Summary: decouple Replication from backing files of WAL Key: HBASE-16900 URL: https://issues.apache.org/jira/browse/HBASE-16900 Project: HBase Issue Type: Sub-task Components: Filesystem Integration, Replication, wal Reporter: Sean Busbey update replication implementation to rely on abstraction of WAL storage rather than reading files itself -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16899) remove directory layout / fs references from compaction
Sean Busbey created HBASE-16899: --- Summary: remove directory layout / fs references from compaction Key: HBASE-16899 URL: https://issues.apache.org/jira/browse/HBASE-16899 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey Ensure compaction code no longer has direct references to legacy implementation or to directories / filesystems directly * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ServerProtobufUtil.java Will also require changes in HRegion and HStore -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16698) Performance issue: handlers stuck waiting for CountDownLatch inside WALKey#getWriteEntry under high writing workload
[
https://issues.apache.org/jira/browse/HBASE-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593072#comment-15593072
]
Enis Soztutar commented on HBASE-16698:
---
Time to resolve this?
[~carp84] thanks for the explanations earlier. One last question, did you do
the tests with 2.0 code or branch-1 code? The issue description says:
{code}
Notice that for master (2.0) branch since we already change the write pipeline
to sync before writing memstore (HBASE-15158), this issue only exists for the
ASYNC_WAL writes scenario.
{code}
so we have committed this only for ASYNC_WAL for the master code? We wait for
the whole sync() to happen before proceeding to the memstore inserts anyway in
master.
> Performance issue: handlers stuck waiting for CountDownLatch inside
> WALKey#getWriteEntry under high writing workload
>
>
> Key: HBASE-16698
> URL: https://issues.apache.org/jira/browse/HBASE-16698
> Project: HBase
> Issue Type: Improvement
> Components: Performance
>Affects Versions: 1.2.3
>Reporter: Yu Li
>Assignee: Yu Li
> Fix For: 2.0.0
>
> Attachments: HBASE-16698.branch-1.patch,
> HBASE-16698.branch-1.v2.patch, HBASE-16698.branch-1.v2.patch,
> HBASE-16698.patch, HBASE-16698.v2.patch, hadoop0495.et2.jstack
>
>
> As titled, on our production environment we observed 98 out of 128 handlers
> get stuck waiting for the CountDownLatch {{seqNumAssignedLatch}} inside
> {{WALKey#getWriteEntry}} under a high writing workload.
> After digging into the problem, we found that the problem is mainly caused by
> advancing mvcc in the append logic. Below is some detailed analysis:
> Under current branch-1 code logic, all batch puts will call
> {{WALKey#getWriteEntry}} after appending edit to WAL, and
> {{seqNumAssignedLatch}} is only released when the relative append call is
> handled by RingBufferEventHandler (see {{FSWALEntry#stampRegionSequenceId}}).
> Because currently we're using a single event handler for the ringbuffer, the
> append calls are handled one by one (actually lot's of our current logic
> depending on this sequential dealing logic), and this becomes a bottleneck
> under high writing workload.
> The worst part is that by default we only use one WAL per RS, so appends on
> all regions are dealt with in sequential, which causes contention among
> different regions...
> To fix this, we could also take use of the "sequential appends" mechanism,
> that we could grab the WriteEntry before publishing append onto ringbuffer
> and use it as sequence id, only that we need to add a lock to make "grab
> WriteEntry" and "append edit" a transaction. This will still cause contention
> inside a region but could avoid contention between different regions. This
> solution is already verified in our online environment and proved to be
> effective.
> Notice that for master (2.0) branch since we already change the write
> pipeline to sync before writing memstore (HBASE-15158), this issue only
> exists for the ASYNC_WAL writes scenario.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16898) remove directory layout / fs references from MOB
Sean Busbey created HBASE-16898: --- Summary: remove directory layout / fs references from MOB Key: HBASE-16898 URL: https://issues.apache.org/jira/browse/HBASE-16898 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey update mob related classes and utilities to use MasterStorage / RegionStorage and update those apis as needed to do so. hbase-server/src/main/java/org/apache/hadoop/hbase/mob/MobUtils.java hbase-server/src/main/java/org/apache/hadoop/hbase/mob/compactions/MobCompactor.java hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepJob.java hbase-server/src/main/java/org/apache/hadoop/hbase/mob/mapreduce/SweepReducer.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16897) remove direct layout/fs references from mapreduce utilities
Sean Busbey created HBASE-16897: --- Summary: remove direct layout/fs references from mapreduce utilities Key: HBASE-16897 URL: https://issues.apache.org/jira/browse/HBASE-16897 Project: HBase Issue Type: Sub-task Components: Filesystem Integration, mapreduce Reporter: Sean Busbey Ensure MapReduce helpers / utilities do not directly reference filesystem or legacy implementation. make additions / changes to MasterStorage and RegionStorage as needed. starting point: hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableSnapshotInputFormatImpl.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16896) remove directory layout / fs references from TableSnapshotScanner
Sean Busbey created HBASE-16896: --- Summary: remove directory layout / fs references from TableSnapshotScanner Key: HBASE-16896 URL: https://issues.apache.org/jira/browse/HBASE-16896 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey remove use of FSUtils and direct filesystem from TableSnapshotScanner and add any needed APIs for MasterStorage / RegionStorage -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16894) Create more than 1 split per region, generalize HBASE-12590
[
https://issues.apache.org/jira/browse/HBASE-16894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593036#comment-15593036
]
Esteban Gutierrez commented on HBASE-16894:
---
+1 I have seen this request few times and one solution is to override
TableInputFormat#getSplits are you suggesting just to add a tunable parameter
only? or should we pass a custom split list?
> Create more than 1 split per region, generalize HBASE-12590
> ---
>
> Key: HBASE-16894
> URL: https://issues.apache.org/jira/browse/HBASE-16894
> Project: HBase
> Issue Type: Improvement
>Reporter: Enis Soztutar
>
> A common request from users is to be able to better control how many map
> tasks are created per region. Right now, it is always 1 region = 1 input
> split = 1 map task. Same goes for Spark since it uses the TIF. With region
> sizes as large as 50 GBs, it is desirable to be able to create more than 1
> split per region.
> HBASE-12590 adds a config property for MR jobs to be able to handle skew in
> region sizes. The algorithm is roughly:
> {code}
> If (region size >= average size*ratio) : cut the region into two MR input
> splits
> If (average size <= region size < average size*ratio) : one region as one MR
> input split
> If (sum of several continuous regions size < average size * ratio): combine
> these regions into one MR input split.
> {code}
> Although we can set data skew ratio to be 0.5 or something to abuse
> HBASE-12590 into creating more than 1 split task per region, it is not ideal.
> But there is no way to create more with the patch as it is. For example we
> cannot create more than 2 tasks per region.
> If we want to fix this properly, we should extend the approach in
> HBASE-12590, and make it so that the client can specify the desired num of
> mappers, or desired split size, and the TIF generates the splits based on the
> current region sizes very similar to the algorithm in HBASE-12590, but a more
> generic way. This also would eliminate the hand tuning of data skew ratio.
> We also can think about the guidepost approach that Phoenix has in the stats
> table which is used for exactly this purpose. Right now, the region can be
> split into powers of two assuming uniform distribution within the region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16895) Remove directory layout / fs references from HBase IO package
Sean Busbey created HBASE-16895: --- Summary: Remove directory layout / fs references from HBase IO package Key: HBASE-16895 URL: https://issues.apache.org/jira/browse/HBASE-16895 Project: HBase Issue Type: Sub-task Components: Filesystem Integration Reporter: Sean Busbey Priority: Minor Fix For: hbase-14439 remove direct references to the filesystem directories and/or legacy implementation from classes in the hbase io package and make needed supporting changes in the MasterStorage / RegionStorage interface. (alternatively, remove whatever is using them w/reasoning) hbase-server/src/main/java/org/apache/hadoop/hbase/io/FileLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/HFileLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/WALLink.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutput.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputHelper.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16892) Use TableName instead of String in SnapshotDescription
[
https://issues.apache.org/jira/browse/HBASE-16892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15593011#comment-15593011
]
Hadoop QA commented on HBASE-16892:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 39s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s
{color} | {color:red} hbase-server in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1 or 3.0.0-alpha1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 52s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 135m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.namespace.TestNamespaceAuditor |
| | org.apache.hadoop.hbase.TestHBaseTestingUtility |
| | org.apache.hadoop.hbase.master.cleaner.TestSnapshotFromMaster |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12834479/HBASE-16892-v0.patch |
| JIRA Issue | HBASE-16892 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 329ff5be950a 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f1691e3 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBA
[jira] [Updated] (HBASE-16877) Remove directory layout/ filesystem references form Master
[ https://issues.apache.org/jira/browse/HBASE-16877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16877: Description: Remove directory layout/ filesystem references form Master and add required APIs to MasterStorage/ RegionStorage hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java was:Remove directory layout/ filesystem references form Master and add required APIs to MasterStorage/ RegionStorage > Remove directory layout/ filesystem references form Master > -- > > Key: HBASE-16877 > URL: https://issues.apache.org/jira/browse/HBASE-16877 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > > Remove directory layout/ filesystem references form Master and add required > APIs to MasterStorage/ RegionStorage > hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterStatusServlet.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterWalManager.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionPlacementMaintainer.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileLinkCleaner.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureEnv.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java > hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[ https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16862: Resolution: Fixed Fix Version/s: hbase-14439 Status: Resolved (was: Patch Available) > Remove directory layout/ filesystem references form the code in > master/procedure directory > -- > > Key: HBASE-16862 > URL: https://issues.apache.org/jira/browse/HBASE-16862 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: hbase-14439 > > Attachments: HBASE-16862-hbase-14439.v1.patch, > HBASE-16862-hbase-14439.v2.patch > > > Remove directory layout/ filesystem references form the code in > master/procedure directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16862) Remove directory layout/ filesystem references form the code in master/procedure directory
[ https://issues.apache.org/jira/browse/HBASE-16862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-16862: - Attachment: HBASE-16862-hbase-14439.v2.patch Fixed FindBugs problems detected in last patch. > Remove directory layout/ filesystem references form the code in > master/procedure directory > -- > > Key: HBASE-16862 > URL: https://issues.apache.org/jira/browse/HBASE-16862 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Attachments: HBASE-16862-hbase-14439.v1.patch, > HBASE-16862-hbase-14439.v2.patch > > > Remove directory layout/ filesystem references form the code in > master/procedure directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16789) Remove directory layout/ filesystem references from CompactionTool
[ https://issues.apache.org/jira/browse/HBASE-16789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Busbey updated HBASE-16789: Resolution: Fixed Fix Version/s: hbase-14439 Status: Resolved (was: Patch Available) > Remove directory layout/ filesystem references from CompactionTool > -- > > Key: HBASE-16789 > URL: https://issues.apache.org/jira/browse/HBASE-16789 > Project: HBase > Issue Type: Sub-task > Components: Filesystem Integration >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: hbase-14439 > > Attachments: HBASE-16789-hbase-14439.v1.patch > > > Remove directory layout/ filesystem references from CompactionTool and use > APIs provided by MasterStorage/ RegionStorage instead. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16871) Procedure v2 - add waiting procs back to the queue after restart
[ https://issues.apache.org/jira/browse/HBASE-16871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592896#comment-15592896 ] Hudson commented on HBASE-16871: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1822 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1822/]) HBASE-16871 Procedure v2 - add waiting procs back to the queue after (matteo.bertozzi: rev 553373671b736540a6f53de4f57fc5e0ca7373a7) * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java * (edit) hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java * (edit) hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/TestProcedureEvents.java > Procedure v2 - add waiting procs back to the queue after restart > > > Key: HBASE-16871 > URL: https://issues.apache.org/jira/browse/HBASE-16871 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Affects Versions: 2.0.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16871-v0.patch, HBASE-16871-v1.patch, > HBASE-16871-v1.patch > > > Procs in WAITING_TIMEOUT state don't get re-added to the queue after restart. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16893) Use Iterables.removeIf instead of Iterator.remove in HBase filters
[ https://issues.apache.org/jira/browse/HBASE-16893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Yokota updated HBASE-16893: -- Attachment: HBASE-16893.master.002.patch > Use Iterables.removeIf instead of Iterator.remove in HBase filters > -- > > Key: HBASE-16893 > URL: https://issues.apache.org/jira/browse/HBASE-16893 > Project: HBase > Issue Type: Improvement >Reporter: Robert Yokota >Priority: Minor > Attachments: HBASE-16893.master.001.patch, > HBASE-16893.master.002.patch > > > This is a performance improvement to use Iterables.removeIf in the > filterRowCells method of DependentColumnFilter as described here: > https://rayokota.wordpress.com/2016/10/20/tips-on-writing-custom-hbase-filters/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16893) Use Iterables.removeIf instead of Iterator.remove in HBase filters
[ https://issues.apache.org/jira/browse/HBASE-16893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15592837#comment-15592837 ] Robert Yokota commented on HBASE-16893: --- Ok, patch with Collection.removeIf is attached. > Use Iterables.removeIf instead of Iterator.remove in HBase filters > -- > > Key: HBASE-16893 > URL: https://issues.apache.org/jira/browse/HBASE-16893 > Project: HBase > Issue Type: Improvement >Reporter: Robert Yokota >Priority: Minor > Attachments: HBASE-16893.master.001.patch, > HBASE-16893.master.002.patch > > > This is a performance improvement to use Iterables.removeIf in the > filterRowCells method of DependentColumnFilter as described here: > https://rayokota.wordpress.com/2016/10/20/tips-on-writing-custom-hbase-filters/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
