[jira] [Commented] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464170#comment-15464170 ] Phil Yang commented on HBASE-16505: --- Ok, I'll add synchronized and some tests. And if a thread blocked on getResult's wait and another thread call reset, the result will be wrong, right? So I think we need notifyAll in reset? > Add AsyncRegion interface to pass deadline and support async operations > --- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch, > HBASE-16505-v3.patch, HBASE-16505-v4.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Comment: was deleted
(was: upload an implementation doc in
https://docs.google.com/document/d/1VIpYS7c1Dsp38oxX3JnWUjnV0pV-8PgF9pHhz1Cpu0Y/edit
)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[
https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464042#comment-15464042
]
Yu Li commented on HBASE-16505:
---
>From the findbugs output, I think we should set
>{{RegionOperationContext#reset}} method to synchronized, please check. And
>shall we add some UT cases for methods like reset, getResults, etc.?
>[~yangzhe1991]
> Add AsyncRegion interface to pass deadline and support async operations
> ---
>
> Key: HBASE-16505
> URL: https://issues.apache.org/jira/browse/HBASE-16505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch,
> HBASE-16505-v3.patch, HBASE-16505-v4.patch
>
>
> If we want to know the correct setting of timeout in read/write path, we need
> add a new parameter in operation-methods of Region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-15968) MVCC-sensitive semantics of versions
[
https://issues.apache.org/jira/browse/HBASE-15968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Phil Yang updated HBASE-15968:
--
Summary: MVCC-sensitive semantics of versions (was: Strong semantics of
versions)
> MVCC-sensitive semantics of versions
>
>
> Key: HBASE-15968
> URL: https://issues.apache.org/jira/browse/HBASE-15968
> Project: HBase
> Issue Type: New Feature
>Reporter: Phil Yang
>Assignee: Phil Yang
>
> In HBase book, we have a section in Versions called "Current Limitations" see
> http://hbase.apache.org/book.html#_current_limitations
> {quote}
> 28.3. Current Limitations
> 28.3.1. Deletes mask Puts
> Deletes mask puts, even puts that happened after the delete was entered. See
> HBASE-2256. Remember that a delete writes a tombstone, which only disappears
> after then next major compaction has run. Suppose you do a delete of
> everything ⇐ T. After this you do a new put with a timestamp ⇐ T. This put,
> even if it happened after the delete, will be masked by the delete tombstone.
> Performing the put will not fail, but when you do a get you will notice the
> put did have no effect. It will start working again after the major
> compaction has run. These issues should not be a problem if you use
> always-increasing versions for new puts to a row. But they can occur even if
> you do not care about time: just do delete and put immediately after each
> other, and there is some chance they happen within the same millisecond.
> 28.3.2. Major compactions change query results
> …create three cell versions at t1, t2 and t3, with a maximum-versions
> setting of 2. So when getting all versions, only the values at t2 and t3 will
> be returned. But if you delete the version at t2 or t3, the one at t1 will
> appear again. Obviously, once a major compaction has run, such behavior will
> not be the case anymore… (See Garbage Collection in Bending time in HBase.)
> {quote}
> These limitations result from the current implementation on multi-versions:
> we only consider timestamp, no matter when it comes; we will not remove old
> version immediately if there are enough number of new versions.
> So we can get a stronger semantics of versions by two guarantees:
> 1, Delete will not mask Put that comes after it.
> 2, If a version is masked by enough number of higher versions (VERSIONS in
> cf's conf), it will never be seen any more.
> Some examples for understanding:
> (delete t<=3 means use Delete.addColumns to delete all versions whose ts is
> not greater than 3, and delete t3 means use Delete.addColumn to delete the
> version whose ts=3)
> case 1: put t2 -> put t3 -> delete t<=3 -> put t1, and we will get t1 because
> the put is after delete.
> case 2: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3, and we will
> always get t2 no matter if there is a major compaction, because t1 is masked
> when we put t3 so t1 will never be seen.
> case 3: maxversion=2, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get nothing.
> case 4: maxversion=3, put t1 -> put t2 -> put t3 -> delete t2 -> delete t3,
> and we will get t1 because it is not masked.
> case 5: maxversion=2, put t1 -> put t2 -> put t3 -> delete t3 -> put t1, and
> we can get t3+t1 because when we put t1 at second time it is the 2nd latest
> version and it can be read.
> case 6:maxversion=2, put t3->put t2->put t1, and we will get t3+t2 just like
> what we can get now, ts is still the key of versions.
> Different VERSIONS may result in different results even the size of result is
> smaller than VERSIONS(see case 3 and 4). So Get/Scan.setMaxVersions will be
> handled at end after we read correct data according to CF's VERSIONS setting.
> The semantics is different from the current HBase, and we may need more logic
> to support the new semantic, so it is configurable and default is disabled.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16556: --- Resolution: Fixed Status: Resolved (was: Patch Available) > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.branch-1.v0.patch, HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-16505: -- Attachment: HBASE-16505-v4.patch > Add AsyncRegion interface to pass deadline and support async operations > --- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch, > HBASE-16505-v3.patch, HBASE-16505-v4.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16460) Can't rebuild the BucketAllocator's data structures when BucketCache use FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-16460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15464003#comment-15464003
]
Hadoop QA commented on HBASE-16460:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s
{color} | {color:green} master passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 35s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 10s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 137m 28s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegion |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-05 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827029/16460.v6.patch |
| JIRA Issue | HBASE-16460 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 7f1ff286bf1d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / ab07f00 |
[jira] [Commented] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[
https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463995#comment-15463995
]
Hadoop QA commented on HBASE-16505:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s {color}
| {color:red} HBASE-16505 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12827036/HBASE-16505-v3.patch |
| JIRA Issue | HBASE-16505 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3417/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add AsyncRegion interface to pass deadline and support async operations
> ---
>
> Key: HBASE-16505
> URL: https://issues.apache.org/jira/browse/HBASE-16505
> Project: HBase
> Issue Type: Sub-task
>Reporter: Phil Yang
>Assignee: Phil Yang
> Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch,
> HBASE-16505-v3.patch
>
>
> If we want to know the correct setting of timeout in read/write path, we need
> add a new parameter in operation-methods of Region.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463988#comment-15463988 ] Phil Yang commented on HBASE-16505: --- Oh, I copy the signature from RPCServer's call, and it does not throw ServiceException. We should add it back. > Add AsyncRegion interface to pass deadline and support async operations > --- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch, > HBASE-16505-v3.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16505) Add AsyncRegion interface to pass deadline and support async operations
[ https://issues.apache.org/jira/browse/HBASE-16505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Phil Yang updated HBASE-16505: -- Attachment: HBASE-16505-v3.patch Add ServiceException back in call of RpcServerInterface > Add AsyncRegion interface to pass deadline and support async operations > --- > > Key: HBASE-16505 > URL: https://issues.apache.org/jira/browse/HBASE-16505 > Project: HBase > Issue Type: Sub-task >Reporter: Phil Yang >Assignee: Phil Yang > Attachments: HBASE-16505-v1.patch, HBASE-16505-v2.patch, > HBASE-16505-v3.patch > > > If we want to know the correct setting of timeout in read/write path, we need > add a new parameter in operation-methods of Region. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463977#comment-15463977 ] ChiaPing Tsai commented on HBASE-16556: --- All failed tests are passed locally. > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.branch-1.v0.patch, HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463974#comment-15463974 ] Colin Ma commented on HBASE-16414: -- [~apurtell],[~ram_krish], the patch is updated according to the propose on HBASE-9351, add ConnectionHeaderResponse to deal with the Crypto AES negotiation. You can also leave the comments on the [review board|https://reviews.apache.org/r/51089/], thanks for your help. > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16414) Improve performance for RPC encryption with Apache Common Crypto
[ https://issues.apache.org/jira/browse/HBASE-16414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HBASE-16414: - Attachment: HBASE-16414.002.patch > Improve performance for RPC encryption with Apache Common Crypto > > > Key: HBASE-16414 > URL: https://issues.apache.org/jira/browse/HBASE-16414 > Project: HBase > Issue Type: Improvement > Components: IPC/RPC >Affects Versions: 2.0.0 >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HBASE-16414.001.patch, HBASE-16414.002.patch, > HbaseRpcEncryptionWithCrypoto.docx > > > Hbase RPC encryption is enabled by setting “hbase.rpc.protection” to > "privacy". With the token authentication, it utilized DIGEST-MD5 mechanisms > for secure authentication and data protection. For DIGEST-MD5, it uses DES, > 3DES or RC4 to do encryption and it is very slow, especially for Scan. This > will become the bottleneck of the RPC throughput. > Apache Commons Crypto is a cryptographic library optimized with AES-NI. It > provides Java API for both cipher level and Java stream level. Developers can > use it to implement high performance AES encryption/decryption with the > minimum code and effort. Compare with the current implementation of > org.apache.hadoop.hbase.io.crypto.aes.AES, Crypto supports both JCE Cipher > and OpenSSL Cipher which is better performance than JCE Cipher. User can > configure the cipher type and the default is JCE Cipher. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463870#comment-15463870
]
Anoop Sam John commented on HBASE-14882:
[~tedyu], [~enis] Any suggestion for the new class name?
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14882) Provide a Put API that adds the provided family, qualifier, value without copying
[
https://issues.apache.org/jira/browse/HBASE-14882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463868#comment-15463868
]
Anoop Sam John commented on HBASE-14882:
InputsImmutableCell -> I dont think this reflect the actual thing. It is not
that this cell is made of Immutable bytes is the key.. Even when we create KV
also (passing a byte[]) we assume it is immutable. Just refer that.. But
there we assume the entire key, value etc are in single contiguous byte[].
Here each component (RK, CF, value etc) of the Cell is separate byte[].. And
this class is implementing Cell this way. I believe we need a name in that
way. Sorry am not good in naming. :-(
Do we need CellUtil API? I guess no. U can keep the new Class (implementing
Cell) in same place as of KeyValue. Just create new instance of that within
Put#addImmutable.
> Provide a Put API that adds the provided family, qualifier, value without
> copying
> -
>
> Key: HBASE-14882
> URL: https://issues.apache.org/jira/browse/HBASE-14882
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Jerry He
>Assignee: Xiang Li
> Fix For: 2.0.0
>
> Attachments: HBASE-14882.master.000.patch,
> HBASE-14882.master.001.patch
>
>
> In the Put API, we have addImmutable()
> {code}
> /**
>* See {@link #addColumn(byte[], byte[], byte[])}. This version expects
>* that the underlying arrays won't change. It's intended
>* for usage internal HBase to and for advanced client applications.
>*/
> public Put addImmutable(byte [] family, byte [] qualifier, byte [] value)
> {code}
> But in the implementation, the family, qualifier and value are still being
> copied locally to create kv.
> Hopefully we should provide an API that truly uses immutable family,
> qualifier and value.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16460) Can't rebuild the BucketAllocator's data structures when BucketCache use FileIOEngine
[
https://issues.apache.org/jira/browse/HBASE-16460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang updated HBASE-16460:
---
Attachment: 16460.v6.patch
Retry.
> Can't rebuild the BucketAllocator's data structures when BucketCache use
> FileIOEngine
> -
>
> Key: HBASE-16460
> URL: https://issues.apache.org/jira/browse/HBASE-16460
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3, 0.98.22
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
> Attachments: 16460.v6.patch, 16460.v6.patch, HBASE-16460-v1.patch,
> HBASE-16460-v2.patch, HBASE-16460-v2.patch, HBASE-16460-v3.patch,
> HBASE-16460-v4.patch, HBASE-16460-v5.patch, HBASE-16460-v5.patch,
> HBASE-16460.patch
>
>
> When bucket cache use FileIOEngine, it will rebuild the bucket allocator's
> data structures from a persisted map. So it should first read the map from
> persistence file then use the map to new a BucketAllocator. But now the code
> has wrong sequence in retrieveFromFile() method of BucketCache.java.
> {code}
> BucketAllocator allocator = new BucketAllocator(cacheCapacity,
> bucketSizes, backingMap, realCacheSize);
> backingMap = (ConcurrentHashMap)
> ois.readObject();
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[
https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463714#comment-15463714
]
Hadoop QA commented on HBASE-16556:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 53s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
16m 12s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 51s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 83m 21s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 121m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.mapred.TestMultiTableSnapshotInputFormat |
| Timed out junit tests |
[jira] [Commented] (HBASE-16560) Refactors some code around Procedure WALs
[ https://issues.apache.org/jira/browse/HBASE-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463650#comment-15463650 ] Matteo Bertozzi commented on HBASE-16560: - -1 in general since there is no motivation for this patch. +1 for the rename of trackProcIds() with updateMinMaxProcId() +1 for the rename of waitCond/slotCond -1 for removing the loader and move the corrupted wal into ProcedureWALFormatReader. the wal code is not finished yet and we have not handled yet the corrupted wals. the loader will be useful. -0 on the rename of the rollWriter. there is no additional value > Refactors some code around Procedure WALs > - > > Key: HBASE-16560 > URL: https://issues.apache.org/jira/browse/HBASE-16560 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16560.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16560) Refactors some code around Procedure WALs
[ https://issues.apache.org/jira/browse/HBASE-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16560: - Attachment: HBASE-16560.master.001.patch > Refactors some code around Procedure WALs > - > > Key: HBASE-16560 > URL: https://issues.apache.org/jira/browse/HBASE-16560 > Project: HBase > Issue Type: Improvement >Reporter: Appy >Assignee: Appy >Priority: Minor > Attachments: HBASE-16560.master.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16560) Refactors some code around Procedure WALs
Appy created HBASE-16560: Summary: Refactors some code around Procedure WALs Key: HBASE-16560 URL: https://issues.apache.org/jira/browse/HBASE-16560 Project: HBase Issue Type: Improvement Reporter: Appy Assignee: Appy Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reopened HBASE-16556: > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.branch-1.v0.patch, HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16556: --- Status: Patch Available (was: Reopened) > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.branch-1.v0.patch, HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ChiaPing Tsai updated HBASE-16556: -- Attachment: HBASE-16556.branch-1.v0.patch > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.branch-1.v0.patch, HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-16179: -- Assignee: Ted Yu > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16179: --- Attachment: 16179.v1.txt > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: 16179.v0.txt, 16179.v1.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
[ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463445#comment-15463445 ] Ted Yu commented on HBASE-16179: Made some progress: http://pastebin.com/zg3nSUmL > Fix compilation errors when building hbase-spark against Spark 2.0 > -- > > Key: HBASE-16179 > URL: https://issues.apache.org/jira/browse/HBASE-16179 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Attachments: 16179.v0.txt > > > I tried building hbase-spark module against Spark-2.0 snapshot and got the > following compilation errors: > http://pastebin.com/bg3w247a > Some Spark classes such as DataTypeParser and Logging are no longer > accessible to downstream projects. > hbase-spark module should not depend on such classes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16541) Avoid unnecessary cell copy in Result#compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463233#comment-15463233
]
Hudson commented on HBASE-16541:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1540 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1540/])
HBASE-16541 Avoid unnecessary cell copy in Result#compareResults (tedyu: rev
ab07f0087b8c5efcb1b084e42f4941ddb0c8885e)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Avoid unnecessary cell copy in Result#compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16541.v1.patch, HBASE-16541.v1.patch,
> HBASE-16541.v2.patch
>
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463162#comment-15463162 ] Andrew Purtell commented on HBASE-16556: +1 15866 was also committed to branch-1 so this patch should be committed there as well > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16559) Create script to automate RC testing
Dima Spivak created HBASE-16559:
---
Summary: Create script to automate RC testing
Key: HBASE-16559
URL: https://issues.apache.org/jira/browse/HBASE-16559
Project: HBase
Issue Type: Task
Components: scripts
Reporter: Dima Spivak
Assignee: Dima Spivak
Testing RCs would be easier and more repeatable if we had a script under
{{dev-support}} that declared certain typical operations (checking hashes,
running unit tests, spinning up clusters with {{clusterdock}} and running tests
from {{hbase-it}}), and allowed a user to run them on demand. Letting them
specify a JSON file with actions to take (and which arguments to pass to
various tests, for example) would be cool, too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16541) Avoid unnecessary cell copy in Result#compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15463130#comment-15463130
]
Hudson commented on HBASE-16541:
FAILURE: Integrated in Jenkins build HBase-1.4 #393 (See
[https://builds.apache.org/job/HBase-1.4/393/])
HBASE-16541 Avoid unnecessary cell copy in Result#compareResults (tedyu: rev
a0e52a2dabbd56a5126a1b981a5749bfeb89980e)
* (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/Result.java
> Avoid unnecessary cell copy in Result#compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16541.v1.patch, HBASE-16541.v1.patch,
> HBASE-16541.v2.patch
>
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16541) Avoid unnecessary cell copy in Result#compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16541:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 1.4.0
2.0.0
Status: Resolved (was: Patch Available)
Thanks for the patch, Chiaping.
Thanks for the review, Anoop.
> Avoid unnecessary cell copy in Result#compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16541.v1.patch, HBASE-16541.v1.patch,
> HBASE-16541.v2.patch
>
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16541) Avoid unnecessary cell copy in Result#compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16541:
---
Summary: Avoid unnecessary cell copy in Result#compareResults (was: Avoid
unnecessary cell copy in Result.compareResults)
> Avoid unnecessary cell copy in Result#compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Attachments: HBASE-16541.v1.patch, HBASE-16541.v1.patch,
> HBASE-16541.v2.patch
>
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16556) The read/write timeout are not used in HTable.delete(List), HTable.get(List), and HTable.existsAll(List)
[ https://issues.apache.org/jira/browse/HBASE-16556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16556: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the patch, Chiaping. > The read/write timeout are not used in HTable.delete(List), HTable.get(List), > and HTable.existsAll(List) > > > Key: HBASE-16556 > URL: https://issues.apache.org/jira/browse/HBASE-16556 > Project: HBase > Issue Type: Bug >Reporter: ChiaPing Tsai >Assignee: ChiaPing Tsai >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16556.v0.patch > > > [HBASE-15866|https://issues.apache.org/jira/browse/HBASE-15866] splits rpc > timeout into read timeout and write timeout. > The HTable.delete(List), HTable.get(List), and HTable.existsAll(List) call > AP#batch() to submit data, but they don’t use the read/write timeout > argument. So the AP will use the rpc timeout got from hbase.rpc.timeout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16229) Cleaning up size and heapSize calculation
[
https://issues.apache.org/jira/browse/HBASE-16229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462975#comment-15462975
]
Anastasia Braginsky commented on HBASE-16229:
-
Hey, you have done a great job with this fix!!!
I have done *only part* of my review so far. But part of my comments are
published on RB.
Will continue tomorrow! Thanks!
> Cleaning up size and heapSize calculation
> -
>
> Key: HBASE-16229
> URL: https://issues.apache.org/jira/browse/HBASE-16229
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0
>
> Attachments: HBASE-16229.patch, HBASE-16229_V2.patch,
> HBASE-16229_V3.patch, HBASE-16229_V4.patch, HBASE-16229_V5.patch,
> HBASE-16229_V5.patch
>
>
> It is bit ugly now. For eg:
> AbstractMemStore
> {code}
> public final static long FIXED_OVERHEAD = ClassSize.align(
> ClassSize.OBJECT +
> (4 * ClassSize.REFERENCE) +
> (2 * Bytes.SIZEOF_LONG));
> public final static long DEEP_OVERHEAD = ClassSize.align(FIXED_OVERHEAD +
> (ClassSize.ATOMIC_LONG + ClassSize.TIMERANGE_TRACKER +
> ClassSize.CELL_SKIPLIST_SET + ClassSize.CONCURRENT_SKIPLISTMAP));
> {code}
> We include the heap overhead of Segment also here. It will be better the
> Segment contains its overhead part and the Memstore impl uses the heap size
> of all of its segments to calculate its size.
> Also this
> {code}
> public long heapSize() {
> return getActive().getSize();
> }
> {code}
> HeapSize to consider all segment's size not just active's. I am not able to
> see an override method in CompactingMemstore.
> This jira tries to solve some of these.
> When we create a Segment, we seems pass some initial heap size value to it.
> Why? The segment object internally has to know what is its heap size not
> like some one else dictate it.
> More to add when doing this cleanup
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16445) Refactor and reimplement RpcClient
[
https://issues.apache.org/jira/browse/HBASE-16445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462883#comment-15462883
]
Duo Zhang commented on HBASE-16445:
---
The {{TestRegionReplicaFailover}} can not be fixed simply. Let me dig more.
> Refactor and reimplement RpcClient
> --
>
> Key: HBASE-16445
> URL: https://issues.apache.org/jira/browse/HBASE-16445
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-16445-v1.patch, HBASE-16445-v2.patch,
> HBASE-16445.patch
>
>
> There are lots of common logics between RpcClientImpl and AsyncRpcClient. We
> should have much less code comparing to the current implementations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16541) Avoid unnecessary cell copy in Result.compareResults
[
https://issues.apache.org/jira/browse/HBASE-16541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15462500#comment-15462500
]
Anoop Sam John commented on HBASE-16541:
+1
[~tedyu] U will commit this?
> Avoid unnecessary cell copy in Result.compareResults
>
>
> Key: HBASE-16541
> URL: https://issues.apache.org/jira/browse/HBASE-16541
> Project: HBase
> Issue Type: Improvement
>Reporter: ChiaPing Tsai
>Assignee: ChiaPing Tsai
> Attachments: HBASE-16541.v1.patch, HBASE-16541.v1.patch,
> HBASE-16541.v2.patch
>
>
> {code:title=Bar.java|borderStyle=solid}
> // Bytes.equals(a, b) should be replaced by Bytes.equals(a, off, len, b, off
> len);
> public static void compareResults(Result res1, Result res2)
> throws Exception {
> ...
> Cell[] ourKVs = res1.rawCells();
> Cell[] replicatedKVs = res2.rawCells();
> for (int i = 0; i < res1.size(); i++) {
> if (!ourKVs[i].equals(replicatedKVs[i]) ||
> !Bytes.equals(CellUtil.cloneValue(ourKVs[i]),
> CellUtil.cloneValue(replicatedKVs[i]))) {
> throw new Exception("This result was different: "
> + res1.toString() + " compared to " + res2.toString());
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
