[jira] [Updated] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24640: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the reviews [~reidchan] and [~ndimiduk] > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17239013#comment-17239013 ] Andrew Kyle Purtell commented on HBASE-24640: - VisibleForTesting binds us to Guava for no actual benefit. What is it used for? Does an IDE use it? Or some static analysis tool? Not as far as I can see. It would be a good follow up to make a maven enforcer rule that prevents adding it back. > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25308) [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous
[ https://issues.apache.org/jira/browse/HBASE-25308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242012#comment-17242012 ] Andrew Kyle Purtell commented on HBASE-25308: - Can you clarify? What needs to be polished? > [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous > - > > Key: HBASE-25308 > URL: https://issues.apache.org/jira/browse/HBASE-25308 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > > We are again having classpath versioning issues related to Guava in our > branch-1 based application. > Hadoop 3, HBase 2, Phoenix 5, and other projects deal with Guava > cross-version incompatibilities, as they manifest on a combined classpath > with other components, via shading. > I propose to do a global search and replace of all direct uses of Guava in > our branch-1 code base and refer to Guava as provided in hbase-thirdparty's > hbase-shaded-miscellaneous. This will protect HBase branch-1 from Guava > cross-version vagaries just like the same technique protects branch-2 and > branch-2 based releases. > There are a couple of Public or LimitedPrivate interfaces that incorporate > Guava's HostAndPort and Service that will be indirectly impacted. We are > about to release a new minor branch-1 version, 1.7.0, and this would be a > great opportunity to introduce this kind of change in a manner consistent > with semantic versioning and our compatibility policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25308) [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous
[ https://issues.apache.org/jira/browse/HBASE-25308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242015#comment-17242015 ] Andrew Kyle Purtell commented on HBASE-25308: - All the patch does is change imports to use hbase-third party, rearranges import order to conform to our checkstyle rules, and makes fixups for guava changes. This is already polish. I struggle to understand what more polish needs polishing :-) > [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous > - > > Key: HBASE-25308 > URL: https://issues.apache.org/jira/browse/HBASE-25308 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > > We are again having classpath versioning issues related to Guava in our > branch-1 based application. > Hadoop 3, HBase 2, Phoenix 5, and other projects deal with Guava > cross-version incompatibilities, as they manifest on a combined classpath > with other components, via shading. > I propose to do a global search and replace of all direct uses of Guava in > our branch-1 code base and refer to Guava as provided in hbase-thirdparty's > hbase-shaded-miscellaneous. This will protect HBase branch-1 from Guava > cross-version vagaries just like the same technique protects branch-2 and > branch-2 based releases. > There are a couple of Public or LimitedPrivate interfaces that incorporate > Guava's HostAndPort and Service that will be indirectly impacted. We are > about to release a new minor branch-1 version, 1.7.0, and this would be a > great opportunity to introduce this kind of change in a manner consistent > with semantic versioning and our compatibility policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25354) Update create-release scripts; rotted
[ https://issues.apache.org/jira/browse/HBASE-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244156#comment-17244156 ] Andrew Kyle Purtell commented on HBASE-25354: - Thanks [~stack], looking forward to trying them out. > Update create-release scripts; rotted > - > > Key: HBASE-25354 > URL: https://issues.apache.org/jira/browse/HBASE-25354 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A few things noticed when trying to run them just now: > * Some of the packages referenced are no longer available (the jdk and the > gpg). > * Some stuff that was checked in to test gpg works only makes sense in > non-docker context -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25359) create-release scripts releasedocmaker step should be optional
[ https://issues.apache.org/jira/browse/HBASE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244238#comment-17244238 ] Andrew Kyle Purtell edited comment on HBASE-25359 at 12/4/20, 7:04 PM: --- Just to be clear it's not about trust. As RM, I trust the script. However, I have ... requirements? idiosyncrasies? idiot-syncracies? :-) One of them is manual update of changes and retagging when incrementing the RC step. Skipping the 'tag' step totally will meet the objective of this issue but I thought it worth suggesting breaking it up to be a bit more fine-grained. This part changes upstream commit history and the RM may want more fine-grained control over what is happening. Since we are discussing this, although out of scope for this issue, another thing I prefer is parallel upload of signed RC artifacts to SVN and the whole Maven universe including super fat shaded artifacts to the Apache Nexus staging repository. The way we used to do this in make_rc.sh the deploy build stage is decoupled from the assembly stage. Assembly artifacts are available before deploy begins. They _should_ be decoupled. In my experience the upload to Nexus is bottlenecked somewhere on the Apache side. This is the main reason the build script takes so long to run. I observed several uploads of artifacts in excess of 200MB. The total amount of network transfer is on the order of a gigabyte. The Maven deploy could well fail -- it did for me for -- and may need to be retried. Longer term, those shaded modules with multi-hundred-megabyte artifacts should be a separate project spun post release of the base release. was (Author: apurtell): Just to be clear it's not about trust. As RM, I trust the script. However, I have ... requirements? idiosyncrasies? idiot-syncracies? :-) One of them is manual update of changes and retagging when incrementing the RC step. Skipping the 'tag' step totally will meet the objective of this issue but I thought it worth suggesting breaking it up to be a bit more fine-grained. This part changes upstream commit history and the RM may want more fine-grained control over what is happening. Since we are discussing this, although out of scope for this issue, another thing I prefer is parallel upload of signed RC artifacts to SVN and the whole Maven universe including super fat shaded artifacts to the Apache Nexus staging repository. The way we used to do this in make_rc.sh the deploy build stage is decoupled from the assembly stage. Assembly artifacts are available before deploy begins. They _should_ be decoupled. In my experience the upload to Nexus is bottlenecked somewhere on the Apache side. This is the main reason the build script takes so long to run. I observed several uploads of artifacts in excess of 200MB. The total amount of network transfer is on the order of a gigabyte. > create-release scripts releasedocmaker step should be optional > -- > > Key: HBASE-25359 > URL: https://issues.apache.org/jira/browse/HBASE-25359 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > The create-release scripts assume, when invoking releasedocmaker and > performing surgery on CHANGES.md and RELEASENOTES.md during the 'tag' stage, > that the current RC step is RC0. The entirety of the generated CHANGES.md and > RELEASENOTES.md files are stitched in at the head, just below the ASF notice. > If we are at a RC step that is not zero, wouldn't this duplicate all > CHANGES.md and RELEASENOTES.md content for the release? There would be all > the content added for RC0, then the same content (with delta) added for RC1, > and so on. > For this reason the releasedocmaker invocation should itself be optional. > For RC steps > 0, assume the RM has updated CHANGES.md and RELEASENOTES.md to > reflect the delta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25359) create-release scripts releasedocmaker step should be optional
[ https://issues.apache.org/jira/browse/HBASE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244238#comment-17244238 ] Andrew Kyle Purtell commented on HBASE-25359: - Just to be clear it's not about trust. As RM, I trust the script. However, I have ... requirements? idiosyncrasies? idiot-syncracies? :-) One of them is manual update of changes and retagging when incrementing the RC step. Skipping the 'tag' step totally will meet the objective of this issue but I thought it worth suggesting breaking it up to be a bit more fine-grained. This part changes upstream commit history and the RM may want more fine-grained control over what is happening. Since we are discussing this, although out of scope for this issue, another thing I prefer is parallel upload of signed RC artifacts to SVN and the whole Maven universe including super fat shaded artifacts to the Apache Nexus staging repository. The way we used to do this in make_rc.sh the deploy build stage is decoupled from the assembly stage. Assembly artifacts are available before deploy begins. They _should_ be decoupled. In my experience the upload to Nexus is bottlenecked somewhere on the Apache side. This is the main reason the build script takes so long to run. I observed several uploads of artifacts in excess of 200MB. The total amount of network transfer is on the order of a gigabyte. > create-release scripts releasedocmaker step should be optional > -- > > Key: HBASE-25359 > URL: https://issues.apache.org/jira/browse/HBASE-25359 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > The create-release scripts assume, when invoking releasedocmaker and > performing surgery on CHANGES.md and RELEASENOTES.md during the 'tag' stage, > that the current RC step is RC0. The entirety of the generated CHANGES.md and > RELEASENOTES.md files are stitched in at the head, just below the ASF notice. > If we are at a RC step that is not zero, wouldn't this duplicate all > CHANGES.md and RELEASENOTES.md content for the release? There would be all > the content added for RC0, then the same content (with delta) added for RC1, > and so on. > For this reason the releasedocmaker invocation should itself be optional. > For RC steps > 0, assume the RM has updated CHANGES.md and RELEASENOTES.md to > reflect the delta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25292) Improve InetSocketAddress usage discipline
[ https://issues.apache.org/jira/browse/HBASE-25292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25292: Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > Improve InetSocketAddress usage discipline > -- > > Key: HBASE-25292 > URL: https://issues.apache.org/jira/browse/HBASE-25292 > Project: HBase > Issue Type: Bug > Components: Client, HFile >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > > We sometimes cache InetSocketAddress in data structures in an attempt to > optimize away potential nameservice (DNS) lookups. This is, in general, an > anti-pattern, because once an InetSocketAddress is resolved, resolution is > never attempted again. The ideal pattern for connect() is ISA instantiation > just before the connect() call, with no reuse of the ISA instance. For bind() > we presume the local identity won't change while the process is live so usage > and caching can be relaxed in that case. > If I can restate my proposal for a usage convention for InetSocketAddress, it > would be this: Network identities should be bound late. This means addresses > should be resolved at the last possible moment. Also, network identity > mappings can change, so our code should not inappropriately cache them; > otherwise we might miss a change and fail to operate normally. > I have reviewed the code for InetSocketAddress usage and in my opinion > sometimes we are caching ISA acceptably, and in other cases we are not. > Correct cases: > * We cache ISA for RPC connections, so we don't potentially do a lookup for > every Call. However, we resolve the address earlier than we need to. The code > can be improved by moving resolution to just before where we connect(). > * Use of ISA with bind. Typical uses like bindAddress, listenerAddress, > initialIsa, or localAddress. > ** (There is no harm to keep direct use of ISA for bind() but these could > all be replaced with Address and on-demand create of ISA just before bind(). > * ClusterStatusPublisher in master. > * Netty integration. Netty accepts and supplies ISA, no choice there, but we > want to resolve at channel create time and cache for lifetime of the channel > anyway. If a remote host goes away and is replaced with a new identity, a new > channel/connection will be created with a new resolution just before > connect() via higher layer error handling. > Incorrect cases that can be fixed: > * RPC stubs. Remote clients may be recycled and replaced with new instances > where the network identities (DNS name to IP address mapping) have changed--. > HBASE-14544 attempts to work around DNS instability in data centers of years > past in a way that, in my opinion, is the wrong thing to do in the modern > era. In modern datacenters, in public cloud, and especially in kubernetes > environments, DNS mappings are dynamic and subject to frequent change. It is > just never the right thing to do to cache them. I intend to propose a revert > of HBASE-14544. Reverting this simplifies some code a bit. That is the only > reason: this is in my opinion some legacy that can be dropped, one fewer > configuration variable (yay!), but if this part of the proposal is > controversial it can be skipped. > * RPC stubs again. When looking up the IP address of the remote host when > creating a stub key we also make a key even if the resolution fails. This is > the wrong thing to do. If we can't resolve the remote address, we can't > contact the server. Making a stub that can't communicate is pointless. Throw > an exception instead. > * Favored nodes. Although the HDFS API requires InetSocketAddress, we don't > have to make up a list right away and cache them forever. We can use Address > to record the list of favored nodes and convert from Address to > InetSocketAddress on demand (when we go to create the HFile). This will allow > us to resolve datanode hostnames just before they are needed. In public > cloud, kubernetes, and or some private datacenter service deployment options, > datanode servers may have their network identities (DNS name -> IP address > mapping) changed over time. We can and should avoid inappropriate caching > that may cause us to indefinitely use an incorrect address when contacting a > favored node. > * Sometimes we use ISA when Address is just as good. For example, the dead > servers list. If we are going to pay some attention to ISA usage discipline, > let's remove the cases where we use ISA as a host and port pair but do not > need to do so. Address works just as well and doesn't present an opportunity > for misuse. Another example would be the
[jira] [Created] (HBASE-25359) create-release scripts releasedocmaker step should be optional
Andrew Kyle Purtell created HBASE-25359: --- Summary: create-release scripts releasedocmaker step should be optional Key: HBASE-25359 URL: https://issues.apache.org/jira/browse/HBASE-25359 Project: HBase Issue Type: Improvement Reporter: Andrew Kyle Purtell The create-release scripts assume, when invoking releasedocmaker and performing surgery on CHANGES.md and RELEASENOTES.md during the 'tag' stage, that the current RC step is RC0. The entirety of the generated CHANGES.md and RELEASENOTES.md files are stitched in at the head, just below the ASF notice. If we are at a RC step that is not zero, wouldn't this duplicate all CHANGES.md and RELEASENOTES.md content for the release? There would be all the content added for RC0, then the same content (with delta) added for RC1, and so on. For this reason the releasedocmaker invocation should itself be optional. For RC steps > 0, assume the RM has updated CHANGES.md and RELEASENOTES.md to reflect the delta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25354) Update create-release scripts; rotted
[ https://issues.apache.org/jira/browse/HBASE-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244190#comment-17244190 ] Andrew Kyle Purtell commented on HBASE-25354: - It's not, but it's certainly not urgent or anything. If a RM wants to opt out of releasedockmaker surgery they can just skip the whole 'tag' step and do it manually. I did that for 2.4.0RC1. > Update create-release scripts; rotted > - > > Key: HBASE-25354 > URL: https://issues.apache.org/jira/browse/HBASE-25354 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A few things noticed when trying to run them just now: > * Some of the packages referenced are no longer available (the jdk and the > gpg). > * Some stuff that was checked in to test gpg works only makes sense in > non-docker context -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25354) Update create-release scripts; rotted
[ https://issues.apache.org/jira/browse/HBASE-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244190#comment-17244190 ] Andrew Kyle Purtell edited comment on HBASE-25354 at 12/4/20, 6:14 PM: --- It's not, but it's certainly not urgent or anything. If a RM wants to opt out of releasedockmaker surgery they can just skip the whole 'tag' step and do it manually. I did that for 2.4.0RC1. Edit: But I'm an odd one, and you don't need to cater to me. was (Author: apurtell): It's not, but it's certainly not urgent or anything. If a RM wants to opt out of releasedockmaker surgery they can just skip the whole 'tag' step and do it manually. I did that for 2.4.0RC1. > Update create-release scripts; rotted > - > > Key: HBASE-25354 > URL: https://issues.apache.org/jira/browse/HBASE-25354 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A few things noticed when trying to run them just now: > * Some of the packages referenced are no longer available (the jdk and the > gpg). > * Some stuff that was checked in to test gpg works only makes sense in > non-docker context -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25354) Update create-release scripts; rotted
[ https://issues.apache.org/jira/browse/HBASE-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244161#comment-17244161 ] Andrew Kyle Purtell commented on HBASE-25354: - Oh, but I made an improvement request: HBASE-25359 > Update create-release scripts; rotted > - > > Key: HBASE-25354 > URL: https://issues.apache.org/jira/browse/HBASE-25354 > Project: HBase > Issue Type: Bug >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A few things noticed when trying to run them just now: > * Some of the packages referenced are no longer available (the jdk and the > gpg). > * Some stuff that was checked in to test gpg works only makes sense in > non-docker context -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25359) create-release scripts releasedocmaker step should be optional
[ https://issues.apache.org/jira/browse/HBASE-25359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17244188#comment-17244188 ] Andrew Kyle Purtell commented on HBASE-25359: - I see, that helps, but what if the RM manually updated the files and still wants to opt out? > create-release scripts releasedocmaker step should be optional > -- > > Key: HBASE-25359 > URL: https://issues.apache.org/jira/browse/HBASE-25359 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Priority: Major > > The create-release scripts assume, when invoking releasedocmaker and > performing surgery on CHANGES.md and RELEASENOTES.md during the 'tag' stage, > that the current RC step is RC0. The entirety of the generated CHANGES.md and > RELEASENOTES.md files are stitched in at the head, just below the ASF notice. > If we are at a RC step that is not zero, wouldn't this duplicate all > CHANGES.md and RELEASENOTES.md content for the release? There would be all > the content added for RC0, then the same content (with delta) added for RC1, > and so on. > For this reason the releasedocmaker invocation should itself be optional. > For RC steps > 0, assume the RM has updated CHANGES.md and RELEASENOTES.md to > reflect the delta. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25246) Backup/Restore hbase cell tags.
[ https://issues.apache.org/jira/browse/HBASE-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25246: Fix Version/s: (was: 2.4.1) (was: 2.3.4) (was: 2.2.7) > Backup/Restore hbase cell tags. > --- > > Key: HBASE-25246 > URL: https://issues.apache.org/jira/browse/HBASE-25246 > Project: HBase > Issue Type: Improvement > Components: backuprestore >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > In PHOENIX-6213 we are planning to add cell tags for Delete mutations. After > having a discussion with hbase community via dev mailing thread, it was > decided that we will pass the tags via an attribute in Mutation object and > persist them to hbase via phoenix co-processor. The intention of PHOENIX-6213 > is to store metadata in Delete marker so that while running Restore tool we > can selectively restore certain Delete markers and ignore others. For that to > happen we need to persist these tags in Backup and retrieve them in Restore > MR jobs (Import/Export tool). > Currently we don't persist the tags in Backup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25246) Backup/Restore hbase cell tags.
[ https://issues.apache.org/jira/browse/HBASE-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243390#comment-17243390 ] Andrew Kyle Purtell commented on HBASE-25246: - This may need to wait until 2.5. Is it in-line with our compatibility guidelines to introduce a feature on a patch release? Is this something that rises to the level of a new feature? Not sure. Anyway I moved the fix version to 2.5.0 for now. > Backup/Restore hbase cell tags. > --- > > Key: HBASE-25246 > URL: https://issues.apache.org/jira/browse/HBASE-25246 > Project: HBase > Issue Type: Improvement > Components: backuprestore >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0, 2.4.1 > > > In PHOENIX-6213 we are planning to add cell tags for Delete mutations. After > having a discussion with hbase community via dev mailing thread, it was > decided that we will pass the tags via an attribute in Mutation object and > persist them to hbase via phoenix co-processor. The intention of PHOENIX-6213 > is to store metadata in Delete marker so that while running Restore tool we > can selectively restore certain Delete markers and ignore others. For that to > happen we need to persist these tags in Backup and retrieve them in Restore > MR jobs (Import/Export tool). > Currently we don't persist the tags in Backup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25246) Backup/Restore hbase cell tags.
[ https://issues.apache.org/jira/browse/HBASE-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243414#comment-17243414 ] Andrew Kyle Purtell commented on HBASE-25246: - [~shahrs87] No need to apologize! If it's not 100% yet it's important to revert and improve. > Backup/Restore hbase cell tags. > --- > > Key: HBASE-25246 > URL: https://issues.apache.org/jira/browse/HBASE-25246 > Project: HBase > Issue Type: Improvement > Components: backuprestore >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > In PHOENIX-6213 we are planning to add cell tags for Delete mutations. After > having a discussion with hbase community via dev mailing thread, it was > decided that we will pass the tags via an attribute in Mutation object and > persist them to hbase via phoenix co-processor. The intention of PHOENIX-6213 > is to store metadata in Delete marker so that while running Restore tool we > can selectively restore certain Delete markers and ignore others. For that to > happen we need to persist these tags in Backup and retrieve them in Restore > MR jobs (Import/Export tool). > Currently we don't persist the tags in Backup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25230) Embedded zookeeper server not clean up the old data
[
https://issues.apache.org/jira/browse/HBASE-25230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25230:
Fix Version/s: (was: 2.4.1)
2.4.0
> Embedded zookeeper server not clean up the old data
> ---
>
> Key: HBASE-25230
> URL: https://issues.apache.org/jira/browse/HBASE-25230
> Project: HBase
> Issue Type: Bug
> Components: Zookeeper
>Reporter: Rural Hunter
>Assignee: Laxman Goswami
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0, 2.2.7, 2.3.4, 2.5.0
>
>
> The size of zookeeper data dir keeps increasing. I checked the source code
> and found the zookeeper data cleanup manager is not started by hbase. The
> standalone zookeeper server runs the data cleanup manager but hbase does not.
> I suggest add the code below at the beginning of
> org.apache.hadoop.hbase.zookeeper.HQuorumPeer.runZKServer():
> {code:java}
> // Start and schedule the the purge task
> DatadirCleanupManager purgeMgr=new DatadirCleanupManager(
> zkConfig.getDataDir(),
> zkConfig.getDataLogDir(),
> zkConfig.getSnapRetainCount(),
> zkConfig.getPurgeInterval());
> purgeMgr.start();
> {code}
> I've been running this code in my hbase cluster for quite a long time without
> problem.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25246) Backup/Restore hbase cell tags.
[ https://issues.apache.org/jira/browse/HBASE-25246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243392#comment-17243392 ] Andrew Kyle Purtell commented on HBASE-25246: - The way the RC process works we already have commits and tags for RC0, so I'm just going to move on and start a RC1. Thanks for the heads up. > Backup/Restore hbase cell tags. > --- > > Key: HBASE-25246 > URL: https://issues.apache.org/jira/browse/HBASE-25246 > Project: HBase > Issue Type: Improvement > Components: backuprestore >Reporter: Rushabh Shah >Assignee: Rushabh Shah >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > In PHOENIX-6213 we are planning to add cell tags for Delete mutations. After > having a discussion with hbase community via dev mailing thread, it was > decided that we will pass the tags via an attribute in Mutation object and > persist them to hbase via phoenix co-processor. The intention of PHOENIX-6213 > is to store metadata in Delete marker so that while running Restore tool we > can selectively restore certain Delete markers and ignore others. For that to > happen we need to persist these tags in Backup and retrieve them in Restore > MR jobs (Import/Export tool). > Currently we don't persist the tags in Backup. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25187) Improve SizeCachedKV variants initialization
[ https://issues.apache.org/jira/browse/HBASE-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25187. - Hadoop Flags: Reviewed Resolution: Fixed > Improve SizeCachedKV variants initialization > > > Key: HBASE-25187 > URL: https://issues.apache.org/jira/browse/HBASE-25187 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0 > > > Currently in SizeCachedKV we get the rowlength and Key length from the > buffers. This can be optimized because we can pass the keylen and row len > while actually creating the cell while reading the cell from the block. Some > times we see that the SizeCachedKV takes the max width in a flame graph - > considering the fact we also do a sanity check on the created KV. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Priority: Critical (was: Major) > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25127) Enhance PerformanceEvaluation to profile meta replica performance.
[ https://issues.apache.org/jira/browse/HBASE-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238254#comment-17238254 ] Andrew Kyle Purtell commented on HBASE-25127: - Thanks [~huaxiangsun] , please resolve this issue after merge. > Enhance PerformanceEvaluation to profile meta replica performance. > -- > > Key: HBASE-25127 > URL: https://issues.apache.org/jira/browse/HBASE-25127 > Project: HBase > Issue Type: Sub-task >Reporter: Huaxiang Sun >Assignee: Clara Xiong >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: Screen Shot 2020-11-13 at 5.30.11 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238252#comment-17238252 ] Andrew Kyle Purtell commented on HBASE-25242: - Thanks for letting me know. Agree, reverts would be best to avoid. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25050) We initialize Filesystems more than once.
[
https://issues.apache.org/jira/browse/HBASE-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell resolved HBASE-25050.
-
Hadoop Flags: Reviewed
Resolution: Fixed
> We initialize Filesystems more than once.
> -
>
> Key: HBASE-25050
> URL: https://issues.apache.org/jira/browse/HBASE-25050
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
> Fix For: 3.0.0-alpha-1, 2.4.0
>
>
> In HFileSystem
> {code}
> // Create the default filesystem with checksum verification switched on.
> // By default, any operation to this FilterFileSystem occurs on
> // the underlying filesystem that has checksums switched on.
> this.fs = FileSystem.get(conf);
> this.useHBaseChecksum = useHBaseChecksum;
> fs.initialize(getDefaultUri(conf), conf);
> {code}
> We call fs.initialize(). Generally the FS would have been created and inited
> either in the FileSystem.get() call above or even when we try to check
> {code}
> FileSystem fs = p.getFileSystem(c);
> {code}
> The FS that gets cached in the hadoop-common layer does the init for us. So
> we doing it again is redundant.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25320) Upgrade hbase-thirdparty dependency to 3.4.1
[ https://issues.apache.org/jira/browse/HBASE-25320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237617#comment-17237617 ] Andrew Kyle Purtell commented on HBASE-25320: - Blocks 2.4.0RC0 > Upgrade hbase-thirdparty dependency to 3.4.1 > > > Key: HBASE-25320 > URL: https://issues.apache.org/jira/browse/HBASE-25320 > Project: HBase > Issue Type: Task > Components: dependencies >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24509) Make PE to have both read and write ability to go in parallel as in YCSB
[ https://issues.apache.org/jira/browse/HBASE-24509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237629#comment-17237629 ] Andrew Kyle Purtell commented on HBASE-24509: - Moved out of 2.4. > Make PE to have both read and write ability to go in parallel as in YCSB > > > Key: HBASE-24509 > URL: https://issues.apache.org/jira/browse/HBASE-24509 > Project: HBase > Issue Type: Improvement > Components: Performance, test >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently we can either run pure writes or pure reads in case of PE. The case > where few threads can do write and few can do reads is not available. We can > improve the PE to atleast support writes and either Scans or Gets in > parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24509) Make PE to have both read and write ability to go in parallel as in YCSB
[ https://issues.apache.org/jira/browse/HBASE-24509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24509: Fix Version/s: (was: 2.4.0) 2.5.0 > Make PE to have both read and write ability to go in parallel as in YCSB > > > Key: HBASE-24509 > URL: https://issues.apache.org/jira/browse/HBASE-24509 > Project: HBase > Issue Type: Improvement > Components: Performance, test >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently we can either run pure writes or pure reads in case of PE. The case > where few threads can do write and few can do reads is not available. We can > improve the PE to atleast support writes and either Scans or Gets in > parallel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25034) Table regions details on master GUI display slowly.
[ https://issues.apache.org/jira/browse/HBASE-25034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25034: Fix Version/s: (was: 2.4.0) 2.5.0 > Table regions details on master GUI display slowly. > --- > > Key: HBASE-25034 > URL: https://issues.apache.org/jira/browse/HBASE-25034 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: DingSheng Li >Priority: Major > Labels: newbie > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0 > > Attachments: The table display after pagination.html > > > When a table has a large number of regions (e.g.,a single table contains more > than 100,000 regions), it takes about 20 to 30 minutes to display the table > regions on the master GUI, which is unacceptable to users. After testing, we > find that web page rendering takes up the most time,and this can be solved by > pagination query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25034) Table regions details on master GUI display slowly.
[ https://issues.apache.org/jira/browse/HBASE-25034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237638#comment-17237638 ] Andrew Kyle Purtell commented on HBASE-25034: - Moving out of 2.4. > Table regions details on master GUI display slowly. > --- > > Key: HBASE-25034 > URL: https://issues.apache.org/jira/browse/HBASE-25034 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.3.2 >Reporter: DingSheng Li >Priority: Major > Labels: newbie > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0 > > Attachments: The table display after pagination.html > > > When a table has a large number of regions (e.g.,a single table contains more > than 100,000 regions), it takes about 20 to 30 minutes to display the table > regions on the master GUI, which is unacceptable to users. After testing, we > find that web page rendering takes up the most time,and this can be solved by > pagination query. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25287) Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files
[
https://issues.apache.org/jira/browse/HBASE-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237656#comment-17237656
]
Andrew Kyle Purtell commented on HBASE-25287:
-
I approved the branch-2 PR but please do not merge it before a PR for master is
merged.
> Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when
> loading files
>
>
> Key: HBASE-25287
> URL: https://issues.apache.org/jira/browse/HBASE-25287
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: 1605328358304-image.png, 1605328417888-image.png,
> 1605504914256-image.png
>
>
> HBASE-9393 found seek+read will leave many CLOSE_WAIT sockets without stream
> unbuffer, which can free sockets and file descriptors held by the stream.
> In our cluster RSes with about one hundred thousand store files, we found the
> number of CLOSE_WAIT sockets increases with the number of regions opened,
> and can up to the operating system open files limit 100.
>
> {code:java}
> 2020-11-12 20:19:02,452 WARN [1282990092@qtp-220038608-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:16030] mortbay.log: EXCEPTION
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at
> org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
> at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686)
> at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}
>
> {code:java}
> [hbase@gha-data-hbase-cat0053 hbase]$ ulimit -SHn
> 100
> {code}
>
>
> The reason of the problem is, when store file opened,
> {code:java}
> private void open() throws IOException {
> fileInfo.initHDFSBlocksDistribution();
> long readahead = fileInfo.isNoReadahead() ? 0L : -1L;
> ReaderContext context = fileInfo.createReaderContext(false, readahead,
> ReaderType.PREAD);
> fileInfo.initHFileInfo(context);
> StoreFileReader reader = fileInfo.preStoreFileReaderOpen(context,
> cacheConf);
> if (reader == null) {
> reader = fileInfo.createReader(context, cacheConf);
> fileInfo.getHFileInfo().initMetaAndIndex(reader.getHFileReader());
> }
> {code}
> only createReader() unbuffered the stream. In initMetaAndIndex(), using the
> stream to read blocks, so it needs to unbuffer() the socket , too.
> We can just add try before fileInfo.initHFileInfo(context); and finally
> unbuffer() the stream at the end of the open() function.
> We fixed it on our cluster, the number of CLOSE_WAIT reduced to about 0.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237600#comment-17237600 ] Andrew Kyle Purtell edited comment on HBASE-25242 at 11/23/20, 6:52 PM: A security issue in a third party dependency has been raised as a concern. We need a new release of hbase-thirdparty, 3.4.1, to complete, before 2.4.0 RC0 can be built. At this point the RC is waiting for the new release of hbase-thirdparty to happen. As of the time of this comment there are 67 hours remaining in that vote. That gives you time here [~brfrn169] [~zhangduo] I have moved this out of 2.4.0 but it can be brought back in if the PR is merged in time. I would encourage the code review on PR #2630 to move with all deliberate haste if you want it to get in. was (Author: apurtell): A security issue in a third party dependency has been raised as a concern. We need a new release of hbase-thirdparty, 3.4.1, to complete, before 2.4.0 RC0 can be built. At this point the RC is waiting for the new release of hbase-thirdparty to happen. That gives you time here [~brfrn169] [~zhangduo] I have moved this out of 2.4.0 but it can be brought back in if the PR is merged in time. I would encourage the code review on PR #2630 to move with all deliberate haste if you want it to get in. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Fix Version/s: (was: 2.5.0) 2.4.0 > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Status: Patch Available (was: Open) > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25278) Add option to toggle CACHE_BLOCKS in count.rb
[ https://issues.apache.org/jira/browse/HBASE-25278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237618#comment-17237618 ] Andrew Kyle Purtell commented on HBASE-25278: - We have approximately 60 hours before the 2.4.0 RC0 is unblocked. If you want to get this in, please merge the PR in that timeframe. > Add option to toggle CACHE_BLOCKS in count.rb > - > > Key: HBASE-25278 > URL: https://issues.apache.org/jira/browse/HBASE-25278 > Project: HBase > Issue Type: New Feature > Components: shell >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.4.0 > > > A trick I've found myself doing a couple of times (hat-tip to [~psomogyi]) is > to edit table.rb so that the `count` shell command will not instruct > RegionServers to not cache any data blocks. This is a quick+dirty way to > force a table to be loaded into block cache (i.e. for performance testing). > We can easily add another option to avoid having to edit the ruby files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21831) Optional store-and-forward of simple mutations for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-21831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237624#comment-17237624 ] Andrew Kyle Purtell commented on HBASE-21831: - This should only be scheduled against trunk version until actually started. > Optional store-and-forward of simple mutations for regions in transition > > > Key: HBASE-21831 > URL: https://issues.apache.org/jira/browse/HBASE-21831 > Project: HBase > Issue Type: New Feature > Components: regionserver, rpc >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1 > > > We have an internal service built on Redis that is considering writing > through to HBase directly for their persistence needs. Their current > experience with Redis is > * Average write latency is ~milliseconds > * p999 write latencies with are "a few seconds" > They want a similar experience when writing simple values directly to HBase. > Infrequent exceptions to this would be acceptable. > * Availability of 99.9% for writes > * Expect most writes to be serviced within a few milliseconds, e.g. few > millis at p95. Still evaluating what the requirement should be (~millis at > p90 vs p95 vs p99). > * Timeout of 2 seconds, should be rare > There is a fallback plan considered if HBase cannot respond within 2 seconds. > However this fallback cannot guarantee durability. Redis or the service's > daemons may go down. They want HBase to provide required durability. > Because this is a caching service, where all writes are expected to be served > again from cache, at least for a while, if HBase were to accept writes such > that they are not immediately visible, it could be fine that they are not > visible for 10-20 minutes in the worst case. This is relatively easy to > achieve as an engineering target should we consider offering a write option > that does not guarantee immediate visibility. (A proposal follows below.) We > are considering store-and-forward of simple mutations and perhaps also simple > deletes, although the latter is not a hard requirement. Out of order > processing of this subset of mutation requests is acceptable because their > data model ensures all values are immutable. Presumably on the HBase side the > timestamps of the requests would be set to the current server wall clock time > when received, so eventually when applied all are available with correct > temporal ordering (within the effective resolution of the server clocks). > Deletes which are not immediately applied (or failed) could cause application > level confusion, and although this would remain a concern for the general > case, for this specific use case, stale reads could be explained to and > tolerated by their users. > The BigTable architecture assigns at most one server to serve a region at a > time. Region Replicas are an enhancement to the base BigTable architecture we > made in HBase which stands up two more read-only replicas for a given region, > meaning a client attempting a read has the option to fail very quickly over > from the primary to a replica for a (potentially stale) read, or distribute > read load over all replicas, or employ a hedged reading strategy. Enabling > region replicas and timeline consistency can lower the availability gap for > reads in the high percentiles from ~minutes to ~milliseconds. However, this > option will not help for write use cases wanting roughly the same thing, > because there can be no fail-over for writes. Writes must still go to the > active primary. When that region is in transition, writes must be held on the > client until it is redeployed. Or, if region replicas are not enabled, when > the sole region is in transition, again, writes must be held on the client > until the region is available again. > Regions enter the in-transition state for two reasons: failures, and > housekeeping (splits and merges, or balancing). Time to region redeployment > after failures depends on a number of factors, like how long it took for us > to become aware of the failure, and how long it takes to split the > write-ahead log of the failed server and distribute the recovered edits to > the reopening region(s). We could in theory improve this behavior by being > more predictive about declaring failure, like employing a phi accrual failure > detector to signal to the master from clients that a regionserver is sick. > Other time-to-recovery issues and mitigations are discussed in a number of > JIRAs and blog posts and not discussed further here. Regarding housekeeping > activities, splits and merges typically complete in under a second. However, > split times up to ~30 seconds have been observed at my place of employ in > rare conditions. In the
[jira] [Commented] (HBASE-20999) Move hbase-REST to new hbase-connectors repository
[ https://issues.apache.org/jira/browse/HBASE-20999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237625#comment-17237625 ] Andrew Kyle Purtell commented on HBASE-20999: - Unscheduling from 2.4. > Move hbase-REST to new hbase-connectors repository > -- > > Key: HBASE-20999 > URL: https://issues.apache.org/jira/browse/HBASE-20999 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors, REST >Reporter: Michael Stack >Assignee: zhuqi >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Move hbase-rest to the new hbase-connectors repository. See parent issue for > context and locale. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21831) Optional store-and-forward of simple mutations for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-21831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21831: Fix Version/s: (was: 2.4.0) (was: 1.7.0) > Optional store-and-forward of simple mutations for regions in transition > > > Key: HBASE-21831 > URL: https://issues.apache.org/jira/browse/HBASE-21831 > Project: HBase > Issue Type: New Feature > Components: regionserver, rpc >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1 > > > We have an internal service built on Redis that is considering writing > through to HBase directly for their persistence needs. Their current > experience with Redis is > * Average write latency is ~milliseconds > * p999 write latencies with are "a few seconds" > They want a similar experience when writing simple values directly to HBase. > Infrequent exceptions to this would be acceptable. > * Availability of 99.9% for writes > * Expect most writes to be serviced within a few milliseconds, e.g. few > millis at p95. Still evaluating what the requirement should be (~millis at > p90 vs p95 vs p99). > * Timeout of 2 seconds, should be rare > There is a fallback plan considered if HBase cannot respond within 2 seconds. > However this fallback cannot guarantee durability. Redis or the service's > daemons may go down. They want HBase to provide required durability. > Because this is a caching service, where all writes are expected to be served > again from cache, at least for a while, if HBase were to accept writes such > that they are not immediately visible, it could be fine that they are not > visible for 10-20 minutes in the worst case. This is relatively easy to > achieve as an engineering target should we consider offering a write option > that does not guarantee immediate visibility. (A proposal follows below.) We > are considering store-and-forward of simple mutations and perhaps also simple > deletes, although the latter is not a hard requirement. Out of order > processing of this subset of mutation requests is acceptable because their > data model ensures all values are immutable. Presumably on the HBase side the > timestamps of the requests would be set to the current server wall clock time > when received, so eventually when applied all are available with correct > temporal ordering (within the effective resolution of the server clocks). > Deletes which are not immediately applied (or failed) could cause application > level confusion, and although this would remain a concern for the general > case, for this specific use case, stale reads could be explained to and > tolerated by their users. > The BigTable architecture assigns at most one server to serve a region at a > time. Region Replicas are an enhancement to the base BigTable architecture we > made in HBase which stands up two more read-only replicas for a given region, > meaning a client attempting a read has the option to fail very quickly over > from the primary to a replica for a (potentially stale) read, or distribute > read load over all replicas, or employ a hedged reading strategy. Enabling > region replicas and timeline consistency can lower the availability gap for > reads in the high percentiles from ~minutes to ~milliseconds. However, this > option will not help for write use cases wanting roughly the same thing, > because there can be no fail-over for writes. Writes must still go to the > active primary. When that region is in transition, writes must be held on the > client until it is redeployed. Or, if region replicas are not enabled, when > the sole region is in transition, again, writes must be held on the client > until the region is available again. > Regions enter the in-transition state for two reasons: failures, and > housekeeping (splits and merges, or balancing). Time to region redeployment > after failures depends on a number of factors, like how long it took for us > to become aware of the failure, and how long it takes to split the > write-ahead log of the failed server and distribute the recovered edits to > the reopening region(s). We could in theory improve this behavior by being > more predictive about declaring failure, like employing a phi accrual failure > detector to signal to the master from clients that a regionserver is sick. > Other time-to-recovery issues and mitigations are discussed in a number of > JIRAs and blog posts and not discussed further here. Regarding housekeeping > activities, splits and merges typically complete in under a second. However, > split times up to ~30 seconds have been observed at my place of employ in > rare conditions. In the instances I have investigated the cause is I/O
[jira] [Updated] (HBASE-20999) Move hbase-REST to new hbase-connectors repository
[ https://issues.apache.org/jira/browse/HBASE-20999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20999: Fix Version/s: (was: 2.4.0) > Move hbase-REST to new hbase-connectors repository > -- > > Key: HBASE-20999 > URL: https://issues.apache.org/jira/browse/HBASE-20999 > Project: HBase > Issue Type: Sub-task > Components: hbase-connectors, REST >Reporter: Michael Stack >Assignee: zhuqi >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Move hbase-rest to the new hbase-connectors repository. See parent issue for > context and locale. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24664) Some changing of split region by overall region size rather than only one store size
[ https://issues.apache.org/jira/browse/HBASE-24664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-24664. - Hadoop Flags: Reviewed Resolution: Fixed > Some changing of split region by overall region size rather than only one > store size > > > Key: HBASE-24664 > URL: https://issues.apache.org/jira/browse/HBASE-24664 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > As a distributed cluster, HBase distribute loads in unit of region, so if > region grows too big, > it will bring some negative effects, such as: > 1. Harder to homogenize disk usage(consider locality) > 2. Might cost more time on region opening > 3. After split, the daughter region might lead to more io cost on compaction > in a short time(if write evenly) > I tried to introduce a new SteppingAllStoresSizeSplitPolicy in HBASE-24530, > but after discussed in comments and related > [thread|https://lists.apache.org/thread.html/r08a8103e2532eb667a0fcb4efa8a4117b3f82e6251bc4bd0bc157c26%40%3Cdev.hbase.apache.org%3E], > finally we decide to change the existing split policy with a new option that > if it should count all store files, and for master it would be true, else > false. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24679) HBase on Cloud Blob FS : Provide config to skip HFile archival while table deletion
[ https://issues.apache.org/jira/browse/HBASE-24679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24679: Fix Version/s: (was: 2.4.0) > HBase on Cloud Blob FS : Provide config to skip HFile archival while table > deletion > > > Key: HBASE-24679 > URL: https://issues.apache.org/jira/browse/HBASE-24679 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When we delete a table as part of delete of table from FS, we do below things > 1. Rename to table directory to come under /hbase/.tmp. This is an atomic > rename op > 2. Go through each of HFiles under every region:cf and archive that one by > one. (Rename the file from .tmp path to go to /hbase/archive) > 3. Delete the table dir under .tmp dir > In case of HDFS this is not a big deal as every rename op is just a meta op > (Though the HFiles archival is a costly only as there will be so many calls > to NN based the table's regions# and total storesfiles#) But on Cloud blob > based FS impl, this is a concerning op. Every rename will be a copy blob op. > And we are doing it twice per each of the HFiles in this table ! > The proposal here is to provide a config option (default to false) to skip > this archival step. > We can provide another config to even avoid the .tmp rename? The atomicity of > the Table delete can be achieved by HM side procedure and proc WAL. In table > delete the 1st step is to delete the table form META anyways -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24664) Some changing of split region by overall region size rather than only one store size
[ https://issues.apache.org/jira/browse/HBASE-24664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237631#comment-17237631 ] Andrew Kyle Purtell commented on HBASE-24664: - This is already committed to branch-2 (for 2.4.) and fix version marked accordingly. Resolving as complete. Open new issues for backports. > Some changing of split region by overall region size rather than only one > store size > > > Key: HBASE-24664 > URL: https://issues.apache.org/jira/browse/HBASE-24664 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Zheng Wang >Assignee: Zheng Wang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > As a distributed cluster, HBase distribute loads in unit of region, so if > region grows too big, > it will bring some negative effects, such as: > 1. Harder to homogenize disk usage(consider locality) > 2. Might cost more time on region opening > 3. After split, the daughter region might lead to more io cost on compaction > in a short time(if write evenly) > I tried to introduce a new SteppingAllStoresSizeSplitPolicy in HBASE-24530, > but after discussed in comments and related > [thread|https://lists.apache.org/thread.html/r08a8103e2532eb667a0fcb4efa8a4117b3f82e6251bc4bd0bc157c26%40%3Cdev.hbase.apache.org%3E], > finally we decide to change the existing split policy with a new option that > if it should count all store files, and for master it would be true, else > false. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25019) check if we're impacted by MASSEMBLY-941 and mitigate if needed
[ https://issues.apache.org/jira/browse/HBASE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237637#comment-17237637 ] Andrew Kyle Purtell commented on HBASE-25019: - Moving out of 2.4. I can check this manually when reviewing the bits for 2.4.0 RC0. I don't think missing executable bits are sufficient to sink a release though. I agree it is inconvenient. I can fix this by hand and repack tarballs before calculating checksums on those tarballs unless someone finds that objectionable. (And agreed up front that is not a real solution.) > check if we're impacted by MASSEMBLY-941 and mitigate if needed > --- > > Key: HBASE-25019 > URL: https://issues.apache.org/jira/browse/HBASE-25019 > Project: HBase > Issue Type: Task > Components: scripts >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.3.1, 1.6.0, 1.7.0 >Reporter: Sean Busbey >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.7.0, 2.3.4, 2.5.0 > > > MASSEMBLY-941 notes a bug starting in version 3.2.0 of the assembly plugin > where scripts lose their executable bit. > We've had this version since updating to the apache parent pom version 22 in > HBASE-23675. We should check our release artifacts to see if we are impacted > and if so downgrade the assembly plugin in our poms to 3.1.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25050) We initialize Filesystems more than once.
[
https://issues.apache.org/jira/browse/HBASE-25050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237657#comment-17237657
]
Andrew Kyle Purtell commented on HBASE-25050:
-
There are about 60 hours remaining before the 2.4.0RC0 is unblocked, if you
want to get this in to master and branch-2 before then.
> We initialize Filesystems more than once.
> -
>
> Key: HBASE-25050
> URL: https://issues.apache.org/jira/browse/HBASE-25050
> Project: HBase
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.4.0, 2.2.6
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Minor
>
> In HFileSystem
> {code}
> // Create the default filesystem with checksum verification switched on.
> // By default, any operation to this FilterFileSystem occurs on
> // the underlying filesystem that has checksums switched on.
> this.fs = FileSystem.get(conf);
> this.useHBaseChecksum = useHBaseChecksum;
> fs.initialize(getDefaultUri(conf), conf);
> {code}
> We call fs.initialize(). Generally the FS would have been created and inited
> either in the FileSystem.get() call above or even when we try to check
> {code}
> FileSystem fs = p.getFileSystem(c);
> {code}
> The FS that gets cached in the hadoop-common layer does the init for us. So
> we doing it again is redundant.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-24804) Follow up work to add client side scan metrics for read replica
[ https://issues.apache.org/jira/browse/HBASE-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237659#comment-17237659 ] Andrew Kyle Purtell commented on HBASE-24804: - Scheduling for 2.5. (although could go into a 2.4.1 if we have one.) > Follow up work to add client side scan metrics for read replica > --- > > Key: HBASE-24804 > URL: https://issues.apache.org/jira/browse/HBASE-24804 > Project: HBase > Issue Type: New Feature > Components: read replicas >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > This is a followup work for HBASE-18436, which adds client metrics for read > replica get. Will add metrics for scan as well. This metrics will be used in > PE and any interested applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237663#comment-17237663 ] Andrew Kyle Purtell commented on HBASE-24640: - Copying from the thread discussion, the full proposal is 2. purge the VisibleForTesting annotation from our codebase for 2.4+, involving: 2a. replace VisibleForTesting with IA.Private anywhere method visibility cannot be limited 2b. perhaps add a new Yetus check that would ban new use of VisibleForTesting > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25242) Add Increment/Append support to RowMutations
[
https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237613#comment-17237613
]
Andrew Kyle Purtell commented on HBASE-25242:
-
FWIW I reviewed the PR and approved it with this comment:
{quote}lgtm
Please wait for @Apache9 's approval before merging.
{quote}
> Add Increment/Append support to RowMutations
>
>
> Key: HBASE-25242
> URL: https://issues.apache.org/jira/browse/HBASE-25242
> Project: HBase
> Issue Type: New Feature
> Components: Client, regionserver
>Reporter: Toshihiro Suzuki
>Assignee: Toshihiro Suzuki
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
>
> Currently, RowMutations supports only Put and Delete. Supporting
> Increment/Append in RowMutations would be helpful for some use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237614#comment-17237614 ] Andrew Kyle Purtell commented on HBASE-25242: - Moved back in. I think this can be done in time. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23639) Moving classes out of hbase-it /test for direct API use of chaos.
[ https://issues.apache.org/jira/browse/HBASE-23639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237620#comment-17237620 ] Andrew Kyle Purtell commented on HBASE-23639: - Moving out of 2.4. > Moving classes out of hbase-it /test for direct API use of chaos. > - > > Key: HBASE-23639 > URL: https://issues.apache.org/jira/browse/HBASE-23639 > Project: HBase > Issue Type: Improvement > Components: API, integration tests, test >Reporter: Mihir Monani >Assignee: Lokesh Khurana >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > Attachments: HBASE-23639-master.patch > > > In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager > which has some part of implementation code for Chaos Action. > > If any user wants to create private implementation which resembles > HBaseClusterManager, then it has to be merged with hbase-it package as > ClusterManager Interface and it's extended implementations are private. > > We should make them public so anyone can have their own implementation and > need to be merged with hbase-it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23602: Fix Version/s: (was: 2.4.0) 2.5.0 > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-21946) Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once HDFS-3246 prepared
[ https://issues.apache.org/jira/browse/HBASE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237623#comment-17237623 ] Andrew Kyle Purtell commented on HBASE-21946: - Moving out of 2.4. > Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once > HDFS-3246 prepared > -- > > Key: HBASE-21946 > URL: https://issues.apache.org/jira/browse/HBASE-21946 > Project: HBase > Issue Type: Improvement > Components: Offheaping >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: HBASE-21946.HBASE-21879.v01.patch, > HBASE-21946.HBASE-21879.v02.patch, HBASE-21946.HBASE-21879.v03.patch, > HBASE-21946.HBASE-21879.v04.patch > > > [~stakiar] is working on HDFS-3246, so now we have to keep the byte[] pread > in HFileBlock reading. Once it get resolved, we can upgrade the hadoop > version and do the replacement. > I think it will be a great p999 latency improvement in 100% Get case, anyway > file a issue address this firstly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-22356) API to get hdfs block distribution from regionservers
[ https://issues.apache.org/jira/browse/HBASE-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-22356: Fix Version/s: (was: 2.4.0) 2.5.0 Status: Open (was: Patch Available) Stale patch, unscheduling. Moving out of 2.4. > API to get hdfs block distribution from regionservers > - > > Key: HBASE-22356 > URL: https://issues.apache.org/jira/browse/HBASE-22356 > Project: HBase > Issue Type: Sub-task > Components: API, Balancer, regionserver >Reporter: Thiruvel Thirumoolan >Assignee: Thiruvel Thirumoolan >Priority: Major > Labels: balancer > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > Attachments: HBASE-22356.master.001.patch, > HBASE-22356.master.002.patch, HBASE-22356.master.003.patch > > > A RegionServer API has to be added which will return HDFSBlockDistribution > for all the regions it hosts. RS already has this info cached and updated > when flush/compaction happens. Master can query and get instead of hitting > the namenode and caching. The larger the cluster becomes, the more costly it > becomes to get this information and more stale the cached information becomes. > This jira is only to add the API to regionserver. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-21946) Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once HDFS-3246 prepared
[ https://issues.apache.org/jira/browse/HBASE-21946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-21946: Fix Version/s: (was: 2.3.4) (was: 2.4.0) 2.5.0 > Replace the byte[] pread by ByteBuffer pread in HFileBlock reading once > HDFS-3246 prepared > -- > > Key: HBASE-21946 > URL: https://issues.apache.org/jira/browse/HBASE-21946 > Project: HBase > Issue Type: Improvement > Components: Offheaping >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: HBASE-21946.HBASE-21879.v01.patch, > HBASE-21946.HBASE-21879.v02.patch, HBASE-21946.HBASE-21879.v03.patch, > HBASE-21946.HBASE-21879.v04.patch > > > [~stakiar] is working on HDFS-3246, so now we have to keep the byte[] pread > in HFileBlock reading. Once it get resolved, we can upgrade the hadoop > version and do the replacement. > I think it will be a great p999 latency improvement in 100% Get case, anyway > file a issue address this firstly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24813) ReplicationSource should clear buffer usage on ReplicationSourceManager upon termination
[ https://issues.apache.org/jira/browse/HBASE-24813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24813: Fix Version/s: (was: 2.4.0) 2.5.0 > ReplicationSource should clear buffer usage on ReplicationSourceManager upon > termination > > > Key: HBASE-24813 > URL: https://issues.apache.org/jira/browse/HBASE-24813 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.6 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0 > > Attachments: TestReplicationSyncUpTool.log, > image-2020-10-09-10-50-00-372.png > > > Following investigations on the issue described by [~elserj] on HBASE-24779, > we found out that once a peer is removed, thus killing peers related > *ReplicationSource* instance, it may leave > *ReplicationSourceManager.totalBufferUsed* inconsistent. This can happen if > *ReplicationSourceWALReader* had put some entries on its queue to be > processed by *ReplicationSourceShipper,* but the peer removal killed the > shipper before it could process the pending entries. When > *ReplicationSourceWALReader* thread add entries to the queue, it increments > *ReplicationSourceManager.totalBufferUsed* with the sum of the entries sizes. > When those entries are read by *ReplicationSourceShipper,* > *ReplicationSourceManager.totalBufferUsed* is then decreased. We should also > decrease *ReplicationSourceManager.totalBufferUsed* when *ReplicationSource* > is terminated, otherwise those unprocessed entries size would be consuming > *ReplicationSourceManager.totalBufferUsed __*indefinitely, unless the RS gets > restarted. This may be a problem for deployments with multiple peers, or if > new peers are added.** -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24679) HBase on Cloud Blob FS : Provide config to skip HFile archival while table deletion
[ https://issues.apache.org/jira/browse/HBASE-24679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237633#comment-17237633 ] Andrew Kyle Purtell commented on HBASE-24679: - This should only be scheduled against trunk (3.0) until there is an actual implementation. > HBase on Cloud Blob FS : Provide config to skip HFile archival while table > deletion > > > Key: HBASE-24679 > URL: https://issues.apache.org/jira/browse/HBASE-24679 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0-alpha-1 > > > When we delete a table as part of delete of table from FS, we do below things > 1. Rename to table directory to come under /hbase/.tmp. This is an atomic > rename op > 2. Go through each of HFiles under every region:cf and archive that one by > one. (Rename the file from .tmp path to go to /hbase/archive) > 3. Delete the table dir under .tmp dir > In case of HDFS this is not a big deal as every rename op is just a meta op > (Though the HFiles archival is a costly only as there will be so many calls > to NN based the table's regions# and total storesfiles#) But on Cloud blob > based FS impl, this is a concerning op. Every rename will be a copy blob op. > And we are doing it twice per each of the HFiles in this table ! > The proposal here is to provide a config option (default to false) to skip > this archival step. > We can provide another config to even avoid the .tmp rename? The atomicity of > the Table delete can be achieved by HM side procedure and proc WAL. In table > delete the 1st step is to delete the table form META anyways -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24813) ReplicationSource should clear buffer usage on ReplicationSourceManager upon termination
[ https://issues.apache.org/jira/browse/HBASE-24813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237634#comment-17237634 ] Andrew Kyle Purtell commented on HBASE-24813: - Moving out of 2.4. > ReplicationSource should clear buffer usage on ReplicationSourceManager upon > termination > > > Key: HBASE-24813 > URL: https://issues.apache.org/jira/browse/HBASE-24813 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 3.0.0-alpha-1, 2.3.1, 2.2.6 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 3.0.0-alpha-1, 2.2.7, 2.3.4, 2.5.0 > > Attachments: TestReplicationSyncUpTool.log, > image-2020-10-09-10-50-00-372.png > > > Following investigations on the issue described by [~elserj] on HBASE-24779, > we found out that once a peer is removed, thus killing peers related > *ReplicationSource* instance, it may leave > *ReplicationSourceManager.totalBufferUsed* inconsistent. This can happen if > *ReplicationSourceWALReader* had put some entries on its queue to be > processed by *ReplicationSourceShipper,* but the peer removal killed the > shipper before it could process the pending entries. When > *ReplicationSourceWALReader* thread add entries to the queue, it increments > *ReplicationSourceManager.totalBufferUsed* with the sum of the entries sizes. > When those entries are read by *ReplicationSourceShipper,* > *ReplicationSourceManager.totalBufferUsed* is then decreased. We should also > decrease *ReplicationSourceManager.totalBufferUsed* when *ReplicationSource* > is terminated, otherwise those unprocessed entries size would be consuming > *ReplicationSourceManager.totalBufferUsed __*indefinitely, unless the RS gets > restarted. This may be a problem for deployments with multiple peers, or if > new peers are added.** -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24850: Fix Version/s: (was: 2.4.0) 2.5.0 3.0.0-alpha-1 > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24850) CellComparator perf improvement
[ https://issues.apache.org/jira/browse/HBASE-24850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237635#comment-17237635 ] Andrew Kyle Purtell commented on HBASE-24850: - Moving out of 2.4. > CellComparator perf improvement > --- > > Key: HBASE-24850 > URL: https://issues.apache.org/jira/browse/HBASE-24850 > Project: HBase > Issue Type: Improvement > Components: Performance, scan >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: ramkrishna.s.vasudevan >Priority: Critical > Fix For: 3.0.0-alpha-1, 2.5.0 > > > We have multiple perf issues in 2.x versions compared to 1.x. Eg: > HBASE-24754, HBASE-24637. > The pattern is clear that where ever we do more and more Cell compares, there > is some degrade. In HBASE-24754, with an old KVComparator style comparator, > we see much better perf for the PutSortReducer. (Again the gain is huge > because of large number of compare ops that test is doing). This issue is to > address and optimize compares generally in CellComparatorImpl itself. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25019) check if we're impacted by MASSEMBLY-941 and mitigate if needed
[ https://issues.apache.org/jira/browse/HBASE-25019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25019: Fix Version/s: (was: 2.4.0) 2.5.0 > check if we're impacted by MASSEMBLY-941 and mitigate if needed > --- > > Key: HBASE-25019 > URL: https://issues.apache.org/jira/browse/HBASE-25019 > Project: HBase > Issue Type: Task > Components: scripts >Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.3.1, 1.6.0, 1.7.0 >Reporter: Sean Busbey >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.7.0, 2.3.4, 2.5.0 > > > MASSEMBLY-941 notes a bug starting in version 3.2.0 of the assembly plugin > where scripts lose their executable bit. > We've had this version since updating to the apache parent pom version 22 in > HBASE-23675. We should check our release artifacts to see if we are impacted > and if so downgrade the assembly plugin in our poms to 3.1.1. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25127) Enhance PerformanceEvaluation to profile meta replica performance.
[ https://issues.apache.org/jira/browse/HBASE-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237641#comment-17237641 ] Andrew Kyle Purtell commented on HBASE-25127: - Ah, my mistake, I see PR 2682 is not yet merged and that is why this is still open. PR 2682 has an approval and can be merged. You have approximately 60 hours remaining before the 2.4.0RC0 is unblocked. If this is not merged by then I will move it out. > Enhance PerformanceEvaluation to profile meta replica performance. > -- > > Key: HBASE-25127 > URL: https://issues.apache.org/jira/browse/HBASE-25127 > Project: HBase > Issue Type: Sub-task >Reporter: Huaxiang Sun >Assignee: Clara Xiong >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: Screen Shot 2020-11-13 at 5.30.11 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25187) Improve SizeCachedKV variants initialization
[ https://issues.apache.org/jira/browse/HBASE-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237645#comment-17237645 ] Andrew Kyle Purtell commented on HBASE-25187: - Moved out of 2.4. There are about 60 hours remaining before the 2.4.0RC0 is unblocked, if you want to get this in before then. > Improve SizeCachedKV variants initialization > > > Key: HBASE-25187 > URL: https://issues.apache.org/jira/browse/HBASE-25187 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0 > > > Currently in SizeCachedKV we get the rowlength and Key length from the > buffers. This can be optimized because we can pass the keylen and row len > while actually creating the cell while reading the cell from the block. Some > times we see that the SizeCachedKV takes the max width in a flame graph - > considering the fact we also do a sanity check on the created KV. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25187) Improve SizeCachedKV variants initialization
[ https://issues.apache.org/jira/browse/HBASE-25187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25187: Fix Version/s: (was: 2.4.0) 2.5.0 > Improve SizeCachedKV variants initialization > > > Key: HBASE-25187 > URL: https://issues.apache.org/jira/browse/HBASE-25187 > Project: HBase > Issue Type: Improvement >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.3.4, 2.5.0 > > > Currently in SizeCachedKV we get the rowlength and Key length from the > buffers. This can be optimized because we can pass the keylen and row len > while actually creating the cell while reading the cell from the block. Some > times we see that the SizeCachedKV takes the max width in a flame graph - > considering the fact we also do a sanity check on the created KV. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25298) hbase.rsgroup.fallback.enable should support dynamic configuration
[ https://issues.apache.org/jira/browse/HBASE-25298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-25298. - Fix Version/s: 3.0.0-alpha-1 Resolution: Fixed The PR was merged to master branch. > hbase.rsgroup.fallback.enable should support dynamic configuration > --- > > Key: HBASE-25298 > URL: https://issues.apache.org/jira/browse/HBASE-25298 > Project: HBase > Issue Type: Improvement >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Baiqiang Zhao >Assignee: Baiqiang Zhao >Priority: Major > Fix For: 3.0.0-alpha-1 > > > Use update_config command to control the switch of RSGroup fallback. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24804) Follow up work to add client side scan metrics for read replica
[ https://issues.apache.org/jira/browse/HBASE-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24804: Affects Version/s: (was: 2.4.0) > Follow up work to add client side scan metrics for read replica > --- > > Key: HBASE-24804 > URL: https://issues.apache.org/jira/browse/HBASE-24804 > Project: HBase > Issue Type: New Feature > Components: read replicas >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > > This is a followup work for HBASE-18436, which adds client metrics for read > replica get. Will add metrics for scan as well. This metrics will be used in > PE and any interested applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24804) Follow up work to add client side scan metrics for read replica
[ https://issues.apache.org/jira/browse/HBASE-24804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24804: Fix Version/s: 2.5.0 3.0.0-alpha-1 > Follow up work to add client side scan metrics for read replica > --- > > Key: HBASE-24804 > URL: https://issues.apache.org/jira/browse/HBASE-24804 > Project: HBase > Issue Type: New Feature > Components: read replicas >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Minor > Fix For: 3.0.0-alpha-1, 2.5.0 > > > This is a followup work for HBASE-18436, which adds client metrics for read > replica get. Will add metrics for scan as well. This metrics will be used in > PE and any interested applications. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237672#comment-17237672 ] Andrew Kyle Purtell commented on HBASE-24640: - [~reidchan] This looks like something I can do for 1.7.0 as well. > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25308) [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous
[ https://issues.apache.org/jira/browse/HBASE-25308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237674#comment-17237674 ] Andrew Kyle Purtell commented on HBASE-25308: - HBASE-24640 proposes to remove VisibleForTesting, which will make the patch for this issue much smaller. > [branch-1] Consume Guava from hbase-thirdparty hbase-shaded-miscellaneous > - > > Key: HBASE-25308 > URL: https://issues.apache.org/jira/browse/HBASE-25308 > Project: HBase > Issue Type: Improvement >Reporter: Andrew Kyle Purtell >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 1.7.0 > > > We are again having classpath versioning issues related to Guava in our > branch-1 based application. > Hadoop 3, HBase 2, Phoenix 5, and other projects deal with Guava > cross-version incompatibilities, as they manifest on a combined classpath > with other components, via shading. > I propose to do a global search and replace of all direct uses of Guava in > our branch-1 code base and refer to Guava as provided in hbase-thirdparty's > hbase-shaded-miscellaneous. This will protect HBase branch-1 from Guava > cross-version vagaries just like the same technique protects branch-2 and > branch-2 based releases. > There are a couple of Public or LimitedPrivate interfaces that incorporate > Guava's HostAndPort and Service that will be indirectly impacted. We are > about to release a new minor branch-1 version, 1.7.0, and this would be a > great opportunity to introduce this kind of change in a manner consistent > with semantic versioning and our compatibility policies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-23639) Moving classes out of hbase-it /test for direct API use of chaos.
[ https://issues.apache.org/jira/browse/HBASE-23639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-23639: Fix Version/s: (was: 2.4.0) 2.5.0 > Moving classes out of hbase-it /test for direct API use of chaos. > - > > Key: HBASE-23639 > URL: https://issues.apache.org/jira/browse/HBASE-23639 > Project: HBase > Issue Type: Improvement > Components: API, integration tests, test >Reporter: Mihir Monani >Assignee: Lokesh Khurana >Priority: Minor > Fix For: 3.0.0-alpha-1, 1.7.0, 2.5.0 > > Attachments: HBASE-23639-master.patch > > > In hbase-it package, Class like RESTApiClusterManager and HBaseClusterManager > which has some part of implementation code for Chaos Action. > > If any user wants to create private implementation which resembles > HBaseClusterManager, then it has to be merged with hbase-it package as > ClusterManager Interface and it's extended implementations are private. > > We should make them public so anyone can have their own implementation and > need to be merged with hbase-it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23602) TTL Before Which No Data is Purged
[ https://issues.apache.org/jira/browse/HBASE-23602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237621#comment-17237621 ] Andrew Kyle Purtell commented on HBASE-23602: - Moving out of 2.4. > TTL Before Which No Data is Purged > -- > > Key: HBASE-23602 > URL: https://issues.apache.org/jira/browse/HBASE-23602 > Project: HBase > Issue Type: New Feature >Reporter: Geoffrey Jacoby >Assignee: Geoffrey Jacoby >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > HBase currently offers operators a choice. They can set > KEEP_DELETED_CELLS=true and VERSIONS to max value, plus no TTL, and they will > always have a complete history of all changes (but high storage costs and > penalties to read performance). Or they can have KEEP_DELETED_CELLS=false and > VERSIONS/TTL set to some reasonable values, but that means that major > compactions can destroy the ability to do a consistent snapshot read of any > prior time. (This limits the usefulness and correctness of, for example, > Phoenix's SCN lookback feature.) > I propose having a new TTL property to give a minimum age that an expired or > deleted Cell would have to achieve before it could be purged. (I see that > HBASE-10118 already does something similar for the delete markers > themselves.) > This would allow operators to have a consistent history for some finite > amount of recent time while still purging out the "long tail" of obsolete / > deleted versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24352) Skip HDFSBlockDistribution calc when FS is not HDFS
[ https://issues.apache.org/jira/browse/HBASE-24352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24352: Fix Version/s: (was: 2.4.0) 2.5.0 > Skip HDFSBlockDistribution calc when FS is not HDFS > --- > > Key: HBASE-24352 > URL: https://issues.apache.org/jira/browse/HBASE-24352 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > HDFSBlockDistribution is used in different places like Balancer, > DateTieredCompaction, CompactionTool etc. In Balancer area there is a config > 'hbase.master.balancer.uselocality' using which we can skip this. But > irrespective of this config, if we are on non HDFS FS, we should skip this. > The block distribution issue many file status commands to underlying FS which > wont be that cheap in a cloud FS. This jira aims at correcting all these > places. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-20897) Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 version" to branch-2 and up
[ https://issues.apache.org/jira/browse/HBASE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-20897: Fix Version/s: (was: 2.4.0) 2.5.0 > Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 > version" to branch-2 and up > - > > Key: HBASE-20897 > URL: https://issues.apache.org/jira/browse/HBASE-20897 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Andrew Kyle Purtell >Assignee: Vikas Vishwakarma >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24081) Provide documentation for running Yetus with HBase
[ https://issues.apache.org/jira/browse/HBASE-24081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24081: Fix Version/s: (was: 2.4.0) (was: 1.7.0) > Provide documentation for running Yetus with HBase > -- > > Key: HBASE-24081 > URL: https://issues.apache.org/jira/browse/HBASE-24081 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A colleague asked how to use Yetus with HBase, so I wrote up a little how-to > doc. Maybe it's useful to someone else? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-20897) Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 version" to branch-2 and up
[ https://issues.apache.org/jira/browse/HBASE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-20897: --- Assignee: (was: Vikas Vishwakarma) > Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 > version" to branch-2 and up > - > > Key: HBASE-20897 > URL: https://issues.apache.org/jira/browse/HBASE-20897 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-20897) Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 version" to branch-2 and up
[ https://issues.apache.org/jira/browse/HBASE-20897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237626#comment-17237626 ] Andrew Kyle Purtell commented on HBASE-20897: - Unassigned. Bumped to 2.5. > Port HBASE-20866 "HBase 1.x scan performance degradation compared to 0.98 > version" to branch-2 and up > - > > Key: HBASE-20897 > URL: https://issues.apache.org/jira/browse/HBASE-20897 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24081) Provide documentation for running Yetus with HBase
[ https://issues.apache.org/jira/browse/HBASE-24081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237627#comment-17237627 ] Andrew Kyle Purtell commented on HBASE-24081: - Moved out from 2.4. We have approximately 60 hours before the 2.4.0 RC0 is unblocked. If you want to get this in, please complete and merge the PRs in that timeframe. Will circle back to review them myself shortly. > Provide documentation for running Yetus with HBase > -- > > Key: HBASE-24081 > URL: https://issues.apache.org/jira/browse/HBASE-24081 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A colleague asked how to use Yetus with HBase, so I wrote up a little how-to > doc. Maybe it's useful to someone else? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24352) Skip HDFSBlockDistribution calc when FS is not HDFS
[ https://issues.apache.org/jira/browse/HBASE-24352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237628#comment-17237628 ] Andrew Kyle Purtell commented on HBASE-24352: - Moved out of 2.4. > Skip HDFSBlockDistribution calc when FS is not HDFS > --- > > Key: HBASE-24352 > URL: https://issues.apache.org/jira/browse/HBASE-24352 > Project: HBase > Issue Type: Improvement >Reporter: Anoop Sam John >Assignee: Anoop Sam John >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > HDFSBlockDistribution is used in different places like Balancer, > DateTieredCompaction, CompactionTool etc. In Balancer area there is a config > 'hbase.master.balancer.uselocality' using which we can skip this. But > irrespective of this config, if we are on non HDFS FS, we should skip this. > The block distribution issue many file status commands to underlying FS which > wont be that cheap in a cloud FS. This jira aims at correcting all these > places. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25127) Enhance PerformanceEvaluation to profile meta replica performance.
[ https://issues.apache.org/jira/browse/HBASE-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237639#comment-17237639 ] Andrew Kyle Purtell commented on HBASE-25127: - Can we close this now that the revert-and-replace of the commit has been finished? There are approximately 60 hours remaining before the 2.4.0RC0 is unblocked. > Enhance PerformanceEvaluation to profile meta replica performance. > -- > > Key: HBASE-25127 > URL: https://issues.apache.org/jira/browse/HBASE-25127 > Project: HBase > Issue Type: Sub-task >Reporter: Huaxiang Sun >Assignee: Clara Xiong >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: Screen Shot 2020-11-13 at 5.30.11 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25127) Enhance PerformanceEvaluation to profile meta replica performance.
[ https://issues.apache.org/jira/browse/HBASE-25127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237639#comment-17237639 ] Andrew Kyle Purtell edited comment on HBASE-25127 at 11/23/20, 7:20 PM: Can we resolve this again this now that the revert-and-replace of the commit has been finished? There are approximately 60 hours remaining before the 2.4.0RC0 is unblocked. was (Author: apurtell): Can we close this now that the revert-and-replace of the commit has been finished? There are approximately 60 hours remaining before the 2.4.0RC0 is unblocked. > Enhance PerformanceEvaluation to profile meta replica performance. > -- > > Key: HBASE-25127 > URL: https://issues.apache.org/jira/browse/HBASE-25127 > Project: HBase > Issue Type: Sub-task >Reporter: Huaxiang Sun >Assignee: Clara Xiong >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > Attachments: Screen Shot 2020-11-13 at 5.30.11 PM.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25287) Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files
[
https://issues.apache.org/jira/browse/HBASE-25287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Kyle Purtell updated HBASE-25287:
Fix Version/s: 2.5.0
3.0.0-alpha-1
Status: Patch Available (was: Open)
There is a PR pending. Moving to Patch available status.
The PR targets branch-2. We require changes to first be committed to master
branch, then backported. I assume a forward port will be relatively simple.
There are about 60 hours remaining before the 2.4.0RC0 is unblocked, if you
want to get this in before then. I have moved this out of 2.4 assuming it will
miss but we can update the fix version if it gets in on time.
> Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when
> loading files
>
>
> Key: HBASE-25287
> URL: https://issues.apache.org/jira/browse/HBASE-25287
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Xiaolin Ha
>Assignee: Xiaolin Ha
>Priority: Major
> Fix For: 3.0.0-alpha-1, 2.5.0
>
> Attachments: 1605328358304-image.png, 1605328417888-image.png,
> 1605504914256-image.png
>
>
> HBASE-9393 found seek+read will leave many CLOSE_WAIT sockets without stream
> unbuffer, which can free sockets and file descriptors held by the stream.
> In our cluster RSes with about one hundred thousand store files, we found the
> number of CLOSE_WAIT sockets increases with the number of regions opened,
> and can up to the operating system open files limit 100.
>
> {code:java}
> 2020-11-12 20:19:02,452 WARN [1282990092@qtp-220038608-1 - Acceptor0
> SelectChannelConnector@0.0.0.0:16030] mortbay.log: EXCEPTION
> java.io.IOException: Too many open files
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
> at
> sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
> at
> org.mortbay.jetty.nio.SelectChannelConnector$1.acceptChannel(SelectChannelConnector.java:75)
> at
> org.mortbay.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:686)
> at
> org.mortbay.io.nio.SelectorManager.doSelect(SelectorManager.java:192)
> at
> org.mortbay.jetty.nio.SelectChannelConnector.accept(SelectChannelConnector.java:124)
> at
> org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:708)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}
>
> {code:java}
> [hbase@gha-data-hbase-cat0053 hbase]$ ulimit -SHn
> 100
> {code}
>
>
> The reason of the problem is, when store file opened,
> {code:java}
> private void open() throws IOException {
> fileInfo.initHDFSBlocksDistribution();
> long readahead = fileInfo.isNoReadahead() ? 0L : -1L;
> ReaderContext context = fileInfo.createReaderContext(false, readahead,
> ReaderType.PREAD);
> fileInfo.initHFileInfo(context);
> StoreFileReader reader = fileInfo.preStoreFileReaderOpen(context,
> cacheConf);
> if (reader == null) {
> reader = fileInfo.createReader(context, cacheConf);
> fileInfo.getHFileInfo().initMetaAndIndex(reader.getHFileReader());
> }
> {code}
> only createReader() unbuffered the stream. In initMetaAndIndex(), using the
> stream to read blocks, so it needs to unbuffer() the socket , too.
> We can just add try before fileInfo.initHFileInfo(context); and finally
> unbuffer() the stream at the end of the open() function.
> We fixed it on our cluster, the number of CLOSE_WAIT reduced to about 0.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24877) Add option to avoid aborting RS process upon uncaught exceptions happen on replication source
[ https://issues.apache.org/jira/browse/HBASE-24877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-24877. - Fix Version/s: 2.4.0 3.0.0-alpha-1 Resolution: Fixed PRs were merged to master and branch-2. Resolving. File new issues for any further backports. > Add option to avoid aborting RS process upon uncaught exceptions happen on > replication source > - > > Key: HBASE-24877 > URL: https://issues.apache.org/jira/browse/HBASE-24877 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, we abort entire RS process if any uncaught exceptions happens on > ReplicationSource initialization. This may be too extreme on certain > deployments, where custom replication endpoint implementations may choose to > do so when remote peers are unavailable, but source cluster shouldn't be > brought down entirely. Similarly, source reader and shipper threads would > cause RS to abort on any runtime exception occurrence while running. > This patch adds configuration option (false by default, to keep the original > behaviour), to avoid aborting entire RS processes under these conditions. > Instead, if ReplicationSource initialization fails with a RuntimeException, > it keeps retrying the source startup. In the case of readers/shippers runtime > errors, it refreshes the replication source, terminating current source and > its readers/shippers and creating new ones. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell reassigned HBASE-24640: --- Assignee: Andrew Kyle Purtell > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237661#comment-17237661 ] Andrew Kyle Purtell commented on HBASE-24640: - I encountered issues with VisibleForTesting on HBASE-25308 . +1 for a total removal > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237667#comment-17237667 ] Andrew Kyle Purtell commented on HBASE-24640: - Here is what I plan to do: * Find all VisibleForTesting annotations. * If the enclosing class or interface is IA Public or LImitedPrivate, replace the VisibleForTesting annotation with IA.Private. * If the enclosing class or interface is IA Private, remove the VisibleForTesting annotation. > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-24640 started by Andrew Kyle Purtell. --- > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24640: Fix Version/s: 2.4.0 1.7.0 3.0.0-alpha-1 > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-24081) Provide documentation for running Yetus with HBase
[ https://issues.apache.org/jira/browse/HBASE-24081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237669#comment-17237669 ] Andrew Kyle Purtell commented on HBASE-24081: - Actually this all looks merged, including a fix. Just resolving instead. > Provide documentation for running Yetus with HBase > -- > > Key: HBASE-24081 > URL: https://issues.apache.org/jira/browse/HBASE-24081 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1 > > > A colleague asked how to use Yetus with HBase, so I wrote up a little how-to > doc. Maybe it's useful to someone else? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24081) Provide documentation for running Yetus with HBase
[ https://issues.apache.org/jira/browse/HBASE-24081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell resolved HBASE-24081. - Fix Version/s: 2.4.0 Resolution: Fixed > Provide documentation for running Yetus with HBase > -- > > Key: HBASE-24081 > URL: https://issues.apache.org/jira/browse/HBASE-24081 > Project: HBase > Issue Type: Task > Components: documentation >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0 > > > A colleague asked how to use Yetus with HBase, so I wrote up a little how-to > doc. Maybe it's useful to someone else? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Fix Version/s: (was: 2.4.0) 2.5.0 > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Not required for 2.4.0. Moving out. Please dont' make a blocker something that is not actually a blocker. This is an nice to have. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Blocker > Fix For: 3.0.0-alpha-1, 2.4.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237600#comment-17237600 ] Andrew Kyle Purtell commented on HBASE-25242: - A security issue in a third party dependency has been raised as a concern. We need a new release of hbase-thirdparty, 3.4.1, to complete, before 2.4.0 RC0 can be built. At this point the RC is waiting for the new release of hbase-thirdparty to happen. That gives you time here [~brfrn169] [~zhangduo] I have moved this out of 2.4.0 but it can be brought back in if the PR is merged in time. I would encourage the code review on PR #2630 to move with all deliberate haste if you want it to get in. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25242: Priority: Major (was: Blocker) > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25242) Add Increment/Append support to RowMutations
[ https://issues.apache.org/jira/browse/HBASE-25242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237591#comment-17237591 ] Andrew Kyle Purtell edited comment on HBASE-25242 at 11/23/20, 6:41 PM: Not required for 2.4.0. Moving out. Please dont' make a blocker something that is not actually a blocker. This is an nice to have. Even on the meta replica issue where we decided to hold up the RC to wait for it, when Stack tried to make it Blocker priority I made the same objection there too. Only true blockers should be filed at Blocker status. A true release blocker is a critical performance regression, or compatibility break, or compilation break, or something that causes widespread test failures, or a violation of Apache Foundation release policy (like RAT audit failure). was (Author: apurtell): Not required for 2.4.0. Moving out. Please dont' make a blocker something that is not actually a blocker. This is an nice to have. > Add Increment/Append support to RowMutations > > > Key: HBASE-25242 > URL: https://issues.apache.org/jira/browse/HBASE-25242 > Project: HBase > Issue Type: New Feature > Components: Client, regionserver >Reporter: Toshihiro Suzuki >Assignee: Toshihiro Suzuki >Priority: Major > Fix For: 3.0.0-alpha-1, 2.5.0 > > > Currently, RowMutations supports only Put and Delete. Supporting > Increment/Append in RowMutations would be helpful for some use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25277) postScannerFilterRow impacts Scan performance a lot in HBase 2.x
[ https://issues.apache.org/jira/browse/HBASE-25277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246745#comment-17246745 ] Andrew Kyle Purtell commented on HBASE-25277: - +1 Please merge to all branch-2.x. This is a perf regression / bug fix that does not change API semantics if a CP implements those hooks. > postScannerFilterRow impacts Scan performance a lot in HBase 2.x > > > Key: HBASE-25277 > URL: https://issues.apache.org/jira/browse/HBASE-25277 > Project: HBase > Issue Type: Bug > Components: Coprocessors, scan >Reporter: Pankaj Kumar >Assignee: Pankaj Kumar >Priority: Critical > Labels: perfomance, scanning > Fix For: 3.0.0-alpha-1, 2.5.0 > > Attachments: test_report.png > > > In our test we observed Scan performance is degraded by more than 60% in > HBase-2.x as compared to 1.3.x. As per the flamegraph report, RS spent 31% > of the time in postScannerFilterRow, however the coprocessors > (AccessController, VisibilityController & ConstraintProcessor) does nothing > in that hook. > HBASE-14489 added the logic to avoid the call to postScannerFilterRow when > not needed which is not working as expected in HBase 2.x. AccessController, > VisibilityController & ConstraintProcessor override the postScannerFilterRow > with dummy (same as RegionObserver) implementation, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be TRUE and call the > hook for all configured CPs while processing each row . Suppose we have > configured 5 region CPs and there are 1 M rows in table, then there will be 1 > * 5 M dummy call to postScannerFilterRow during whole table scan. > We need to remove postScannerFilterRow hook from these CPs as these are not > doing anything. > Another problem is in RegionCoprocessorHost.hasCustomPostScannerFilterRow > init logic, currently it is always TRUE even though we remove > postScannerFilterRow hook from AccessController, VisibilityController & > ConstraintProcessor, because we are finding postScannerFilterRow until it > is found (we look in configured CP's super class also) or clazz is NULL. > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L301 > Supper class of Object (super class of AccessController) will be NULL, so > RegionCoprocessorHost.hasCustomPostScannerFilterRow will be set as TRUE > https://github.com/apache/hbase/blob/035c192eb665469ce0c071db86c78f4a873c123b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java#L279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
Andrew Kyle Purtell created HBASE-25352: --- Summary: API compatibilty checker fails with "Argument list too long" Key: HBASE-25352 URL: https://issues.apache.org/jira/browse/HBASE-25352 Project: HBase Issue Type: Bug Affects Versions: 2.4.0 Reporter: Andrew Kyle Purtell While working on the 2.4.0 RC I hit a stumbling block where the argument list passed to javap by the API compatibility checker is too large for Mac OS. Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
[ https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242874#comment-17242874 ] Andrew Kyle Purtell commented on HBASE-25352: - Have you seen this before? [~stack] [~vjasani] [~ndimiduk] > API compatibilty checker fails with "Argument list too long" > > > Key: HBASE-25352 > URL: https://issues.apache.org/jira/browse/HBASE-25352 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Andrew Kyle Purtell >Priority: Major > > While working on the 2.4.0 RC I hit a stumbling block where the argument list > passed to javap by the API compatibility checker is too large for Mac OS. > Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25352) API compatibilty checker fails with "Argument list too long" on Mac OS
[
https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242876#comment-17242876
]
Andrew Kyle Purtell edited comment on HBASE-25352 at 12/3/20, 2:59 AM:
---
Command executed is
{noformat}
python ./dev-support/checkcompatibility.py -r asf \
--annotation org.apache.yetus.audience.InterfaceAudience.Public \
--annotation org.apache.yetus.audience.InterfaceAudience.LimitedPrivate \
--include-file "hbase-*" rel/2.3.0 2.4.0RC0
{noformat}
Exact error is
{noformat}
open3: exec of javap -s -private -c -verbose < REALLY REALLY LONG LIST OF
CLASSES ELIDED > failed:
Argument list too long at
/Users/apurtell/tmp/hbase-tmp/target/java-acc/modules/Internals/APIDump.pm line
357.
cannot remove path when cwd is
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y/29 for
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y: at
/usr/local/Cellar/perl/5.32.0/lib/perl5/5.32.0/File/Temp.pm line 916.
INFO:root:Results: {}
Traceback (most recent call last):
File "./dev-support/checkcompatibility.py", line 530, in
main()
File "./dev-support/checkcompatibility.py", line 526, in main
args.compare_warnings))
File "./dev-support/checkcompatibility.py", line 232, in compare_results
if tool_results[check][issue_type] > known_count]
KeyError: 'binary'
{noformat}
I'm running Mac OS Catalina. ARG_MAX is 262,144 chars.
{noformat}
$ getconf ARG_MAX
262144
{noformat}
On Linux it is considerably larger at 2,097,152 chars:
{noformat}
$ getconf ARG_MAX
2097152
{noformat}
which is I suppose why hitting this is uncommon.
I don't think anything can be done without dev of the API checker itself.
Perhaps if it is still being maintained someone else has hit this and there is
a fix available. If not I could raise a PR. Practically speaking, this is not
super critical because Linux environments will continue to work.
was (Author: apurtell):
Command executed is
{noformat}
python ./dev-support/checkcompatibility.py -r asf \
--annotation org.apache.yetus.audience.InterfaceAudience.Public \
--annotation org.apache.yetus.audience.InterfaceAudience.LimitedPrivate \
--include-file "hbase-*" rel/2.3.0 2.4.0RC0
{noformat}
Exact error is
{noformat}
open3: exec of javap -s -private -c -verbose < REALLY REALLY LONG LIST OF
CLASSES ELIDED > failed:
Argument list too long at
/Users/apurtell/tmp/hbase-tmp/target/java-acc/modules/Internals/APIDump.pm line
357.
cannot remove path when cwd is
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y/29 for
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y: at
/usr/local/Cellar/perl/5.32.0/lib/perl5/5.32.0/File/Temp.pm line 916.
INFO:root:Results: {}
Traceback (most recent call last):
File "./dev-support/checkcompatibility.py", line 530, in
main()
File "./dev-support/checkcompatibility.py", line 526, in main
args.compare_warnings))
File "./dev-support/checkcompatibility.py", line 232, in compare_results
if tool_results[check][issue_type] > known_count]
KeyError: 'binary'
{noformat}
I'm running Mac OS Catalina. ARG_MAX is 262,144 chars.
{noformat}
$ getconf ARG_MAX
262144
{noformat}
On Linux it is considerably larger:
{noformat}
getconf ARG_MAX
2097152
{noformat}
which is I suppose why hitting this is uncommon.
I don't think anything can be done without dev of the API checker itself.
Perhaps if it is still being maintained someone else has hit this and there is
a fix available. If not I could raise a PR. Practically speaking, this is not
super critical because Linux environments will continue to work.
> API compatibilty checker fails with "Argument list too long" on Mac OS
> --
>
> Key: HBASE-25352
> URL: https://issues.apache.org/jira/browse/HBASE-25352
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Andrew Kyle Purtell
>Priority: Minor
>
> While working on the 2.4.0 RC I hit a stumbling block where the argument list
> passed to javap by the API compatibility checker is too large for Mac OS.
> Attempted execution of the forked process fails with "Argument list too long".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
[ https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25352: Priority: Minor (was: Major) > API compatibilty checker fails with "Argument list too long" > > > Key: HBASE-25352 > URL: https://issues.apache.org/jira/browse/HBASE-25352 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Andrew Kyle Purtell >Priority: Minor > > While working on the 2.4.0 RC I hit a stumbling block where the argument list > passed to javap by the API compatibility checker is too large for Mac OS. > Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HBASE-25352) API compatibilty checker fails with "Argument list too long" on Mac OS
[ https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-25352: Summary: API compatibilty checker fails with "Argument list too long" on Mac OS (was: API compatibilty checker fails with "Argument list too long") > API compatibilty checker fails with "Argument list too long" on Mac OS > -- > > Key: HBASE-25352 > URL: https://issues.apache.org/jira/browse/HBASE-25352 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Andrew Kyle Purtell >Priority: Minor > > While working on the 2.4.0 RC I hit a stumbling block where the argument list > passed to javap by the API compatibility checker is too large for Mac OS. > Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
[ https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242875#comment-17242875 ] Andrew Kyle Purtell edited comment on HBASE-25352 at 12/3/20, 2:50 AM: --- I suppose this isn't a problem if the Docker version of the build script is used, because the Docker image is Linux based. That option was not available to me today because the dockerfile had bit rotted. (Stack has fixed it since, I believe.) Going to switch to a Linux VM just to run the compat checker now, will report back. was (Author: apurtell): I suppose this isn't a problem if the Docker version of the build script is used. That option was not available to me today because the dockerfile had bit rotted. (Stack has fixed it since, I believe.) Going to switch to a Linux VM just to run the compat checker now, will report back. > API compatibilty checker fails with "Argument list too long" > > > Key: HBASE-25352 > URL: https://issues.apache.org/jira/browse/HBASE-25352 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Andrew Kyle Purtell >Priority: Major > > While working on the 2.4.0 RC I hit a stumbling block where the argument list > passed to javap by the API compatibility checker is too large for Mac OS. > Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
[ https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242875#comment-17242875 ] Andrew Kyle Purtell commented on HBASE-25352: - I suppose this isn't a problem if the Docker version of the build script is used. That option was not available to me today because the dockerfile had bit rotted. (Stack has fixed it since, I believe.) Going to switch to a Linux VM just to run the compat checker now, will report back. > API compatibilty checker fails with "Argument list too long" > > > Key: HBASE-25352 > URL: https://issues.apache.org/jira/browse/HBASE-25352 > Project: HBase > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Andrew Kyle Purtell >Priority: Major > > While working on the 2.4.0 RC I hit a stumbling block where the argument list > passed to javap by the API compatibility checker is too large for Mac OS. > Attempted execution of the forked process fails with "Argument list too long". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-25352) API compatibilty checker fails with "Argument list too long"
[
https://issues.apache.org/jira/browse/HBASE-25352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242876#comment-17242876
]
Andrew Kyle Purtell commented on HBASE-25352:
-
Command executed is
{noformat}
python ./dev-support/checkcompatibility.py -r asf \
--annotation org.apache.yetus.audience.InterfaceAudience.Public \
--annotation org.apache.yetus.audience.InterfaceAudience.LimitedPrivate \
--include-file "hbase-*" rel/2.3.0 2.4.0RC0
{noformat}
Exact error is
{noformat}
open3: exec of javap -s -private -c -verbose < REALLY REALLY LONG LIST OF
CLASSES ELIDED > failed:
Argument list too long at
/Users/apurtell/tmp/hbase-tmp/target/java-acc/modules/Internals/APIDump.pm line
357.
cannot remove path when cwd is
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y/29 for
/private/var/folders/mq/40qp3y1s12x_mt548v69qcnwgp/T/mjqXHZty0Y: at
/usr/local/Cellar/perl/5.32.0/lib/perl5/5.32.0/File/Temp.pm line 916.
INFO:root:Results: {}
Traceback (most recent call last):
File "./dev-support/checkcompatibility.py", line 530, in
main()
File "./dev-support/checkcompatibility.py", line 526, in main
args.compare_warnings))
File "./dev-support/checkcompatibility.py", line 232, in compare_results
if tool_results[check][issue_type] > known_count]
KeyError: 'binary'
{noformat}
I'm running Mac OS Catalina. ARG_MAX is 262,144 chars.
{noformat}
$ getconf ARG_MAX
262144
{noformat}
On Linux it is considerably larger:
{noformat}
getconf ARG_MAX
2097152
{noformat}
which is I suppose why hitting this is uncommon.
I don't think anything can be done without dev of the API checker itself.
Perhaps if it is still being maintained someone else has hit this and there is
a fix available. If not I could raise a PR. Practically speaking, this is not
super critical because Linux environments will continue to work.
> API compatibilty checker fails with "Argument list too long"
>
>
> Key: HBASE-25352
> URL: https://issues.apache.org/jira/browse/HBASE-25352
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Andrew Kyle Purtell
>Priority: Major
>
> While working on the 2.4.0 RC I hit a stumbling block where the argument list
> passed to javap by the API compatibility checker is too large for Mac OS.
> Attempted execution of the forked process fails with "Argument list too long".
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HBASE-24640) Purge use of VisibleForTesting
[ https://issues.apache.org/jira/browse/HBASE-24640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Kyle Purtell updated HBASE-24640: Status: Patch Available (was: In Progress) > Purge use of VisibleForTesting > -- > > Key: HBASE-24640 > URL: https://issues.apache.org/jira/browse/HBASE-24640 > Project: HBase > Issue Type: Task > Components: community >Affects Versions: 3.0.0-alpha-1, 2.4.0 >Reporter: Nick Dimiduk >Assignee: Andrew Kyle Purtell >Priority: Major > Fix For: 3.0.0-alpha-1, 1.7.0, 2.4.0 > > > From the dev-list thread ["[DISCUSS] VisibleForTesting annotation as it > pertains to our API compatibility > guidelines"|https://lists.apache.org/thread.html/rc7c7c66f134fe135d0a4454a883215e26ff3d20e5a31ecd6a2d1db77%40%3Cdev.hbase.apache.org%3E], > when used in classes annotated with interface audience other than > IA.Private, the VisibleForTesting annotation is confusing and considered > harmful. The consensus is that we do not want to use this annotation as part > of the definition of our public APIs, and we need to remove the point of > confusion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
