Re: [DISCUSS] Disable HDFS readahead for PREAD reads
Thanks everyone! I'll get going on making this change in the near future. On Fri, Jun 2, 2023 at 9:37 AM Nick Dimiduk wrote: > Hi Bryan, > > Based on your observations, I'm in favor of changing the default for new > minor releases going forward, and maybe some comments about this in the > online book. I'm also in favor of exposing configurable readahead for > STREAM reads, enabling easier experimentation for those keen to do so. > > Thanks, > Nick > > On Fri, Jun 2, 2023 at 9:42 AM Xiaolin Ha wrote: > > > +1 for disable readahead for pread > > > > Bryan Beaudreault 于2023年5月31日周三 20:44写道： > > > > > Hello team, > > > > > > I recently discovered "hbase.store.reader.no-readahead", which defaults > > to > > > false (so readahead is enabled). This only applies to PREAD reads, not > > > STREAM reads which always use readahead. When readahead is enabled, the > > > default readahead amount in the DFSClient is 4mb. In my opinion this is > > > extremely huge for HBase's use-case. > > > > > > Further, reads in HBase are always for a block at a time and blocks > > > typically have more than one row in them. So we are already reading > > ahead a > > > bit via block reads. And lastly, readahead is typically useful for > > > sequential read scenarios. It's unlikely for someone to do sequential > IO > > > via PREAD, instead they would use Scans (thus STREAM). In the case > where > > > someone is doing sequential IO via PREAD, they'd get some natural > > readahead > > > due to our reading of blocks at a time. > > > > > > I disabled readahead on about 50 servers across various clusters in our > > > production environment, and saw a massive (10x or more) drop in disk IO > > for > > > random read and mixed read cases. Scan workloads were mostly unaffected > > due > > > to not using this setting. I also did a targeted load test of a > cluster, > > > with and without readahead, and was able to get double the random read > > > throughput with it disabled. > > > > > > I'd like to update the default for this config to "true", thus > disabling > > > readahead for PREAD by default. I also think it's worth investigating > > > making readahead configurable for STREAM reads, perhaps based on the > > scan's > > > max result size or blockBytesScanned of the last next() call. > > > > > > Any objections to changing the default? > > > > > > https://issues.apache.org/jira/browse/HBASE-27896 > > > > > >
Re: [DISCUSS] Introduce new git repository for kustomize definitions (was: Kubernetes Orchestration for ZK, HDFS, and HBase)
Thanks everyone who has replied. If there are no further concerns raised over the weekend, I’ll get started on Monday with Infrastructure to provision the new repository. Thanks, Nick On Fri, 9 Jun 2023 at 18:08, Peter Somogyi wrote: > +1 on the new repository. > > On Wed, Jun 7, 2023 at 5:33 PM Viraj Jasani wrote: > > > +1 for hbase-kustomize > > > > On Mon, Jun 5, 2023 at 4:44 AM Nick Dimiduk wrote: > > > > > Hello, > > > > > > As I work through the integration of these kustomize definitions into > the > > > existing java project structure that is hbase-operator-tools, I'm > > > increasingly of the opinion that this is too much of a clash of > > concerns. I > > > think that this contribution would make better sense as its own > > repository > > > with its own release cycle. I'm neither aware of nor can I imagine a > > > technical coupling between the kustomize resources and the rest of the > > > utilities in operator-tools. Likewise, this change set introduces new > > > requirements (docker, buildx, KinD and/or minikube) to the build and > test > > > environment that are not otherwise needed by operator-tools. > > > > > > What do you think? SHould we request a new repository for the > > > kustomize files? I propose hbase-kustomize.git. > > > > > > Thanks, > > > Nick > > > > > > On Mon, May 22, 2023 at 4:33 PM Nick Dimiduk > > wrote: > > > > > > > I went ahead and rebuilt the Hadoop image module in the same style. I > > > > rebased the zookeeper-single and hdfs kustimize implementations onto > > the > > > > same structure. So, PR’s #118, #119, #120, and #121 are all in this > > > style. > > > > I don’t have a place for running integration tests, but unit tests > are > > > now > > > > running in Jenkins. > > > > > > > > I appreciate any attention you can provide. > > > > > > > > Thanks, > > > > Nick > > > > > > > > On Mon, 22 May 2023 at 16:02, TAK-LON WU wrote: > > > > > > > >> Sorry that I’m on vacation and will be back online after 06/06 , but > > > >> thanks > > > >> for putting the PR out and I believe someone on our side will review > > it > > > . > > > >> ( > > > >> or when I come back I will review them) > > > >> > > > >> -Stephen > > > >> > > > >> On Mon, May 22, 2023 at 6:08 PM Nick Dimiduk > > > wrote: > > > >> > > > >> > Heya team, > > > >> > > > > >> > I have rebuilt one of the early PRs so that the docker image build > > > >> pieces > > > >> > are integrated with the maven build. If this is acceptable to the > > > >> > reviewers, I'll go forward with integrating the other images and > > > >> > kustomize/kuttl tests in the same way. > > > >> > > > > >> > Please take a look. > > > >> > > > > >> > Thanks, > > > >> > Nick > > > >> > > > > >> > https://github.com/apache/hbase-operator-tools/pull/118 > > > >> > > > > >> > On Fri, May 12, 2023 at 4:19 PM Nick Dimiduk > > > > >> wrote: > > > >> > > > > >> > > Heya team, > > > >> > > > > > >> > > I have created individual pull-requests for each of the major > > > >> functional > > > >> > > pieces outlined in the initial branch. These await review. > > > >> > > > > > >> > > I've now started working integrating the test harness into the > > maven > > > >> > > build. After a brief detour for a Yetus plugin, I'm now looking > > > >> instead > > > >> > at > > > >> > > maven integration via exec-maven-plugin. I'm also investigating > > how > > > to > > > >> > pull > > > >> > > the container image build up into maven as well. > > > >> > > > > > >> > > Thanks, > > > >> > > Nic, > > > >> > > > > > >> > > On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk < > ndimi...@apache.org> > > > >> wrote: > > > >> > > > > > >> > >> Heya team, > > > >> > >> > > > >> > >> I've created a feature ticket [0] from which this contribution > > can > > > >> hang. > > > >> > >> I've created an associated release version [1] and feature > branch > > > [2] > > > >> > >> against which we can target PRs while things take shape. I've > > > >> published > > > >> > my > > > >> > >> initial extraction of this feature as a whole for your review > [3] > > > -- > > > >> > take a > > > >> > >> look at the big picture there. For each commit on that branch, > > I've > > > >> > created > > > >> > >> a sub-task on HBASE-27827. Probably reviewers will find other > > items > > > >> we > > > >> > need > > > >> > >> to peel off as sub-tasks. I'll start turning each of these > > commits > > > >> into > > > >> > PRs > > > >> > >> suitable for the Apache repo and your perusal. > > > >> > >> > > > >> > >> I think we're getting due for the 1.3 release of Operator > Tools, > > > so I > > > >> > >> expect this will land in 1.4. > > > >> > >> > > > >> > >> Thanks, > > > >> > >> Nick > > > >> > >> > > > >> > >> [0]: https://issues.apache.org/jira/browse/HBASE-27827 > > > >> > >> [1]: > > > >> https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199 > > > >> > >> [2]: > > > >> > >> > > > >> > > > > >> > > > > > >
Re: [DISCUSS] Introduce new git repository for kustomize definitions (was: Kubernetes Orchestration for ZK, HDFS, and HBase)
+1 on the new repository. On Wed, Jun 7, 2023 at 5:33 PM Viraj Jasani wrote: > +1 for hbase-kustomize > > On Mon, Jun 5, 2023 at 4:44 AM Nick Dimiduk wrote: > > > Hello, > > > > As I work through the integration of these kustomize definitions into the > > existing java project structure that is hbase-operator-tools, I'm > > increasingly of the opinion that this is too much of a clash of > concerns. I > > think that this contribution would make better sense as its own > repository > > with its own release cycle. I'm neither aware of nor can I imagine a > > technical coupling between the kustomize resources and the rest of the > > utilities in operator-tools. Likewise, this change set introduces new > > requirements (docker, buildx, KinD and/or minikube) to the build and test > > environment that are not otherwise needed by operator-tools. > > > > What do you think? SHould we request a new repository for the > > kustomize files? I propose hbase-kustomize.git. > > > > Thanks, > > Nick > > > > On Mon, May 22, 2023 at 4:33 PM Nick Dimiduk > wrote: > > > > > I went ahead and rebuilt the Hadoop image module in the same style. I > > > rebased the zookeeper-single and hdfs kustimize implementations onto > the > > > same structure. So, PR’s #118, #119, #120, and #121 are all in this > > style. > > > I don’t have a place for running integration tests, but unit tests are > > now > > > running in Jenkins. > > > > > > I appreciate any attention you can provide. > > > > > > Thanks, > > > Nick > > > > > > On Mon, 22 May 2023 at 16:02, TAK-LON WU wrote: > > > > > >> Sorry that I’m on vacation and will be back online after 06/06 , but > > >> thanks > > >> for putting the PR out and I believe someone on our side will review > it > > . > > >> ( > > >> or when I come back I will review them) > > >> > > >> -Stephen > > >> > > >> On Mon, May 22, 2023 at 6:08 PM Nick Dimiduk > > wrote: > > >> > > >> > Heya team, > > >> > > > >> > I have rebuilt one of the early PRs so that the docker image build > > >> pieces > > >> > are integrated with the maven build. If this is acceptable to the > > >> > reviewers, I'll go forward with integrating the other images and > > >> > kustomize/kuttl tests in the same way. > > >> > > > >> > Please take a look. > > >> > > > >> > Thanks, > > >> > Nick > > >> > > > >> > https://github.com/apache/hbase-operator-tools/pull/118 > > >> > > > >> > On Fri, May 12, 2023 at 4:19 PM Nick Dimiduk > > >> wrote: > > >> > > > >> > > Heya team, > > >> > > > > >> > > I have created individual pull-requests for each of the major > > >> functional > > >> > > pieces outlined in the initial branch. These await review. > > >> > > > > >> > > I've now started working integrating the test harness into the > maven > > >> > > build. After a brief detour for a Yetus plugin, I'm now looking > > >> instead > > >> > at > > >> > > maven integration via exec-maven-plugin. I'm also investigating > how > > to > > >> > pull > > >> > > the container image build up into maven as well. > > >> > > > > >> > > Thanks, > > >> > > Nic, > > >> > > > > >> > > On Thu, May 4, 2023 at 2:42 PM Nick Dimiduk > > >> wrote: > > >> > > > > >> > >> Heya team, > > >> > >> > > >> > >> I've created a feature ticket [0] from which this contribution > can > > >> hang. > > >> > >> I've created an associated release version [1] and feature branch > > [2] > > >> > >> against which we can target PRs while things take shape. I've > > >> published > > >> > my > > >> > >> initial extraction of this feature as a whole for your review [3] > > -- > > >> > take a > > >> > >> look at the big picture there. For each commit on that branch, > I've > > >> > created > > >> > >> a sub-task on HBASE-27827. Probably reviewers will find other > items > > >> we > > >> > need > > >> > >> to peel off as sub-tasks. I'll start turning each of these > commits > > >> into > > >> > PRs > > >> > >> suitable for the Apache repo and your perusal. > > >> > >> > > >> > >> I think we're getting due for the 1.3 release of Operator Tools, > > so I > > >> > >> expect this will land in 1.4. > > >> > >> > > >> > >> Thanks, > > >> > >> Nick > > >> > >> > > >> > >> [0]: https://issues.apache.org/jira/browse/HBASE-27827 > > >> > >> [1]: > > >> https://issues.apache.org/jira/browse/HBASE/fixforversion/12353199 > > >> > >> [2]: > > >> > >> > > >> > > > >> > > > https://github.com/apache/hbase-operator-tools/tree/HBASE-27827-kubernetes-deployment > > >> > >> [3]: > > >> > >> > > >> > > > >> > > > https://github.com/apache/hbase-operator-tools/compare/HBASE-27827-kubernetes-deployment...ndimiduk:hbase-operator-tools:HBASE-27827-kubernetes-deployment > > >> > >> > > >> > >> > > >> > >> > > >> > >> On Mon, Mar 13, 2023 at 11:28 AM Nick Dimiduk < > ndimi...@apache.org > > > > > >> > >> wrote: > > >> > >> > > >> > >>> Heya team, > > >> > >>> > > >> > >>> Over here at $dayjob, we have an increasing reliance on > Kubernetes > > >> for > > >> > >>> both development and production workloads. Our tools are
[jira] [Resolved] (HBASE-27907) [hbase-thirdparty] Include netty-tcnative in hbase-shaded-netty
[
https://issues.apache.org/jira/browse/HBASE-27907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Connell resolved HBASE-27907.
-
Resolution: Not A Problem
> [hbase-thirdparty] Include netty-tcnative in hbase-shaded-netty
> ---
>
> Key: HBASE-27907
> URL: https://issues.apache.org/jira/browse/HBASE-27907
> Project: HBase
> Issue Type: Improvement
> Components: thirdparty
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Minor
>
> At my company, HubSpot, we are starting to use HBase's TLS support in our
> clusters. I've done some experiments with different TLS implementations to
> see which has the best performance.
> I have tested Java 11 without {{{}UseAESCTRInstrinics{}}}, Java 11 with
> {{{}UseAESCTRInstrinsics{}}}, with and without {{netty-tcnative}} installed,
> on both {{x86_64}} and {{{}arm64{}}}. I will attach some charts.
> -I have found that using the {{netty-tcnative}} library gives the best
> performance overall, and I would like to see it included in HBase by default.-
> This ticket is for tracking the change to hbase-thirdparty and producing a
> new version of that. Another ticket will track the inclusion of that new
> version in HBase, which requires a few code changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27923) NettyRpcServer may hange if it should skip initial sasl handshake
chenglei created HBASE-27923: Summary: NettyRpcServer may hange if it should skip initial sasl handshake Key: HBASE-27923 URL: https://issues.apache.org/jira/browse/HBASE-27923 Project: HBase Issue Type: Bug Components: netty, security Affects Versions: 2.5.5, 3.0.0-alpha-4, 4.0.0-alpha-1 Reporter: chenglei -- This message was sent by Atlassian Jira (v8.20.10#820010)
