I have seen this leak too, in testing with 2.5. I don't think it is a blocker for branching, but yes I would prefer a solution committed to branch-2.6 before a 2.6.0 release, if that can be possible.
On Wed, Aug 30, 2023 at 7:43 PM 张铎(Duo Zhang) <[email protected]> wrote: > There is a possible memory leak problem, HBASE-27941, which is found > when running ITBLL against branch-3. > > Viraj reported that it also affects branch-2.x. > > Buy anyway, since it also affects branch-2.5, should not be a blocker > for branch-2.6. > > Andrew Purtell <[email protected]> 于2023年8月31日周四 00:50写道： > > > > Are there any more important or serious issues with branch-2 which are > not > > resolved? > > > > Planning to cut branch-2.6 from branch-2 on 9/5 and stabilize it for a > > 2.6.0 release from there. > > > > On Wed, Aug 23, 2023 at 7:41 PM 张铎(Duo Zhang) <[email protected]> > wrote: > > > > > Let's get HBASE-28028 in too. Since Andrew is still on vacation, let > > > me find another committer to help review the PR. > > > > > > Thanks. > > > > > > Bryan Beaudreault <[email protected]> 于2023年8月24日周四 02:23写道： > > > > > > > > We resolved HBASE-28010, so we should be good to go there now. > > > > > > > > On Sun, Aug 20, 2023 at 7:22 AM Bryan Beaudreault < > > > [email protected]> > > > > wrote: > > > > > > > > > I made > > > > > https://issues.apache.org/jira/browse/HBASE-28010 a blocker so we > > > don’t > > > > > release a broken new feature. We will have it fixed early this > week. > > > > > > > > > > On Fri, Aug 18, 2023 at 11:08 AM Andrew Purtell < > > > [email protected]> > > > > > wrote: > > > > > > > > > >> Let’s do it. We can use the new branch to stabilize for release. > > > > >> > > > > >> I am back from vacation in two weeks and can then lend time and > AWS > > > > >> resources for testing (and presumably fixing). > > > > >> > > > > >> > On Aug 18, 2023, at 10:53 AM, 张铎 <[email protected]> wrote: > > > > >> > > > > > >> > HBASE-27947 has been resolved. > > > > >> > > > > > >> > So I think now we are good to cut branch-2.6? > > > > >> > > > > > >> > Another good news is we are also close to fix the WAL value > > > > >> > compression issue in HBASE-28028. > > > > >> > > > > > >> > Thanks. > > > > >> > > > > > >> > Bryan Beaudreault <[email protected]> 于2023年6月23日周五 > 02:47写道： > > > > >> >> > > > > >> >> Thanks! > > > > >> >> > > > > >> >> We're looking into one other emergent issue that we uncovered > > > during > > > > >> the > > > > >> >> rollout of server side TLS on RegionServers. It seems > > > > >> nettyDirectMemory has > > > > >> >> increased substantially when under load with it enabled. > Details in > > > > >> >> https://issues.apache.org/jira/browse/HBASE-27947. > > > > >> >> > > > > >> >> > > > > >> >>> On Thu, Jun 22, 2023 at 12:02 PM 张铎(Duo Zhang) < > > > [email protected] > > > > >> > > > > > >> >>> wrote: > > > > >> >>> > > > > >> >>> PR is ready > > > > >> >>> > > > > >> >>> https://github.com/apache/hbase/pull/5305 > > > > >> >>> > > > > >> >>> PTAL. > > > > >> >>> > > > > >> >>> Thanks. > > > > >> >>> > > > > >> >>> 张铎(Duo Zhang) <[email protected]> 于2023年6月22日周四 21:40写道： > > > > >> >>>> > > > > >> >>>> Ah, missed your last comment on HBASE-27782. > > > > >> >>>> > > > > >> >>>> Let me take a look. > > > > >> >>>> > > > > >> >>>> Netty has some rules about how the exceptions are passed > through > > > the > > > > >> >>>> pipeline(especially the order, forward or backward...) but > > > honestly I > > > > >> >>>> always forget it just a day later after I finished the > code... > > > > >> >>>> > > > > >> >>>> Bryan Beaudreault <[email protected]> 于2023年6月17日周六 > > > 00:43写道： > > > > >> >>>>> > > > > >> >>>>> In terms of TLS: > > > > >> >>>>> > > > > >> >>>>> - All of our clients (many thousands) in production are > using > > > the > > > > >> >>>>> NettyRpcConnection with TLS enabled. However, these clients > are > > > > >> >>> currently > > > > >> >>>>> connecting to the RegionServer/HMaster through an haproxy > > > process > > > > >> >>> local to > > > > >> >>>>> each server which handles SSL termination. So not quite > > > end-to-end > > > > >> yet. > > > > >> >>>>> - On the server side, most of our QA environment (a thousand > > > > >> >>> regionservers > > > > >> >>>>> and ~200 hmasters) are running it. So these are accepting > TLS > > > from > > > > >> >>> clients > > > > >> >>>>> and using TLS for intra-cluster communication. > > > > >> >>>>> > > > > >> >>>>> The migration is tricky for us due to the scale and the fact > > > that we > > > > >> >>> need > > > > >> >>>>> to migrate off haproxy at the same time. Hopefully we should > > > have > > > > >> some > > > > >> >>> of > > > > >> >>>>> production running end-to-end TLS within the next month or > so. > > > > >> >>>>> > > > > >> >>>>> From what we've seen in QA so far, there have not been any > major > > > > >> >>> issues. We > > > > >> >>>>> also couldn't discern any performance issues in testing, > though > > > we > > > > >> were > > > > >> >>>>> comparing against our legacy haproxy setup and can't really > > > compare > > > > >> >>> against > > > > >> >>>>> kerberos. > > > > >> >>>>> > > > > >> >>>>> One outstanding issue is > > > > >> >>> https://issues.apache.org/jira/browse/HBASE-27782, > > > > >> >>>>> which we still see periodically. It doesn't seem to cause > actual > > > > >> >>> issues, > > > > >> >>>>> since the RpcClient still handles it gracefully, but it does > > > cause > > > > >> >>> noise > > > > >> >>>>> and may have implications. > > > > >> >>>>> > > > > >> >>>>> On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) < > > > > >> [email protected]> > > > > >> >>>>> wrote: > > > > >> >>>>> > > > > >> >>>>>> So any updates here? > > > > >> >>>>>> > > > > >> >>>>>> Do we have any good news about the TLS usage in production > so > > > we > > > > >> can > > > > >> >>>>>> move forward on release 2.6.x? > > > > >> >>>>>> > > > > >> >>>>>> Thanks. > > > > >> >>>>>> > > > > >> >>>>>> Andrew Purtell <[email protected]> 于2023年4月7日周五 09:37写道： > > > > >> >>>>>>> > > > > >> >>>>>>> Agreed, that sounds like a good plan. > > > > >> >>>>>>> > > > > >> >>>>>>> On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) < > > > > >> >>> [email protected]> > > > > >> >>>>>> wrote: > > > > >> >>>>>>> > > > > >> >>>>>>>> I think we could follow the old pattern when we cut a new > > > release > > > > >> >>>>>> branch. > > > > >> >>>>>>>> That is, after the new release branch is cut and the new > > > minor > > > > >> >>> release > > > > >> >>>>>> is > > > > >> >>>>>>>> out, we will do a final release of the oldest release > line > > > and > > > > >> >>> then > > > > >> >>>>>> mark it > > > > >> >>>>>>>> as EOL. > > > > >> >>>>>>>> > > > > >> >>>>>>>> So here, I think once we cut branch-2.6 and release > 2.6.0, we > > > > >> >>> can do a > > > > >> >>>>>>>> final release for 2.4.x and mark 2.4.x as EOL. > > > > >> >>>>>>>> > > > > >> >>>>>>>> Thanks. > > > > >> >>>>>>>> > > > > >> >>>>>>>> Bryan Beaudreault <[email protected]> > 于2023年3月27日周一 > > > > >> >>> 09:57写道： > > > > >> >>>>>>>> > > > > >> >>>>>>>>> Primary development on hbase-backup and TLS is complete. > > > There > > > > >> >>> are a > > > > >> >>>>>>>> couple > > > > >> >>>>>>>>> minor things I may want to add to TLS in the future, > such as > > > > >> >>>>>> pluggable > > > > >> >>>>>>>> cert > > > > >> >>>>>>>>> verification. But those are not needed for initial > release > > > IMO. > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> We are almost ready integrating hbase-backup in > production. > > > > >> >>> We’ve > > > > >> >>>>>> fixed a > > > > >> >>>>>>>>> few minor things (all committed) but otherwise it’s > worked > > > > >> >>> well so > > > > >> >>>>>> far in > > > > >> >>>>>>>>> tests. > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> We are a bit delayed in integrating TLS. I’m hopeful it > will > > > > >> >>> happen > > > > >> >>>>>> in > > > > >> >>>>>>>> the > > > > >> >>>>>>>>> next 2-3 months. It’s a big project for us, so not > quick, > > > but > > > > >> >>>>>> definitely > > > > >> >>>>>>>> on > > > > >> >>>>>>>>> the roadmap. > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> It seems like cloudera may be closer to integrating TLS > in > > > > >> >>>>>> production. > > > > >> >>>>>>>>> Balazs recently filed and fixed HBASE-27673 related to > mTLS. > > > > >> >>> Maybe > > > > >> >>>>>> he can > > > > >> >>>>>>>>> chime in on his status, or let me know if I am totally > off > > > > >> >>> base :) > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> On Sun, Mar 26, 2023 at 9:25 PM Andrew Purtell < > > > > >> >>>>>> [email protected] > > > > >> >>>>>>>>> > > > > >> >>>>>>>>> wrote: > > > > >> >>>>>>>>> > > > > >> >>>>>>>>>> Before we open a new code line should we discuss EOL of > > > 2.4? > > > > >> >>> After > > > > >> >>>>>> the > > > > >> >>>>>>>>>> first 2.6 release? It’s not required of course but cuts > > > down > > > > >> >>> the > > > > >> >>>>>> amount > > > > >> >>>>>>>>> of > > > > >> >>>>>>>>>> labor to have two 2.x code lines (presumably, one as > stable > > > > >> >>> and > > > > >> >>>>>> one as > > > > >> >>>>>>>>>> next) rather than three. Perhaps even before that, > should > > > we > > > > >> >>> move > > > > >> >>>>>> the > > > > >> >>>>>>>>>> stable pointer to the latest 2.5 release? > > > > >> >>>>>>>>>> > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> On Mar 26, 2023, at 5:59 PM, 张铎 < > [email protected]> > > > > >> >>> wrote: > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> Bump. > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> I believe the mTLS and backup related code have all > been > > > > >> >>>>>> finished on > > > > >> >>>>>>>>>>> branch-2? > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> Are there any other things which block us making the > > > > >> >>> branch-2.6 > > > > >> >>>>>>>> branch? > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> Thanks. > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>> Mallikarjun <[email protected]> 于2022年10月17日周一 > > > > >> >>> 02:09写道： > > > > >> >>>>>>>>>>> > > > > >> >>>>>>>>>>>> On hbase-backup, we are using in production for more > then > > > > >> >>> 1 > > > > >> >>>>>> year. I > > > > >> >>>>>>>>> can > > > > >> >>>>>>>>>>>> vouch for it to be stable enough to be in a release > > > > >> >>> version so > > > > >> >>>>>> that > > > > >> >>>>>>>>> more > > > > >> >>>>>>>>>>>> people can use it and polished it further. > > > > >> >>>>>>>>>>>> > > > > >> >>>>>>>>>>>>> On Sun, Oct 16, 2022, 11:25 PM Andrew Purtell < > > > > >> >>>>>>>>>> [email protected]> > > > > >> >>>>>>>>>>>>> wrote: > > > > >> >>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>> My understanding is some folks evaluating and > polishing > > > > >> >>> TLS for > > > > >> >>>>>>>> their > > > > >> >>>>>>>>>>>>> production are also considering hbase-backup in the > same > > > > >> >>> way, > > > > >> >>>>>> which > > > > >> >>>>>>>>> is > > > > >> >>>>>>>>>>>> why > > > > >> >>>>>>>>>>>>> I linked them together. If that is incorrect then > they > > > > >> >>> both are > > > > >> >>>>>>>> still > > > > >> >>>>>>>>>>>> worth > > > > >> >>>>>>>>>>>>> considering in my opinion but would have a more > tenuous > > > > >> >>> link. > > > > >> >>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>> Where we are with hbase-backup is it should > probably be > > > > >> >>> ported > > > > >> >>>>>> to > > > > >> >>>>>>>>> where > > > > >> >>>>>>>>>>>>> more people would be inclined to evaluate it, in > order > > > > >> >>> for it > > > > >> >>>>>> to > > > > >> >>>>>>>> make > > > > >> >>>>>>>>>>>> more > > > > >> >>>>>>>>>>>>> progress. A new minor releasing line would fit. On > the > > > > >> >>> other > > > > >> >>>>>> hand > > > > >> >>>>>>>> if > > > > >> >>>>>>>>> it > > > > >> >>>>>>>>>>>> is > > > > >> >>>>>>>>>>>>> too unpolished then the experience would be poor. > > > > >> >>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>> On Oct 16, 2022, at 5:35 AM, 张铎 < > [email protected] > > > > > > > > >> >>>>>> wrote: > > > > >> >>>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>> I believe the second one is still ongoing? > > > > >> >>>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>> Andrew Purtell <[email protected]> > 于2022年10月14日周五 > > > > >> >>> 05:37写道： > > > > >> >>>>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>>> We will begin releasing activity for the 2.6 code > line > > > > >> >>> and > > > > >> >>>>>> as a > > > > >> >>>>>>>>>>>>>>> prerequisite to that we shall need to make a new > > > branch > > > > >> >>>>>>>> branch-2.6 > > > > >> >>>>>>>>>>>> from > > > > >> >>>>>>>>>>>>>>> branch-2. > > > > >> >>>>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>>> Before we do that let's make sure all commits for > the > > > > >> >>> key > > > > >> >>>>>>>> features > > > > >> >>>>>>>>> of > > > > >> >>>>>>>>>>>>> 2.6 > > > > >> >>>>>>>>>>>>>>> are settled in branch-2 before the branching > point. > > > > >> >>> Those key > > > > >> >>>>>>>>>> features > > > > >> >>>>>>>>>>>>> are: > > > > >> >>>>>>>>>>>>>>> - mTLS RPC > > > > >> >>>>>>>>>>>>>>> - hbase-backup backport > > > > >> >>>>>>>>>>>>>>> > > > > >> >>>>>>>>>>>>>>> -- > > > > >> >>>>>>>>>>>>>>> Best regards, > > > > >> >>>>>>>>>>>>>>> Andrew > > > > >> >>>>>>>>>>>>> > > > > >> >>>>>>>>>>>> > > > > >> >>>>>>>>>> > > > > >> >>>>>>>>> > > > > >> >>>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> > > > > >> >>>>>>> -- > > > > >> >>>>>>> Best regards, > > > > >> >>>>>>> Andrew > > > > >> >>>>>>> > > > > >> >>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - > > > > >> >>>>>>> It's what we’ve earned > > > > >> >>>>>>> Welcome, apocalypse, what’s taken you so long? > > > > >> >>>>>>> Bring us the fitting end that we’ve been counting on > > > > >> >>>>>>> - A23, Welcome, Apocalypse > > > > >> >>>>>> > > > > >> >>> > > > > >> > > > > > > > > > > > > > > -- > > Best regards, > > Andrew > > > > Unrest, ignorance distilled, nihilistic imbeciles - > > It's what we’ve earned > > Welcome, apocalypse, what’s taken you so long? > > Bring us the fitting end that we’ve been counting on > > - A23, Welcome, Apocalypse > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
