There is a possible memory leak problem, HBASE-27941, which is found when running ITBLL against branch-3.
Viraj reported that it also affects branch-2.x. Buy anyway, since it also affects branch-2.5, should not be a blocker for branch-2.6. Andrew Purtell <[email protected]> 于2023年8月31日周四 00:50写道： > > Are there any more important or serious issues with branch-2 which are not > resolved? > > Planning to cut branch-2.6 from branch-2 on 9/5 and stabilize it for a > 2.6.0 release from there. > > On Wed, Aug 23, 2023 at 7:41 PM 张铎(Duo Zhang) <[email protected]> wrote: > > > Let's get HBASE-28028 in too. Since Andrew is still on vacation, let > > me find another committer to help review the PR. > > > > Thanks. > > > > Bryan Beaudreault <[email protected]> 于2023年8月24日周四 02:23写道： > > > > > > We resolved HBASE-28010, so we should be good to go there now. > > > > > > On Sun, Aug 20, 2023 at 7:22 AM Bryan Beaudreault < > > [email protected]> > > > wrote: > > > > > > > I made > > > > https://issues.apache.org/jira/browse/HBASE-28010 a blocker so we > > don’t > > > > release a broken new feature. We will have it fixed early this week. > > > > > > > > On Fri, Aug 18, 2023 at 11:08 AM Andrew Purtell < > > [email protected]> > > > > wrote: > > > > > > > >> Let’s do it. We can use the new branch to stabilize for release. > > > >> > > > >> I am back from vacation in two weeks and can then lend time and AWS > > > >> resources for testing (and presumably fixing). > > > >> > > > >> > On Aug 18, 2023, at 10:53 AM, 张铎 <[email protected]> wrote: > > > >> > > > > >> > HBASE-27947 has been resolved. > > > >> > > > > >> > So I think now we are good to cut branch-2.6? > > > >> > > > > >> > Another good news is we are also close to fix the WAL value > > > >> > compression issue in HBASE-28028. > > > >> > > > > >> > Thanks. > > > >> > > > > >> > Bryan Beaudreault <[email protected]> 于2023年6月23日周五 02:47写道： > > > >> >> > > > >> >> Thanks! > > > >> >> > > > >> >> We're looking into one other emergent issue that we uncovered > > during > > > >> the > > > >> >> rollout of server side TLS on RegionServers. It seems > > > >> nettyDirectMemory has > > > >> >> increased substantially when under load with it enabled. Details in > > > >> >> https://issues.apache.org/jira/browse/HBASE-27947. > > > >> >> > > > >> >> > > > >> >>> On Thu, Jun 22, 2023 at 12:02 PM 张铎(Duo Zhang) < > > [email protected] > > > >> > > > > >> >>> wrote: > > > >> >>> > > > >> >>> PR is ready > > > >> >>> > > > >> >>> https://github.com/apache/hbase/pull/5305 > > > >> >>> > > > >> >>> PTAL. > > > >> >>> > > > >> >>> Thanks. > > > >> >>> > > > >> >>> 张铎(Duo Zhang) <[email protected]> 于2023年6月22日周四 21:40写道： > > > >> >>>> > > > >> >>>> Ah, missed your last comment on HBASE-27782. > > > >> >>>> > > > >> >>>> Let me take a look. > > > >> >>>> > > > >> >>>> Netty has some rules about how the exceptions are passed through > > the > > > >> >>>> pipeline(especially the order, forward or backward...) but > > honestly I > > > >> >>>> always forget it just a day later after I finished the code... > > > >> >>>> > > > >> >>>> Bryan Beaudreault <[email protected]> 于2023年6月17日周六 > > 00:43写道： > > > >> >>>>> > > > >> >>>>> In terms of TLS: > > > >> >>>>> > > > >> >>>>> - All of our clients (many thousands) in production are using > > the > > > >> >>>>> NettyRpcConnection with TLS enabled. However, these clients are > > > >> >>> currently > > > >> >>>>> connecting to the RegionServer/HMaster through an haproxy > > process > > > >> >>> local to > > > >> >>>>> each server which handles SSL termination. So not quite > > end-to-end > > > >> yet. > > > >> >>>>> - On the server side, most of our QA environment (a thousand > > > >> >>> regionservers > > > >> >>>>> and ~200 hmasters) are running it. So these are accepting TLS > > from > > > >> >>> clients > > > >> >>>>> and using TLS for intra-cluster communication. > > > >> >>>>> > > > >> >>>>> The migration is tricky for us due to the scale and the fact > > that we > > > >> >>> need > > > >> >>>>> to migrate off haproxy at the same time. Hopefully we should > > have > > > >> some > > > >> >>> of > > > >> >>>>> production running end-to-end TLS within the next month or so. > > > >> >>>>> > > > >> >>>>> From what we've seen in QA so far, there have not been any major > > > >> >>> issues. We > > > >> >>>>> also couldn't discern any performance issues in testing, though > > we > > > >> were > > > >> >>>>> comparing against our legacy haproxy setup and can't really > > compare > > > >> >>> against > > > >> >>>>> kerberos. > > > >> >>>>> > > > >> >>>>> One outstanding issue is > > > >> >>> https://issues.apache.org/jira/browse/HBASE-27782, > > > >> >>>>> which we still see periodically. It doesn't seem to cause actual > > > >> >>> issues, > > > >> >>>>> since the RpcClient still handles it gracefully, but it does > > cause > > > >> >>> noise > > > >> >>>>> and may have implications. > > > >> >>>>> > > > >> >>>>> On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) < > > > >> [email protected]> > > > >> >>>>> wrote: > > > >> >>>>> > > > >> >>>>>> So any updates here? > > > >> >>>>>> > > > >> >>>>>> Do we have any good news about the TLS usage in production so > > we > > > >> can > > > >> >>>>>> move forward on release 2.6.x? > > > >> >>>>>> > > > >> >>>>>> Thanks. > > > >> >>>>>> > > > >> >>>>>> Andrew Purtell <[email protected]> 于2023年4月7日周五 09:37写道： > > > >> >>>>>>> > > > >> >>>>>>> Agreed, that sounds like a good plan. > > > >> >>>>>>> > > > >> >>>>>>> On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) < > > > >> >>> [email protected]> > > > >> >>>>>> wrote: > > > >> >>>>>>> > > > >> >>>>>>>> I think we could follow the old pattern when we cut a new > > release > > > >> >>>>>> branch. > > > >> >>>>>>>> That is, after the new release branch is cut and the new > > minor > > > >> >>> release > > > >> >>>>>> is > > > >> >>>>>>>> out, we will do a final release of the oldest release line > > and > > > >> >>> then > > > >> >>>>>> mark it > > > >> >>>>>>>> as EOL. > > > >> >>>>>>>> > > > >> >>>>>>>> So here, I think once we cut branch-2.6 and release 2.6.0, we > > > >> >>> can do a > > > >> >>>>>>>> final release for 2.4.x and mark 2.4.x as EOL. > > > >> >>>>>>>> > > > >> >>>>>>>> Thanks. > > > >> >>>>>>>> > > > >> >>>>>>>> Bryan Beaudreault <[email protected]> 于2023年3月27日周一 > > > >> >>> 09:57写道： > > > >> >>>>>>>> > > > >> >>>>>>>>> Primary development on hbase-backup and TLS is complete. > > There > > > >> >>> are a > > > >> >>>>>>>> couple > > > >> >>>>>>>>> minor things I may want to add to TLS in the future, such as > > > >> >>>>>> pluggable > > > >> >>>>>>>> cert > > > >> >>>>>>>>> verification. But those are not needed for initial release > > IMO. > > > >> >>>>>>>>> > > > >> >>>>>>>>> We are almost ready integrating hbase-backup in production. > > > >> >>> We’ve > > > >> >>>>>> fixed a > > > >> >>>>>>>>> few minor things (all committed) but otherwise it’s worked > > > >> >>> well so > > > >> >>>>>> far in > > > >> >>>>>>>>> tests. > > > >> >>>>>>>>> > > > >> >>>>>>>>> We are a bit delayed in integrating TLS. I’m hopeful it will > > > >> >>> happen > > > >> >>>>>> in > > > >> >>>>>>>> the > > > >> >>>>>>>>> next 2-3 months. It’s a big project for us, so not quick, > > but > > > >> >>>>>> definitely > > > >> >>>>>>>> on > > > >> >>>>>>>>> the roadmap. > > > >> >>>>>>>>> > > > >> >>>>>>>>> It seems like cloudera may be closer to integrating TLS in > > > >> >>>>>> production. > > > >> >>>>>>>>> Balazs recently filed and fixed HBASE-27673 related to mTLS. > > > >> >>> Maybe > > > >> >>>>>> he can > > > >> >>>>>>>>> chime in on his status, or let me know if I am totally off > > > >> >>> base :) > > > >> >>>>>>>>> > > > >> >>>>>>>>> On Sun, Mar 26, 2023 at 9:25 PM Andrew Purtell < > > > >> >>>>>> [email protected] > > > >> >>>>>>>>> > > > >> >>>>>>>>> wrote: > > > >> >>>>>>>>> > > > >> >>>>>>>>>> Before we open a new code line should we discuss EOL of > > 2.4? > > > >> >>> After > > > >> >>>>>> the > > > >> >>>>>>>>>> first 2.6 release? It’s not required of course but cuts > > down > > > >> >>> the > > > >> >>>>>> amount > > > >> >>>>>>>>> of > > > >> >>>>>>>>>> labor to have two 2.x code lines (presumably, one as stable > > > >> >>> and > > > >> >>>>>> one as > > > >> >>>>>>>>>> next) rather than three. Perhaps even before that, should > > we > > > >> >>> move > > > >> >>>>>> the > > > >> >>>>>>>>>> stable pointer to the latest 2.5 release? > > > >> >>>>>>>>>> > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> On Mar 26, 2023, at 5:59 PM, 张铎 <[email protected]> > > > >> >>> wrote: > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> Bump. > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> I believe the mTLS and backup related code have all been > > > >> >>>>>> finished on > > > >> >>>>>>>>>>> branch-2? > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> Are there any other things which block us making the > > > >> >>> branch-2.6 > > > >> >>>>>>>> branch? > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> Thanks. > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>> Mallikarjun <[email protected]> 于2022年10月17日周一 > > > >> >>> 02:09写道： > > > >> >>>>>>>>>>> > > > >> >>>>>>>>>>>> On hbase-backup, we are using in production for more then > > > >> >>> 1 > > > >> >>>>>> year. I > > > >> >>>>>>>>> can > > > >> >>>>>>>>>>>> vouch for it to be stable enough to be in a release > > > >> >>> version so > > > >> >>>>>> that > > > >> >>>>>>>>> more > > > >> >>>>>>>>>>>> people can use it and polished it further. > > > >> >>>>>>>>>>>> > > > >> >>>>>>>>>>>>> On Sun, Oct 16, 2022, 11:25 PM Andrew Purtell < > > > >> >>>>>>>>>> [email protected]> > > > >> >>>>>>>>>>>>> wrote: > > > >> >>>>>>>>>>>>> > > > >> >>>>>>>>>>>>> My understanding is some folks evaluating and polishing > > > >> >>> TLS for > > > >> >>>>>>>> their > > > >> >>>>>>>>>>>>> production are also considering hbase-backup in the same > > > >> >>> way, > > > >> >>>>>> which > > > >> >>>>>>>>> is > > > >> >>>>>>>>>>>> why > > > >> >>>>>>>>>>>>> I linked them together. If that is incorrect then they > > > >> >>> both are > > > >> >>>>>>>> still > > > >> >>>>>>>>>>>> worth > > > >> >>>>>>>>>>>>> considering in my opinion but would have a more tenuous > > > >> >>> link. > > > >> >>>>>>>>>>>>> > > > >> >>>>>>>>>>>>> Where we are with hbase-backup is it should probably be > > > >> >>> ported > > > >> >>>>>> to > > > >> >>>>>>>>> where > > > >> >>>>>>>>>>>>> more people would be inclined to evaluate it, in order > > > >> >>> for it > > > >> >>>>>> to > > > >> >>>>>>>> make > > > >> >>>>>>>>>>>> more > > > >> >>>>>>>>>>>>> progress. A new minor releasing line would fit. On the > > > >> >>> other > > > >> >>>>>> hand > > > >> >>>>>>>> if > > > >> >>>>>>>>> it > > > >> >>>>>>>>>>>> is > > > >> >>>>>>>>>>>>> too unpolished then the experience would be poor. > > > >> >>>>>>>>>>>>> > > > >> >>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>> On Oct 16, 2022, at 5:35 AM, 张铎 <[email protected] > > > > > > >> >>>>>> wrote: > > > >> >>>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>> I believe the second one is still ongoing? > > > >> >>>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>> Andrew Purtell <[email protected]> 于2022年10月14日周五 > > > >> >>> 05:37写道： > > > >> >>>>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>>> We will begin releasing activity for the 2.6 code line > > > >> >>> and > > > >> >>>>>> as a > > > >> >>>>>>>>>>>>>>> prerequisite to that we shall need to make a new > > branch > > > >> >>>>>>>> branch-2.6 > > > >> >>>>>>>>>>>> from > > > >> >>>>>>>>>>>>>>> branch-2. > > > >> >>>>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>>> Before we do that let's make sure all commits for the > > > >> >>> key > > > >> >>>>>>>> features > > > >> >>>>>>>>> of > > > >> >>>>>>>>>>>>> 2.6 > > > >> >>>>>>>>>>>>>>> are settled in branch-2 before the branching point. > > > >> >>> Those key > > > >> >>>>>>>>>> features > > > >> >>>>>>>>>>>>> are: > > > >> >>>>>>>>>>>>>>> - mTLS RPC > > > >> >>>>>>>>>>>>>>> - hbase-backup backport > > > >> >>>>>>>>>>>>>>> > > > >> >>>>>>>>>>>>>>> -- > > > >> >>>>>>>>>>>>>>> Best regards, > > > >> >>>>>>>>>>>>>>> Andrew > > > >> >>>>>>>>>>>>> > > > >> >>>>>>>>>>>> > > > >> >>>>>>>>>> > > > >> >>>>>>>>> > > > >> >>>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> > > > >> >>>>>>> -- > > > >> >>>>>>> Best regards, > > > >> >>>>>>> Andrew > > > >> >>>>>>> > > > >> >>>>>>> Unrest, ignorance distilled, nihilistic imbeciles - > > > >> >>>>>>> It's what we’ve earned > > > >> >>>>>>> Welcome, apocalypse, what’s taken you so long? > > > >> >>>>>>> Bring us the fitting end that we’ve been counting on > > > >> >>>>>>> - A23, Welcome, Apocalypse > > > >> >>>>>> > > > >> >>> > > > >> > > > > > > > > > -- > Best regards, > Andrew > > Unrest, ignorance distilled, nihilistic imbeciles - > It's what we’ve earned > Welcome, apocalypse, what’s taken you so long? > Bring us the fitting end that we’ve been counting on > - A23, Welcome, Apocalypse
