2.1 update for July and August. 7 PRs in progress. 1,253 marked as DONE. (+87) 6 left marked as a Blocker for 2.1. [1] 2 left as TODO for Compaction follow on work. [2] 1 left as TODO for ZK follow on work. [3] Once #2197 is completed and merged, there will be some follow on work.
[1]: https://github.com/apache/accumulo/issues?q=is%3Aissue%20is%3Aopen%20label%3Ablocker%20project%3Aapache%2Faccumulo%2F3 [2]: https://github.com/apache/accumulo/projects/13 [3]: https://github.com/apache/accumulo/projects/24 On Wed, Jun 29, 2022 at 9:11 AM Mike Miller <mmil...@apache.org> wrote: > It has been a month so I am sending a 2.1 update. > 10 PRs in progress. > 1,166 marked as DONE. (+36) > 33 left in the TODO state. I went through these and dropped some from 2.1. > https://github.com/apache/accumulo/projects/3 > > 3 marked as TODO as Compaction follow on work: > https://github.com/apache/accumulo/projects/13 > > 2 in progress & 2 TODO for the Single Node ZK follow on work. > https://github.com/apache/accumulo/projects/24 > > > On Fri, May 27, 2022 at 11:30 AM Mike Miller <mmil...@apache.org> wrote: > >> Update on 2.1 progress. Come on Folks, let's Hold The Line. [1] >> 14 Pull requests in progress. (-7 digression from last week) >> 1,130 marked as DONE. (+19) >> https://github.com/apache/accumulo/projects/3 >> >> 3 marked as TODO as Compaction follow on work: >> https://github.com/apache/accumulo/projects/13 >> >> 2 in progress & 2 TODO for the Single Node ZK follow on work. >> https://github.com/apache/accumulo/projects/24 >> >> [1]: https://www.youtube.com/watch?v=htgr3pvBr-I >> >> On Fri, May 20, 2022 at 11:25 AM Mike Miller <mmil...@apache.org> wrote: >> >>> Update on 2.1 progress. >>> 7 Pull requests in progress. >>> 1111 marked as DONE. ( I just missed sending this at 11:11 EST) >>> https://github.com/apache/accumulo/projects/3 >>> >>> 3 marked as TODO as Compaction follow on work: >>> https://github.com/apache/accumulo/projects/13 >>> >>> 2 marked as TODO for the Single Node ZK follow on work. >>> https://github.com/apache/accumulo/projects/24 >>> >>> >>> >>> On Fri, May 13, 2022 at 9:38 AM Mike Miller <mmil...@apache.org> wrote: >>> >>>> Update on 2.1 progress. >>>> 10 Pull Requests in progress. >>>> 1,097 marked as DONE. >>>> https://github.com/apache/accumulo/projects/3 >>>> >>>> 3 Tickets marked as TODO as Compaction follow on work: >>>> https://github.com/apache/accumulo/projects/13 >>>> >>>> Only 1 Ticket marked as TODO for the ZK follow on work. I thought there >>>> would be more here: >>>> https://github.com/apache/accumulo/projects/24 >>>> >>>> On Fri, Apr 8, 2022 at 7:00 AM Mike Miller <mmil...@apache.org> wrote: >>>> >>>>> Update on 2.1 progress. For pull requests: >>>>> 15 currently in progress. >>>>> 32 are open as TODO. But a lot of these will get bumped to the next >>>>> version. >>>>> 1,025 DONE. Wow! Good work everyone. >>>>> >>>>> https://github.com/apache/accumulo/projects/3 >>>>> >>>>> On Wed, Apr 6, 2022 at 4:55 PM Christopher <ctubb...@apache.org> >>>>> wrote: >>>>> >>>>>> After some additional consideration, and getting a better >>>>>> understanding of >>>>>> how the code is expected to work from discussing it with Dave... I'm a >>>>>> little more inclined to support #2422 in 2.1, provided: >>>>>> >>>>>> 1. There's time for me to review it, >>>>>> 2. It is sufficiently decoupled from the existing code and marked >>>>>> experimental, so that we have the flexibility to alter its design, if >>>>>> it >>>>>> seems appropriate after it gets some exposure after the release, >>>>>> 3. Unit tests and integration tests are reliably passing (as stable >>>>>> as, or >>>>>> more stable than, they are currently), >>>>>> 4. No serious issues are discovered during review, and >>>>>> 5. It doesn't delay a release past early June, as I think this is a >>>>>> reasonable target date. >>>>>> >>>>>> This my wishlist before I can get behind it with a +1 for 2.1. If >>>>>> these >>>>>> aren't met, I do not intend to veto, but I'd be a -0 on its inclusion >>>>>> to >>>>>> 2.1. Of course, once I review it, my thoughts may change a bit. >>>>>> >>>>>> On Mon, Apr 4, 2022 at 7:07 PM Mike Miller <mmil...@apache.org> >>>>>> wrote: >>>>>> >>>>>> > I think I can finish the FATE refactor PR [1] for 2.1. I had been >>>>>> keeping >>>>>> > it up to date with the latest in main but stopped because it was >>>>>> too much >>>>>> > work. I was waiting until the ZK property changes are completed >>>>>> before >>>>>> > resolving the latest conflicts. I don't think it is much of a risk. >>>>>> It is >>>>>> > mostly cleanup and refactoring to remove generics from the >>>>>> serialization >>>>>> > code. It will be some work to revisit but I think the risk is >>>>>> pretty low. >>>>>> > It would allow changing the serialization, which we may be able to >>>>>> get into >>>>>> > 2.1 as well. >>>>>> > >>>>>> > [1] https://github.com/apache/accumulo/pull/2475 >>>>>> > >>>>>> > On Mon, Apr 4, 2022 at 11:50 AM Keith Turner <ke...@deenlo.com> >>>>>> wrote: >>>>>> > >>>>>> > > On Mon, Apr 4, 2022 at 11:17 AM Christopher <ctubb...@apache.org> >>>>>> wrote: >>>>>> > > > >>>>>> > > > I haven't seen the metrics test fail very often lately. If it's >>>>>> > stable, I >>>>>> > > > don't mind removing the blocker on that issue, but I'd be >>>>>> reluctant to >>>>>> > > > close it entirely just yet, until we can verify it doesn't >>>>>> happen >>>>>> > > anymore. >>>>>> > > > >>>>>> > > > As for the original list of potential issues to include, I'm in >>>>>> favor >>>>>> > of >>>>>> > > > trying to get #2197 in. It was started awhile ago, is >>>>>> relatively simple >>>>>> > > and >>>>>> > > > well understood by several of us already... it just needs a bit >>>>>> of >>>>>> > > > attention to finalize reviews so it can be merged. >>>>>> > > > >>>>>> > > > However, I'm reluctant to include #2422, because I don't think >>>>>> it's >>>>>> > near >>>>>> > > > ready enough, and by the time it is, it will be very last >>>>>> minute, and I >>>>>> > > > don't want to delay 2.1 further for it. Even if it's included >>>>>> as an >>>>>> > > > experimental feature, I think it has huge potential to be >>>>>> disruptive, >>>>>> > or >>>>>> > > to >>>>>> > > > have a lot of churn by the time people actually have a chance >>>>>> to review >>>>>> > > it >>>>>> > > > thoroughly. Furthermore, I think there are possible >>>>>> alternatives (like >>>>>> > a >>>>>> > > > fully client-side implementation, based on offline scanners) >>>>>> that would >>>>>> > > > avoid the tight coupling of a new service to Accumulo's core >>>>>> code. This >>>>>> > > >>>>>> > > There are some advantages to scan servers over direct file access >>>>>> to >>>>>> > > consider. One is scalability of computation, if a web server is >>>>>> > > serving N client queries with scan servers those can potentially >>>>>> go to >>>>>> > > different scan servers. With direct file access, all N queries >>>>>> and >>>>>> > > their iterator stacks would have to run in the web server. >>>>>> Another is >>>>>> > > scalability of caching/memory. When web servers send queries to >>>>>> scan >>>>>> > > servers using a sticky algorithm for assigning tablets to groups >>>>>> of >>>>>> > > scan servers, it could lead to good cache utilization and sharing >>>>>> that >>>>>> > > may not be possible when running scans directly in the web >>>>>> server. So >>>>>> > > scan servers allow scaling cache and computations for queries >>>>>> > > independently of web servers in way that may not be possible with >>>>>> > > direct file access. >>>>>> > > >>>>>> > > Another advantage to consider is isolation. With direct file >>>>>> access >>>>>> > > and queries running directly in a web server, a bad query could >>>>>> bring >>>>>> > > down a web server and lots of unrelated queries. Having a bad >>>>>> query >>>>>> > > bring down a scan server may be less disruptive. >>>>>> > > >>>>>> > > > thread isn't for discussing this in depth, so we can have that >>>>>> > discussion >>>>>> > > > in a separate thread, but I'm generally opposed to including it >>>>>> this >>>>>> > late >>>>>> > > > in 2.1's development, given the timing, size and scope, tight >>>>>> coupling, >>>>>> > > and >>>>>> > > > current state. >>>>>> > > > >>>>>> > > > I don't know enough about #2475 to have a strong opinion, but >>>>>> it looks >>>>>> > > big, >>>>>> > > > and possibly high-risk, given the critical code it touches. It >>>>>> > currently >>>>>> > > > has a substantial number of conflicts with the main branch. >>>>>> However, I >>>>>> > > was >>>>>> > > > thinking that *some* minimal refactoring (like low-risk >>>>>> automatic >>>>>> > > > refactoring, like moving packages) could be done. So, if that's >>>>>> all >>>>>> > this >>>>>> > > > does, it might be okay. Otherwise, maybe it can be simplified? >>>>>> At the >>>>>> > > very >>>>>> > > > least, I was thinking it would be a good opportunity to move the >>>>>> > > > `org.apache.accumulo.fate` packages into an appropriate >>>>>> > > > `org.apache.accumulo.core` parent package (some would go to >>>>>> > > o.a.a.core.fate >>>>>> > > > and others might go to o.a.a.core.util or similar) to keep the >>>>>> package >>>>>> > > > namespaces standardized, which is helpful to avoid naming >>>>>> collisions >>>>>> > and >>>>>> > > > jar sealing issues, as well as for less complicated jigsaw >>>>>> module >>>>>> > > > definitions in future. Since 2.1 FaTE is already incompatible >>>>>> with >>>>>> > prior >>>>>> > > > versions, a rename at this time would be less disruptive. >>>>>> > > > >>>>>> > > > Another task I had wanted to be done for 2.1, before I got >>>>>> distracted >>>>>> > > > fixing test failures during and after Christmas and trying to >>>>>> work >>>>>> > > through >>>>>> > > > the singleton manager zookeeper stuff to see what we could >>>>>> simplify. >>>>>> > > What I >>>>>> > > > had wanted done was to standardize the way we pass table >>>>>> identifiers >>>>>> > > (name, >>>>>> > > > IDs) across the RPC layer, since we currently do that >>>>>> inconsistently. I >>>>>> > > > don't remember if there's an existing ticket open for it, but I >>>>>> have a >>>>>> > > > working branch I had started working out of for it before >>>>>> Christmas. >>>>>> > It's >>>>>> > > > relatively simple work, and would set us up for some much >>>>>> better APIs >>>>>> > > going >>>>>> > > > forward, as well as help with logging information about table >>>>>> actions. >>>>>> > If >>>>>> > > > necessary, it could be bumped to a future version, but then >>>>>> we'd have >>>>>> > > more >>>>>> > > > churn in the thrift layer. So, I'd prefer to get it for 2.1 to >>>>>> avoid >>>>>> > > that. >>>>>> > > > >>>>>> > > > As for planning, I was thinking early May for a code freeze >>>>>> (except bug >>>>>> > > > fixes and small improvements found during testing), so we can >>>>>> try to >>>>>> > > > release towards the end of May/early June. If we go with that >>>>>> timeline, >>>>>> > > > that's not a lot of time to wrap up features and have time for >>>>>> > > > review/testing, so we may need to be selective about what we >>>>>> hold off >>>>>> > > until >>>>>> > > > the next version, unless we want to further delay 2.1. >>>>>> > > > >>>>>> > > > >>>>>> > > > On Mon, Apr 4, 2022 at 9:13 AM Dave Marion <dmario...@gmail.com >>>>>> > >>>>>> > wrote: >>>>>> > > > >>>>>> > > > > I think [3] is OBE and can be closed. >>>>>> > > > > >>>>>> > > > > On Mon, Apr 4, 2022 at 9:11 AM Mike Miller < >>>>>> mmil...@apache.org> >>>>>> > wrote: >>>>>> > > > > >>>>>> > > > > > Yes I agree, that was the goal of this email thread. I >>>>>> found a few >>>>>> > > more >>>>>> > > > > > tickets that should be addressed for the next release. >>>>>> > > > > > >>>>>> > > > > > Ivan - There was some work done on this PR but it has been >>>>>> some >>>>>> > > time. Do >>>>>> > > > > > you want to take a look at it? Implement a Thread limit. [1] >>>>>> > > > > > Keith T - I think we should get this one merged to fix that >>>>>> > > consistency >>>>>> > > > > > check bug I found. It looks like it is finished. [2] >>>>>> > > > > > Dave & Dom - Were you guys able to figure out a fix for the >>>>>> new >>>>>> > > external >>>>>> > > > > > compaction metrics test? [3] >>>>>> > > > > > >>>>>> > > > > > FYI we have 6 blockers for 2.1: >>>>>> > > > > > https://github.com/apache/accumulo/labels/blocker >>>>>> > > > > > >>>>>> > > > > > This is almost definitely going into 2.1 [4]. Thanks Jeff! >>>>>> > > > > > >>>>>> > > > > > [1] https://github.com/apache/accumulo/pull/1487 >>>>>> > > > > > [2] https://github.com/apache/accumulo/pull/2574 >>>>>> > > > > > [3] https://github.com/apache/accumulo/issues/2406 >>>>>> > > > > > [4] https://github.com/apache/accumulo/pull/2215 >>>>>> > > > > > >>>>>> > > > > > On Fri, Apr 1, 2022 at 2:21 PM Dave Marion < >>>>>> dmario...@gmail.com> >>>>>> > > wrote: >>>>>> > > > > > >>>>>> > > > > > > I think it would be useful to do some release planning so >>>>>> that we >>>>>> > > know >>>>>> > > > > > what >>>>>> > > > > > > features we are working towards and in which release they >>>>>> will be >>>>>> > > in. >>>>>> > > > > > This >>>>>> > > > > > > would be helpful for determining what existing PRs need >>>>>> to make >>>>>> > it >>>>>> > > into >>>>>> > > > > > > 2.1.0. 2.1.0 is the LTM release, so patches for existing >>>>>> features >>>>>> > > will >>>>>> > > > > be >>>>>> > > > > > > backported (2.1.1, 2.1.2, 2.1.3, etc.) However, as >>>>>> defined in >>>>>> > [1], >>>>>> > > > > > features >>>>>> > > > > > > that don't make it into 2.1.0 will go into the next >>>>>> non-LTM >>>>>> > release >>>>>> > > > > > (2.2.0) >>>>>> > > > > > > and any patches to bugs in those features will go into >>>>>> the next >>>>>> > > non-LTM >>>>>> > > > > > > release after that (2.3.0). >>>>>> > > > > > > >>>>>> > > > > > > I'm not trying to hold up the 2.1.0 release by suggesting >>>>>> that we >>>>>> > > > > perform >>>>>> > > > > > > this activity. I'm just asking what the future holds, >>>>>> even if >>>>>> > it's >>>>>> > > just >>>>>> > > > > > one >>>>>> > > > > > > feature in the next non-LTM release. My concern is that >>>>>> the next >>>>>> > > > > release >>>>>> > > > > > > will be open-ended and anything not included in 2.1.0 >>>>>> might not >>>>>> > > get put >>>>>> > > > > > > into a release for a very long time. >>>>>> > > > > > > >>>>>> > > > > > > [1] >>>>>> https://accumulo.apache.org/contributor/versioning.html#LTM >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > > On Thu, Mar 31, 2022 at 11:43 AM Mike Miller < >>>>>> mmil...@apache.org >>>>>> > > >>>>>> > > > > wrote: >>>>>> > > > > > > >>>>>> > > > > > > > Starting an email chain of things that folks want to >>>>>> finish for >>>>>> > > 2.1. >>>>>> > > > > > Here >>>>>> > > > > > > > is what we currently have in the works that are most >>>>>> likely >>>>>> > going >>>>>> > > > > into >>>>>> > > > > > > 2.1: >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2569 >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2600 >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2293 >>>>>> > > > > > > > >>>>>> > > > > > > > Some things that may go into 2.1: >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2422 >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2475 >>>>>> > > > > > > > https://github.com/apache/accumulo/pull/2197 >>>>>> > > > > > > > >>>>>> > > > > > > > I created a Project for follow on work to the ZK >>>>>> property >>>>>> > > change. I >>>>>> > > > > was >>>>>> > > > > > > > planning on putting tasks in there that we want to >>>>>> complete for >>>>>> > > 2.1. >>>>>> > > > > > But >>>>>> > > > > > > we >>>>>> > > > > > > > could also use it for post 2.1 work. >>>>>> > > > > > > > https://github.com/apache/accumulo/projects/24 >>>>>> > > > > > > > https://github.com/apache/accumulo/issues/2469 >>>>>> > > > > > > > >>>>>> > > > > > > > FYI a draft copy of the release notes has already been >>>>>> on the >>>>>> > > > > website: >>>>>> > > > > > > > https://accumulo.apache.org/release/accumulo-2.1.0/ >>>>>> > > > > > > > >>>>>> > > > > > > > This may be a good thread to discuss whether or not a >>>>>> task >>>>>> > needs >>>>>> > > to >>>>>> > > > > go >>>>>> > > > > > > into >>>>>> > > > > > > > 2.1 or should wait for the next version. We currently >>>>>> have 32 >>>>>> > > open >>>>>> > > > > pull >>>>>> > > > > > > > requests so please email me if there is one that you >>>>>> would like >>>>>> > > > > > > prioritized >>>>>> > > > > > > > for 2.1. >>>>>> > > > > > > > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > >>>>>> > >>>>>> >>>>>
