Update on 2.1 progress.
7 Pull requests in progress.
1111 marked as DONE. ( I just missed sending this at 11:11 EST)
https://github.com/apache/accumulo/projects/3

3 marked as TODO as Compaction follow on work:
https://github.com/apache/accumulo/projects/13

2 marked as TODO for the Single Node ZK follow on work.
https://github.com/apache/accumulo/projects/24



On Fri, May 13, 2022 at 9:38 AM Mike Miller <mmil...@apache.org> wrote:

> Update on 2.1 progress.
> 10 Pull Requests in progress.
> 1,097 marked as DONE.
> https://github.com/apache/accumulo/projects/3
>
> 3 Tickets marked as TODO as Compaction follow on work:
> https://github.com/apache/accumulo/projects/13
>
> Only 1 Ticket marked as TODO for the ZK follow on work. I thought there
> would be more here:
> https://github.com/apache/accumulo/projects/24
>
> On Fri, Apr 8, 2022 at 7:00 AM Mike Miller <mmil...@apache.org> wrote:
>
>> Update on 2.1 progress. For pull requests:
>> 15 currently in progress.
>> 32 are open as TODO. But a lot of these will get bumped to the next
>> version.
>> 1,025 DONE. Wow! Good work everyone.
>>
>> https://github.com/apache/accumulo/projects/3
>>
>> On Wed, Apr 6, 2022 at 4:55 PM Christopher <ctubb...@apache.org> wrote:
>>
>>> After some additional consideration, and getting a better understanding
>>> of
>>> how the code is expected to work from discussing it with Dave... I'm a
>>> little more inclined to support #2422 in 2.1, provided:
>>>
>>> 1. There's time for me to review it,
>>> 2. It is sufficiently decoupled from the existing code and marked
>>> experimental, so that we have the flexibility to alter its design, if it
>>> seems appropriate after it gets some exposure after the release,
>>> 3. Unit tests and integration tests are reliably passing (as stable as,
>>> or
>>> more stable than, they are currently),
>>> 4. No serious issues are discovered during review, and
>>> 5. It doesn't delay a release past early June, as I think this is a
>>> reasonable target date.
>>>
>>> This my wishlist before I can get behind it with a +1 for 2.1. If these
>>> aren't met, I do not intend to veto, but I'd be a -0 on its inclusion to
>>> 2.1. Of course, once I review it, my thoughts may change a bit.
>>>
>>> On Mon, Apr 4, 2022 at 7:07 PM Mike Miller <mmil...@apache.org> wrote:
>>>
>>> > I think I can finish the FATE refactor PR [1] for 2.1. I had been
>>> keeping
>>> > it up to date with the latest in main but stopped because it was too
>>> much
>>> > work. I was waiting until the ZK property changes are completed before
>>> > resolving the latest conflicts. I don't think it is much of a risk. It
>>> is
>>> > mostly cleanup and refactoring to remove generics from the
>>> serialization
>>> > code. It will be some work to revisit but I think the risk is pretty
>>> low.
>>> > It would allow changing the serialization, which we may be able to get
>>> into
>>> > 2.1 as well.
>>> >
>>> > [1] https://github.com/apache/accumulo/pull/2475
>>> >
>>> > On Mon, Apr 4, 2022 at 11:50 AM Keith Turner <ke...@deenlo.com> wrote:
>>> >
>>> > > On Mon, Apr 4, 2022 at 11:17 AM Christopher <ctubb...@apache.org>
>>> wrote:
>>> > > >
>>> > > > I haven't seen the metrics test fail very often lately. If it's
>>> > stable, I
>>> > > > don't mind removing the blocker on that issue, but I'd be
>>> reluctant to
>>> > > > close it entirely just yet, until we can verify it doesn't happen
>>> > > anymore.
>>> > > >
>>> > > > As for the original list of potential issues to include, I'm in
>>> favor
>>> > of
>>> > > > trying to get #2197 in. It was started awhile ago, is relatively
>>> simple
>>> > > and
>>> > > > well understood by several of us already... it just needs a bit of
>>> > > > attention to finalize reviews so it can be merged.
>>> > > >
>>> > > > However, I'm reluctant to include #2422, because I don't think it's
>>> > near
>>> > > > ready enough, and by the time it is, it will be very last minute,
>>> and I
>>> > > > don't want to delay 2.1 further for it. Even if it's included as an
>>> > > > experimental feature, I think it has huge potential to be
>>> disruptive,
>>> > or
>>> > > to
>>> > > > have a lot of churn by the time people actually have a chance to
>>> review
>>> > > it
>>> > > > thoroughly. Furthermore, I think there are possible alternatives
>>> (like
>>> > a
>>> > > > fully client-side implementation, based on offline scanners) that
>>> would
>>> > > > avoid the tight coupling of a new service to Accumulo's core code.
>>> This
>>> > >
>>> > > There are some advantages to scan servers over direct file access to
>>> > > consider.  One is scalability of computation, if a web server is
>>> > > serving N client queries with scan servers those can potentially go
>>> to
>>> > > different scan servers.  With direct file access, all N queries and
>>> > > their iterator stacks would have to run in the web server.  Another
>>> is
>>> > > scalability of caching/memory.  When web servers send queries to scan
>>> > > servers using a sticky algorithm for assigning tablets to groups of
>>> > > scan servers, it could lead to good cache utilization and sharing
>>> that
>>> > > may not be possible when running scans directly in the web server. So
>>> > > scan servers allow scaling cache and computations for queries
>>> > > independently of web servers in way that may not be possible with
>>> > > direct file access.
>>> > >
>>> > > Another advantage to consider is isolation.  With direct file access
>>> > > and queries running directly in a web server, a bad query could bring
>>> > > down a web server and lots of unrelated queries.  Having a bad query
>>> > > bring down a scan server may be less disruptive.
>>> > >
>>> > > > thread isn't for discussing this in depth, so we can have that
>>> > discussion
>>> > > > in a separate thread, but I'm generally opposed to including it
>>> this
>>> > late
>>> > > > in 2.1's development, given the timing, size and scope, tight
>>> coupling,
>>> > > and
>>> > > > current state.
>>> > > >
>>> > > > I don't know enough about #2475 to have a strong opinion, but it
>>> looks
>>> > > big,
>>> > > > and possibly high-risk, given the critical code it touches. It
>>> > currently
>>> > > > has a substantial number of conflicts with the main branch.
>>> However, I
>>> > > was
>>> > > > thinking that *some* minimal refactoring (like low-risk automatic
>>> > > > refactoring, like moving packages) could be done. So, if that's all
>>> > this
>>> > > > does, it might be okay. Otherwise, maybe it can be simplified? At
>>> the
>>> > > very
>>> > > > least, I was thinking it would be a good opportunity to move the
>>> > > > `org.apache.accumulo.fate` packages into an appropriate
>>> > > > `org.apache.accumulo.core` parent package (some would go to
>>> > > o.a.a.core.fate
>>> > > > and others might go to o.a.a.core.util or similar) to keep the
>>> package
>>> > > > namespaces standardized, which is helpful to avoid naming
>>> collisions
>>> > and
>>> > > > jar sealing issues, as well as for less complicated jigsaw module
>>> > > > definitions in future. Since 2.1 FaTE is already incompatible with
>>> > prior
>>> > > > versions, a rename at this time would be less disruptive.
>>> > > >
>>> > > > Another task I had wanted to be done for 2.1, before I got
>>> distracted
>>> > > > fixing test failures during and after Christmas and trying to work
>>> > > through
>>> > > > the singleton manager zookeeper stuff to see what we could
>>> simplify.
>>> > > What I
>>> > > > had wanted done was to standardize the way we pass table
>>> identifiers
>>> > > (name,
>>> > > > IDs) across the RPC layer, since we currently do that
>>> inconsistently. I
>>> > > > don't remember if there's an existing ticket open for it, but I
>>> have a
>>> > > > working branch I had started working out of for it before
>>> Christmas.
>>> > It's
>>> > > > relatively simple work, and would set us up for some much better
>>> APIs
>>> > > going
>>> > > > forward, as well as help with logging information about table
>>> actions.
>>> > If
>>> > > > necessary, it could be bumped to a future version, but then we'd
>>> have
>>> > > more
>>> > > > churn in the thrift layer. So, I'd prefer to get it for 2.1 to
>>> avoid
>>> > > that.
>>> > > >
>>> > > > As for planning, I was thinking early May for a code freeze
>>> (except bug
>>> > > > fixes and small improvements found during testing), so we can try
>>> to
>>> > > > release towards the end of May/early June. If we go with that
>>> timeline,
>>> > > > that's not a lot of time to wrap up features and have time for
>>> > > > review/testing, so we may need to be selective about what we hold
>>> off
>>> > > until
>>> > > > the next version, unless we want to further delay 2.1.
>>> > > >
>>> > > >
>>> > > > On Mon, Apr 4, 2022 at 9:13 AM Dave Marion <dmario...@gmail.com>
>>> > wrote:
>>> > > >
>>> > > > > I think [3] is OBE and can be closed.
>>> > > > >
>>> > > > > On Mon, Apr 4, 2022 at 9:11 AM Mike Miller <mmil...@apache.org>
>>> > wrote:
>>> > > > >
>>> > > > > > Yes I agree, that was the goal of this email thread. I found a
>>> few
>>> > > more
>>> > > > > > tickets that should be addressed for the next release.
>>> > > > > >
>>> > > > > > Ivan - There was some work done on this PR but it has been some
>>> > > time. Do
>>> > > > > > you want to take a look at it? Implement a Thread limit. [1]
>>> > > > > > Keith T - I think we should get this one merged to fix that
>>> > > consistency
>>> > > > > > check bug I found. It looks like it is finished. [2]
>>> > > > > > Dave & Dom - Were you guys able to figure out a fix for the new
>>> > > external
>>> > > > > > compaction metrics test? [3]
>>> > > > > >
>>> > > > > > FYI we have 6 blockers for 2.1:
>>> > > > > > https://github.com/apache/accumulo/labels/blocker
>>> > > > > >
>>> > > > > > This is almost definitely going into 2.1 [4]. Thanks Jeff!
>>> > > > > >
>>> > > > > > [1] https://github.com/apache/accumulo/pull/1487
>>> > > > > > [2] https://github.com/apache/accumulo/pull/2574
>>> > > > > > [3] https://github.com/apache/accumulo/issues/2406
>>> > > > > > [4] https://github.com/apache/accumulo/pull/2215
>>> > > > > >
>>> > > > > > On Fri, Apr 1, 2022 at 2:21 PM Dave Marion <
>>> dmario...@gmail.com>
>>> > > wrote:
>>> > > > > >
>>> > > > > > > I think it would be useful to do some release planning so
>>> that we
>>> > > know
>>> > > > > > what
>>> > > > > > > features we are working towards and in which release they
>>> will be
>>> > > in.
>>> > > > > > This
>>> > > > > > > would be helpful for determining what existing PRs need to
>>> make
>>> > it
>>> > > into
>>> > > > > > > 2.1.0. 2.1.0 is the LTM release, so patches for existing
>>> features
>>> > > will
>>> > > > > be
>>> > > > > > > backported (2.1.1, 2.1.2, 2.1.3, etc.) However, as defined in
>>> > [1],
>>> > > > > > features
>>> > > > > > > that don't make it into 2.1.0 will go into the next non-LTM
>>> > release
>>> > > > > > (2.2.0)
>>> > > > > > > and any patches to bugs in those features will go into the
>>> next
>>> > > non-LTM
>>> > > > > > > release after that (2.3.0).
>>> > > > > > >
>>> > > > > > > I'm not trying to hold up the 2.1.0 release by suggesting
>>> that we
>>> > > > > perform
>>> > > > > > > this activity. I'm just asking what the future holds, even if
>>> > it's
>>> > > just
>>> > > > > > one
>>> > > > > > > feature in the next non-LTM release. My concern is that the
>>> next
>>> > > > > release
>>> > > > > > > will be open-ended and anything not included in 2.1.0 might
>>> not
>>> > > get put
>>> > > > > > > into a release for a very long time.
>>> > > > > > >
>>> > > > > > > [1]
>>> https://accumulo.apache.org/contributor/versioning.html#LTM
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Thu, Mar 31, 2022 at 11:43 AM Mike Miller <
>>> mmil...@apache.org
>>> > >
>>> > > > > wrote:
>>> > > > > > >
>>> > > > > > > > Starting an email chain of things that folks want to
>>> finish for
>>> > > 2.1.
>>> > > > > > Here
>>> > > > > > > > is what we currently have in the works that are most likely
>>> > going
>>> > > > > into
>>> > > > > > > 2.1:
>>> > > > > > > > https://github.com/apache/accumulo/pull/2569
>>> > > > > > > > https://github.com/apache/accumulo/pull/2600
>>> > > > > > > > https://github.com/apache/accumulo/pull/2293
>>> > > > > > > >
>>> > > > > > > > Some things that may go into 2.1:
>>> > > > > > > > https://github.com/apache/accumulo/pull/2422
>>> > > > > > > > https://github.com/apache/accumulo/pull/2475
>>> > > > > > > > https://github.com/apache/accumulo/pull/2197
>>> > > > > > > >
>>> > > > > > > > I created a Project for follow on work to the ZK property
>>> > > change. I
>>> > > > > was
>>> > > > > > > > planning on putting tasks in there that we want to
>>> complete for
>>> > > 2.1.
>>> > > > > > But
>>> > > > > > > we
>>> > > > > > > > could also use it for post 2.1 work.
>>> > > > > > > > https://github.com/apache/accumulo/projects/24
>>> > > > > > > > https://github.com/apache/accumulo/issues/2469
>>> > > > > > > >
>>> > > > > > > > FYI a draft copy of the release notes has already been on
>>> the
>>> > > > > website:
>>> > > > > > > > https://accumulo.apache.org/release/accumulo-2.1.0/
>>> > > > > > > >
>>> > > > > > > > This may be a good thread to discuss whether or not a task
>>> > needs
>>> > > to
>>> > > > > go
>>> > > > > > > into
>>> > > > > > > > 2.1 or should wait for the next version. We currently have
>>> 32
>>> > > open
>>> > > > > pull
>>> > > > > > > > requests so please email me if there is one that you would
>>> like
>>> > > > > > > prioritized
>>> > > > > > > > for 2.1.
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > >
>>> >
>>>
>>

Reply via email to