Pavel, all the interesting points will be anyway published here in English (as the principal "if it's not on devlist it doesn't happened" is still relevant). This is just a quick call for a group of developers. Later we can do a separate presentation of idea and discussion in English as we did for the Ignite 3.0 draft of changes.
вт, 3 нояб. 2020 г. в 13:52, Pavel Tupitsyn <ptupit...@apache.org>: > Kseniya, > > Thanks for scheduling this call. > Do you think we can switch to English if non-Russian speaking community > members decide to join? > > On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <romanova.ks....@gmail.com > > > wrote: > > > Let's do this community discussion open. Here's the link on zoom call in > > Russian for Friday 6 PM: > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ > > > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhi...@apache.org>: > > > > > Time works for me. > > > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk < > alexey.goncha...@gmail.com > > > > > > написал(а): > > > > > > > > Nikolay, > > > > > > > > I am up for the call. I will try to explain my reasoning in greater > > > detail > > > > and will be glad to hear the concerns. Will this Friday, Nov 6th, > work? > > > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhi...@apache.org>: > > > > > > > >> Igniters, should we have a call for this topic? > > > >> > > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org> > > > >> написал(а): > > > >>> > > > >>>> not intend to rewrite everything from scratch > > > >>> > > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3 > > > >>>> regardless of how we choose to proceed. > > > >>> > > > >>> Alexey, thank you for the explanation, this addresses all of my > > > concerns. > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < > > > >> andrey.mashen...@gmail.com> > > > >>> wrote: > > > >>> > > > >>>> Hi, Igniters. > > > >>>> > > > >>>> * AFAIU, we need a new repo if we want to apply different > > restrictions > > > >> to > > > >>>> pull requests, > > > >>>> otherwise I see no difference for myself. > > > >>>> E.g. make static analysis (do we have?), compile, styles, and > > javadoc > > > >>>> checks mandatory. > > > >>>> > > > >>>> I think that relaxed requirements here will lead to bad product > > > quality. > > > >>>> > > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow. > > > >>>> During active development tests will be broken most of time, so, > > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and > > > >> featured > > > >>>> environment to run them and of course make test's code clear and > > avoid > > > >>>> bad/non-relevant ones. > > > >>>> > > > >>>> * I like bottom-up approach. > > > >>>> With it we could make a better framework. I mean clear component > > > >> lifecycle, > > > >>>> component wiring mechanics, general methods to approach core > > > components > > > >>>> such as exchange/communication > > > >>>> to avoid code mess like we have in ExchangeFuture with all these > > > custom > > > >>>> callbacks for each component, interfaces like > > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and > > > >>>> a pack of > > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > > > >>>> and so on in various unexpected places. > > > >>>> Hope, we will be able to port most of the good code to the new > > > framework > > > >>>> version. > > > >>>> > > > >>>> > > > >>>> > > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > > > >>>> alexey.goncha...@gmail.com> > > > >>>> wrote: > > > >>>> > > > >>>>> Nikolay, Pavel, > > > >>>>> > > > >>>>> Thanks for the feedback! First of all, I wanted to stress that I > do > > > not > > > >>>>> intend to rewrite everything from scratch (I never used this > > phrase). > > > >>>> There > > > >>>>> are significant parts of code that would be moved with minimal > > > >>>>> modifications. Second, I never said that we will get rid of the > old > > > >> tests > > > >>>>> codebase. Every single test from Ignite 2.x should be moved to > > > Ignite 3 > > > >>>>> regardless of how we choose to proceed. > > > >>>>> > > > >>>>> My point is that for some parts of the code a clean bottom-up > > > >>>>> implementation will be cheaper in many ways. Let me give you a > few > > > >>>> concrete > > > >>>>> examples: > > > >>>>> > > > >>>>> - I think no one can object that we need a cleanly separated > > > >>>> persistence > > > >>>>> layer for Ignite. There is a very raw draft IEP for this > already. > > On > > > >>>> the > > > >>>>> other hand, I think we also can agree that we need a split-brain > > > >>>>> resistant > > > >>>>> replication protocol for caches. There is also an IEP for this. > > > >>>> Neither > > > >>>>> of > > > >>>>> the changes is a good fit for 2.x because they are likely to > > > >> introduce > > > >>>>> breaking changes in the persistence layer, configuration and > > > >> behavior. > > > >>>>> Additionally, these components are now tightly coupled, so there > > is > > > >> no > > > >>>>> way > > > >>>>> these two changes can be implemented in parallel and then merged > > > >>>>> together > > > >>>>> easily. So what we will end up with is having to implement these > > > >>>> changes > > > >>>>> sequentially, fixing all existing tests twice, and essentially > > > >>>> throwing > > > >>>>> away half of the work done because the other part of the change > is > > > >>>>> re-implemented > > > >>>>> - Similar example goes with getting rid of IgniteInternalFuture > > and > > > >>>>> replacing it with CompletableFuture, and any other change that > > > >> touches > > > >>>>> the > > > >>>>> asynchronous part of the code. > > > >>>>> > > > >>>>> Third, I do not see how this choice affects the UX of Ignite. The > > end > > > >>>> user > > > >>>>> experience must be fixed in the IEP regardless of the development > > > >> process > > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just > > > confirms > > > >>>>> that. > > > >>>>> > > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if > > > >>>> reformulate, > > > >>>>> my point is that we might agree to have a single development > master > > > >>>> branch > > > >>>>> with 'disassembled' end-to-end functionality for some period of > > time > > > to > > > >>>>> speed up development, and re-assemble the core features after > > having > > > >>>>> submodules tested independently. > > > >>>>> > > > >>>>> Nikolay, > > > >>>>>> We have many features that have to evolve. > > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > > >>>>> This is not very specific. In the end, resources are limited and > we > > > >> will > > > >>>>> not be able to drive both tracks simultaneously, especially > after a > > > >>>> couple > > > >>>>> of features having been implemented for Ignite 3.0. If there are > > > indeed > > > >>>>> some major changes that we want to do in Ignite 2.x instead of > > > putting > > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any, > > > >> that's > > > >>>>> why I am eager to move forward with Ignite 3.0. > > > >>>>> > > > >>>>>> We have bugs and issues that can be fixed in 2.x without > breaking > > > >>>> backward > > > >>>>> compatibility. > > > >>>>>> We have many users who are happy with the 2.x with all it’s > > issues. > > > >>>>> These changes will be covered by end-to-end tests and migrated to > > > >> Ignite > > > >>>>> 3.0, so I see no issues here. > > > >>>>> > > > >>>>> Finally, Anton & Nikolay > > > >>>>> I do not have an estimate for this simply because the activity is > > > >>>>> community-driven and it depends on the number of people willing > to > > > >>>>> contribute. With the current pace, I would hope to have an RC of > > > Ignite > > > >>>> 3.0 > > > >>>>> to be ready by the end of 2021. My gut feeling is that by moving > > with > > > >>>>> incremental changes, we will not be able to implement even half > of > > > the > > > >>>>> wishlist by that time. > > > >>>>> I doubt that releasing several major releases with breaking > changes > > > >> will > > > >>>>> make Ignite users happy either because each upgrade will cost > > Ignite > > > >>>> users > > > >>>>> money, so the fewer major versions we release, the better. Thus > my > > > wish > > > >>>> to > > > >>>>> include all breaking changes in one release. > > > >>>>> > > > >>>>> I'll be now quiet for a while, let's see what other community > > members > > > >>>>> think. > > > >>>>> > > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn < > ptupit...@apache.org > > >: > > > >>>>> > > > >>>>>> 1. Rewriting from scratch is never a good idea. > > > >>>>>> We don't want to follow the path of Netscape and lose all our > > users > > > >>>>>> by the time we have a working 3.0 [1] > > > >>>>>> > > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain, > > > what's > > > >>>> the > > > >>>>>> problem with a branch? > > > >>>>>> > > > >>>>>> 3. We should keep existing integration tests when possible. > > > >>>>>> We have accumulated a lot of edge case knowledge over the years, > > > >>>>>> it is not a good idea to send all of that down the drain. > > > >>>>>> Yes, integration tests are slow, but they are the most valuable. > > > >>>>>> I think we can move more stuff into nightly runs and have a fast > > and > > > >>>>> modern > > > >>>>>> basic suite. > > > >>>>>> > > > >>>>>> > > > >>>>>> Alexey, you are much more familiar with the Ignite core codebase > > > than > > > >>>>> most > > > >>>>>> of us, > > > >>>>>> can you please explain in more detail which particular feature, > in > > > >> your > > > >>>>>> opinion, > > > >>>>>> mandates this "start from scratch" approach? > > > >>>>>> Is it really not possible at all to follow a less radical way? > > > >>>>>> > > > >>>>>> > > > >>>>>> [1] > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >> > > > > > > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > > > >>>>>> > > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov < > > nizhi...@apache.org > > > > > > > >>>>>> wrote: > > > >>>>>> > > > >>>>>>> Hello, Alexey. > > > >>>>>>> > > > >>>>>>> I think that «rewriting from scratch» approach has a high risk > to > > > >>>> make > > > >>>>>> new > > > >>>>>>> features unusable. > > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or > bad > > > >>>>>> features. > > > >>>>>>> Nevertheless, it happen. > > > >>>>>>> > > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we > > will > > > >>>> move > > > >>>>>>> step by step without keeping backward compatibility > > > >>>>>>> With the step by step approach, we can focus on each component > > > >>>>>> separately. > > > >>>>>>> > > > >>>>>>>> What new features are we planning to implement for Ignite 2.x? > > > >>>>>>> > > > >>>>>>> We have many features that have to evolve. > > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > > >>>>>>> We have bugs and issues that can be fixed in 2.x without > breaking > > > >>>>>> backward > > > >>>>>>> compatibility. > > > >>>>>>> We have many users who are happy with the 2.x with all it’s > > issues. > > > >>>>>>> > > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > > > >>>>> написал(а): > > > >>>>>>>> > > > >>>>>>>> Alexey, > > > >>>>>>>> > > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain > > > >>>>>>> production-ready > > > >>>>>>>> AI 3.0 in case of a "new repo" choice? > > > >>>>>>>> > > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > > > >>>>>>> alexey.goncha...@gmail.com> > > > >>>>>>>> wrote: > > > >>>>>>>> > > > >>>>>>>>> Nikolay, > > > >>>>>>>>> > > > >>>>>>>>> What new features are we planning to implement for Ignite > 2.x? > > I > > > >>>>> think > > > >>>>>>> once > > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease > > the > > > >>>>>>> activity > > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel > > development > > > >>>>> will > > > >>>>>> be > > > >>>>>>>>> overwhelming regardless of how we choose to proceed. > > > >>>>>>>>> > > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov < > > > nizhi...@apache.org > > > >>>>> : > > > >>>>>>>>> > > > >>>>>>>>>> To be clear: > > > >>>>>>>>>> > > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0 > > > >>>>> (perhaps, a > > > >>>>>>>>> new > > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new > > Ignite > > > >>>>> 3.0 > > > >>>>>>>>>> TeamCity project. > > > >>>>>>>>>> > > > >>>>>>>>>> +1 for new Team City project. > > > >>>>>>>>>> +1 for new branch for Ignite3. > > > >>>>>>>>>> -1 for new repo. > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > > > >>>> nizhikov....@gmail.com > > > >>>>>> > > > >>>>>>>>>> написал(а): > > > >>>>>>>>>>> > > > >>>>>>>>>>> Hello, Alexey. > > > >>>>>>>>>>> > > > >>>>>>>>>>> I think it will hurt our project more than help. > > > >>>>>>>>>>> Developing new features for 2 separate branches with the > > > >>>> different > > > >>>>>>> APIs > > > >>>>>>>>>> and internal structure is overwhelming > > > >>>>>>>>>>> > > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3? > > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new > > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc? > > > >>>>>>>>>>> > > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > > > >>>>>>>>> alexey.goncha...@gmail.com> > > > >>>>>>>>>> написал(а): > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Igniters, > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the > Ignite > > > >>>> 3.0 > > > >>>>>>>>>>>> development which has occurred to me some time ago. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which > > > imply > > > >>>>>> major > > > >>>>>>>>>>>> changes to the codebase (the change in replication > protocol > > > and > > > >>>>>> thus > > > >>>>>>>>>>>> transactions, change in binary format, updated > metastorage, > > > >>>> etc). > > > >>>>>> We > > > >>>>>>>>>> also > > > >>>>>>>>>>>> planned significant changes in public APIs: configuration > > > >>>> format > > > >>>>>>>>> change, > > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode > > rework. > > > >>>>> The > > > >>>>>>>>>> wishlist > > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to > change > > > the > > > >>>>> old > > > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces > > of > > > >>>>> code > > > >>>>>>>>> that > > > >>>>>>>>>> do > > > >>>>>>>>>>>> not require significant rework. Personally, I would go > with > > > the > > > >>>>>>> second > > > >>>>>>>>>>>> option for the following reasons: > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in > the > > > >>>>> project > > > >>>>>>>>> and > > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new > > baseline > > > >>>> at > > > >>>>>> the > > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end > > > >>>> scenario, > > > >>>>>>>>> thus > > > >>>>>>>>>> we > > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice > > was > > > >>>>> hard > > > >>>>>> to > > > >>>>>>>>>>>> implement because of tight coupling between Ignite > > components > > > >>>> and > > > >>>>>>>>>> inability > > > >>>>>>>>>>>> to instantiate components without an instance of > > > KernalContext. > > > >>>>> For > > > >>>>>>>>>>>> example, we should be able to thoroughly test internal > > > >>>>> primitives, > > > >>>>>>>>>> such as > > > >>>>>>>>>>>> replication protocol (without actual communication), > > > >>>> distributed > > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc. > > > >>>>>>>>>>>> - We will significantly reduce the development cycle in > the > > > >>>>>> beginning > > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time > > > with > > > >>>>>> empty > > > >>>>>>>>>> TC; > > > >>>>>>>>>>>> in the new approach developer will be able to run ALL > tests > > > >>>>> locally > > > >>>>>>>>> in > > > >>>>>>>>>> a > > > >>>>>>>>>>>> matter of minutes) > > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by > > integrating > > > >>>> TC > > > >>>>>>>>> build > > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently > > > >>>>>> integrated > > > >>>>>>>>>> to PR > > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check > > > >>>>>>>>>>>> - We will still have to re-write all tests, but only once. > > If > > > >>>> we > > > >>>>>> try > > > >>>>>>>>> to > > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the > > tests > > > >>>>> for > > > >>>>>>>>>> every > > > >>>>>>>>>>>> major change (public API change, configuration change) > > > >>>>>>>>>>>> - We will have fewer conflicts when working together. For > > > >>>>> example, > > > >>>>>> I > > > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting > > rid > > > >>>> of > > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for > > example > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> Technically, I would suggest creating a new repository for > > > >>>> Ignite > > > >>>>>> 3.0 > > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer > to > > > me) > > > >>>>>> and a > > > >>>>>>>>>> new > > > >>>>>>>>>>>> Ignite 3.0 TeamCity project. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this > > > >>>> approach > > > >>>>>> will > > > >>>>>>>>>> give > > > >>>>>>>>>>>> us more benefits than trying to make such major changes in > > the > > > >>>>>>>>> existing > > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like > > before > > > >>>> to > > > >>>>>>>>>> discuss > > > >>>>>>>>>>>> this. > > > >>>>>>>>>>>> > > > >>>>>>>>>>>> WDYT? > > > >>>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>>> > > > >>>>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>> > > > >>>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> Best regards, > > > >>>> Andrey V. Mashenkov > > > >>>> > > > >> > > > >> > > > > > > > > >
