Kseniya, Thanks for scheduling this call. Do you think we can switch to English if non-Russian speaking community members decide to join?
On Tue, Nov 3, 2020 at 1:32 PM Kseniya Romanova <romanova.ks....@gmail.com> wrote: > Let's do this community discussion open. Here's the link on zoom call in > Russian for Friday 6 PM: > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/274360378/ > > вт, 3 нояб. 2020 г. в 12:49, Nikolay Izhikov <nizhi...@apache.org>: > > > Time works for me. > > > > > 3 нояб. 2020 г., в 12:40, Alexey Goncharuk <alexey.goncha...@gmail.com > > > > написал(а): > > > > > > Nikolay, > > > > > > I am up for the call. I will try to explain my reasoning in greater > > detail > > > and will be glad to hear the concerns. Will this Friday, Nov 6th, work? > > > > > > вт, 3 нояб. 2020 г. в 10:09, Nikolay Izhikov <nizhi...@apache.org>: > > > > > >> Igniters, should we have a call for this topic? > > >> > > >>> 2 нояб. 2020 г., в 18:53, Pavel Tupitsyn <ptupit...@apache.org> > > >> написал(а): > > >>> > > >>>> not intend to rewrite everything from scratch > > >>> > > >>>> Every single test from Ignite 2.x should be moved to Ignite 3 > > >>>> regardless of how we choose to proceed. > > >>> > > >>> Alexey, thank you for the explanation, this addresses all of my > > concerns. > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On Mon, Nov 2, 2020 at 6:43 PM Andrey Mashenkov < > > >> andrey.mashen...@gmail.com> > > >>> wrote: > > >>> > > >>>> Hi, Igniters. > > >>>> > > >>>> * AFAIU, we need a new repo if we want to apply different > restrictions > > >> to > > >>>> pull requests, > > >>>> otherwise I see no difference for myself. > > >>>> E.g. make static analysis (do we have?), compile, styles, and > javadoc > > >>>> checks mandatory. > > >>>> > > >>>> I think that relaxed requirements here will lead to bad product > > quality. > > >>>> > > >>>> * Agree with Pavel, we should 'keep' integrations tests somehow. > > >>>> During active development tests will be broken most of time, so, > > >>>> I'd port them e.g. suite-by-suite once we will have a stable and > > >> featured > > >>>> environment to run them and of course make test's code clear and > avoid > > >>>> bad/non-relevant ones. > > >>>> > > >>>> * I like bottom-up approach. > > >>>> With it we could make a better framework. I mean clear component > > >> lifecycle, > > >>>> component wiring mechanics, general methods to approach core > > components > > >>>> such as exchange/communication > > >>>> to avoid code mess like we have in ExchangeFuture with all these > > custom > > >>>> callbacks for each component, interfaces like > > >>>> PartitionsExchangeAware, IgniteChangeGlobalStateSupport and > > >>>> a pack of > > >> start/stop/onKernalStart/onPluginStart/onActivate/onDisconnected > > >>>> and so on in various unexpected places. > > >>>> Hope, we will be able to port most of the good code to the new > > framework > > >>>> version. > > >>>> > > >>>> > > >>>> > > >>>> On Mon, Nov 2, 2020 at 6:18 PM Alexey Goncharuk < > > >>>> alexey.goncha...@gmail.com> > > >>>> wrote: > > >>>> > > >>>>> Nikolay, Pavel, > > >>>>> > > >>>>> Thanks for the feedback! First of all, I wanted to stress that I do > > not > > >>>>> intend to rewrite everything from scratch (I never used this > phrase). > > >>>> There > > >>>>> are significant parts of code that would be moved with minimal > > >>>>> modifications. Second, I never said that we will get rid of the old > > >> tests > > >>>>> codebase. Every single test from Ignite 2.x should be moved to > > Ignite 3 > > >>>>> regardless of how we choose to proceed. > > >>>>> > > >>>>> My point is that for some parts of the code a clean bottom-up > > >>>>> implementation will be cheaper in many ways. Let me give you a few > > >>>> concrete > > >>>>> examples: > > >>>>> > > >>>>> - I think no one can object that we need a cleanly separated > > >>>> persistence > > >>>>> layer for Ignite. There is a very raw draft IEP for this already. > On > > >>>> the > > >>>>> other hand, I think we also can agree that we need a split-brain > > >>>>> resistant > > >>>>> replication protocol for caches. There is also an IEP for this. > > >>>> Neither > > >>>>> of > > >>>>> the changes is a good fit for 2.x because they are likely to > > >> introduce > > >>>>> breaking changes in the persistence layer, configuration and > > >> behavior. > > >>>>> Additionally, these components are now tightly coupled, so there > is > > >> no > > >>>>> way > > >>>>> these two changes can be implemented in parallel and then merged > > >>>>> together > > >>>>> easily. So what we will end up with is having to implement these > > >>>> changes > > >>>>> sequentially, fixing all existing tests twice, and essentially > > >>>> throwing > > >>>>> away half of the work done because the other part of the change is > > >>>>> re-implemented > > >>>>> - Similar example goes with getting rid of IgniteInternalFuture > and > > >>>>> replacing it with CompletableFuture, and any other change that > > >> touches > > >>>>> the > > >>>>> asynchronous part of the code. > > >>>>> > > >>>>> Third, I do not see how this choice affects the UX of Ignite. The > end > > >>>> user > > >>>>> experience must be fixed in the IEP regardless of the development > > >> process > > >>>>> and the fact that we have gaps in this area in Ignite 2.x just > > confirms > > >>>>> that. > > >>>>> > > >>>>> Pavel, agree that a repo/branch is a technicality, I guess if > > >>>> reformulate, > > >>>>> my point is that we might agree to have a single development master > > >>>> branch > > >>>>> with 'disassembled' end-to-end functionality for some period of > time > > to > > >>>>> speed up development, and re-assemble the core features after > having > > >>>>> submodules tested independently. > > >>>>> > > >>>>> Nikolay, > > >>>>>> We have many features that have to evolve. > > >>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > >>>>> This is not very specific. In the end, resources are limited and we > > >> will > > >>>>> not be able to drive both tracks simultaneously, especially after a > > >>>> couple > > >>>>> of features having been implemented for Ignite 3.0. If there are > > indeed > > >>>>> some major changes that we want to do in Ignite 2.x instead of > > putting > > >>>>> effort into 3.0 - let's discuss them. I am just not aware of any, > > >> that's > > >>>>> why I am eager to move forward with Ignite 3.0. > > >>>>> > > >>>>>> We have bugs and issues that can be fixed in 2.x without breaking > > >>>> backward > > >>>>> compatibility. > > >>>>>> We have many users who are happy with the 2.x with all it’s > issues. > > >>>>> These changes will be covered by end-to-end tests and migrated to > > >> Ignite > > >>>>> 3.0, so I see no issues here. > > >>>>> > > >>>>> Finally, Anton & Nikolay > > >>>>> I do not have an estimate for this simply because the activity is > > >>>>> community-driven and it depends on the number of people willing to > > >>>>> contribute. With the current pace, I would hope to have an RC of > > Ignite > > >>>> 3.0 > > >>>>> to be ready by the end of 2021. My gut feeling is that by moving > with > > >>>>> incremental changes, we will not be able to implement even half of > > the > > >>>>> wishlist by that time. > > >>>>> I doubt that releasing several major releases with breaking changes > > >> will > > >>>>> make Ignite users happy either because each upgrade will cost > Ignite > > >>>> users > > >>>>> money, so the fewer major versions we release, the better. Thus my > > wish > > >>>> to > > >>>>> include all breaking changes in one release. > > >>>>> > > >>>>> I'll be now quiet for a while, let's see what other community > members > > >>>>> think. > > >>>>> > > >>>>> пн, 2 нояб. 2020 г. в 15:52, Pavel Tupitsyn <ptupit...@apache.org > >: > > >>>>> > > >>>>>> 1. Rewriting from scratch is never a good idea. > > >>>>>> We don't want to follow the path of Netscape and lose all our > users > > >>>>>> by the time we have a working 3.0 [1] > > >>>>>> > > >>>>>> 2. Not sure about new repo - seems like some pain and no gain, > > what's > > >>>> the > > >>>>>> problem with a branch? > > >>>>>> > > >>>>>> 3. We should keep existing integration tests when possible. > > >>>>>> We have accumulated a lot of edge case knowledge over the years, > > >>>>>> it is not a good idea to send all of that down the drain. > > >>>>>> Yes, integration tests are slow, but they are the most valuable. > > >>>>>> I think we can move more stuff into nightly runs and have a fast > and > > >>>>> modern > > >>>>>> basic suite. > > >>>>>> > > >>>>>> > > >>>>>> Alexey, you are much more familiar with the Ignite core codebase > > than > > >>>>> most > > >>>>>> of us, > > >>>>>> can you please explain in more detail which particular feature, in > > >> your > > >>>>>> opinion, > > >>>>>> mandates this "start from scratch" approach? > > >>>>>> Is it really not possible at all to follow a less radical way? > > >>>>>> > > >>>>>> > > >>>>>> [1] > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >> > > > https://www.joelonsoftware.com/2000/04/06/things-you-should-never-do-part-i/ > > >>>>>> > > >>>>>> On Mon, Nov 2, 2020 at 2:25 PM Nikolay Izhikov < > nizhi...@apache.org > > > > > >>>>>> wrote: > > >>>>>> > > >>>>>>> Hello, Alexey. > > >>>>>>> > > >>>>>>> I think that «rewriting from scratch» approach has a high risk to > > >>>> make > > >>>>>> new > > >>>>>>> features unusable. > > >>>>>>> At the time Ignite2 was started no-one wants to do bad UX or bad > > >>>>>> features. > > >>>>>>> Nevertheless, it happen. > > >>>>>>> > > >>>>>>> I think we can avoid it with the Ignite3 and successors if we > will > > >>>> move > > >>>>>>> step by step without keeping backward compatibility > > >>>>>>> With the step by step approach, we can focus on each component > > >>>>>> separately. > > >>>>>>> > > >>>>>>>> What new features are we planning to implement for Ignite 2.x? > > >>>>>>> > > >>>>>>> We have many features that have to evolve. > > >>>>>>> Snapshots, rebalance, tooling, tracing, zookeeper support, etc. > > >>>>>>> We have bugs and issues that can be fixed in 2.x without breaking > > >>>>>> backward > > >>>>>>> compatibility. > > >>>>>>> We have many users who are happy with the 2.x with all it’s > issues. > > >>>>>>> > > >>>>>>>> 2 нояб. 2020 г., в 14:09, Anton Vinogradov <a...@apache.org> > > >>>>> написал(а): > > >>>>>>>> > > >>>>>>>> Alexey, > > >>>>>>>> > > >>>>>>>> Do we have any estimates of how fast we'll be able to gain > > >>>>>>> production-ready > > >>>>>>>> AI 3.0 in case of a "new repo" choice? > > >>>>>>>> > > >>>>>>>> On Mon, Nov 2, 2020 at 2:01 PM Alexey Goncharuk < > > >>>>>>> alexey.goncha...@gmail.com> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Nikolay, > > >>>>>>>>> > > >>>>>>>>> What new features are we planning to implement for Ignite 2.x? > I > > >>>>> think > > >>>>>>> once > > >>>>>>>>> we commence working on Ignite 3.0, we should gradually cease > the > > >>>>>>> activity > > >>>>>>>>> on Ignite 2.x to mere bugfixes because such parallel > development > > >>>>> will > > >>>>>> be > > >>>>>>>>> overwhelming regardless of how we choose to proceed. > > >>>>>>>>> > > >>>>>>>>> пн, 2 нояб. 2020 г. в 13:38, Nikolay Izhikov < > > nizhi...@apache.org > > >>>>> : > > >>>>>>>>> > > >>>>>>>>>> To be clear: > > >>>>>>>>>> > > >>>>>>>>>>> I would suggest creating a new repository for Ignite 3.0 > > >>>>> (perhaps, a > > >>>>>>>>> new > > >>>>>>>>>> clean branch, but a new repo looks nicer to me) and a new > Ignite > > >>>>> 3.0 > > >>>>>>>>>> TeamCity project. > > >>>>>>>>>> > > >>>>>>>>>> +1 for new Team City project. > > >>>>>>>>>> +1 for new branch for Ignite3. > > >>>>>>>>>> -1 for new repo. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>>> 2 нояб. 2020 г., в 13:35, Nikolay Izhikov < > > >>>> nizhikov....@gmail.com > > >>>>>> > > >>>>>>>>>> написал(а): > > >>>>>>>>>>> > > >>>>>>>>>>> Hello, Alexey. > > >>>>>>>>>>> > > >>>>>>>>>>> I think it will hurt our project more than help. > > >>>>>>>>>>> Developing new features for 2 separate branches with the > > >>>> different > > >>>>>>> APIs > > >>>>>>>>>> and internal structure is overwhelming > > >>>>>>>>>>> > > >>>>>>>>>>> Maybe we should relax a bit requirements for Ignite3? > > >>>>>>>>>>> Maybe we should move step by step and make Ignite3 with new > > >>>>>>>>>> configuration than Ignite4 with new transactions, etc? > > >>>>>>>>>>> > > >>>>>>>>>>>> 2 нояб. 2020 г., в 13:14, Alexey Goncharuk < > > >>>>>>>>> alexey.goncha...@gmail.com> > > >>>>>>>>>> написал(а): > > >>>>>>>>>>>> > > >>>>>>>>>>>> Igniters, > > >>>>>>>>>>>> > > >>>>>>>>>>>> I wanted to pitch a rather radical idea regarding the Ignite > > >>>> 3.0 > > >>>>>>>>>>>> development which has occurred to me some time ago. > > >>>>>>>>>>>> > > >>>>>>>>>>>> We already have several IEPs targeted to Ignite 3.0 which > > imply > > >>>>>> major > > >>>>>>>>>>>> changes to the codebase (the change in replication protocol > > and > > >>>>>> thus > > >>>>>>>>>>>> transactions, change in binary format, updated metastorage, > > >>>> etc). > > >>>>>> We > > >>>>>>>>>> also > > >>>>>>>>>>>> planned significant changes in public APIs: configuration > > >>>> format > > >>>>>>>>> change, > > >>>>>>>>>>>> improvements in cache APIs, SQL APIs, transaction mode > rework. > > >>>>> The > > >>>>>>>>>> wishlist > > >>>>>>>>>>>> of changes for Ignite 3.0 is huge. > > >>>>>>>>>>>> > > >>>>>>>>>>>> So, I was wondering whether it makes sense to try to change > > the > > >>>>> old > > >>>>>>>>>>>> codebase, or start with a new baseline and move old pieces > of > > >>>>> code > > >>>>>>>>> that > > >>>>>>>>>> do > > >>>>>>>>>>>> not require significant rework. Personally, I would go with > > the > > >>>>>>> second > > >>>>>>>>>>>> option for the following reasons: > > >>>>>>>>>>>> > > >>>>>>>>>>>> - We have a chance to shift the development paradigm in the > > >>>>> project > > >>>>>>>>> and > > >>>>>>>>>>>> introduce the practice of true unit-tests. In the new > baseline > > >>>> at > > >>>>>> the > > >>>>>>>>>>>> beginning there will be no ability to run an end-to-end > > >>>> scenario, > > >>>>>>>>> thus > > >>>>>>>>>> we > > >>>>>>>>>>>> will be forced to write unit-tests. So far, such practice > was > > >>>>> hard > > >>>>>> to > > >>>>>>>>>>>> implement because of tight coupling between Ignite > components > > >>>> and > > >>>>>>>>>> inability > > >>>>>>>>>>>> to instantiate components without an instance of > > KernalContext. > > >>>>> For > > >>>>>>>>>>>> example, we should be able to thoroughly test internal > > >>>>> primitives, > > >>>>>>>>>> such as > > >>>>>>>>>>>> replication protocol (without actual communication), > > >>>> distributed > > >>>>>>>>>>>> metastorage contracts, persistence layer, etc. > > >>>>>>>>>>>> - We will significantly reduce the development cycle in the > > >>>>>> beginning > > >>>>>>>>>>>> (right now the RunAll takes two hours of astronomical time > > with > > >>>>>> empty > > >>>>>>>>>> TC; > > >>>>>>>>>>>> in the new approach developer will be able to run ALL tests > > >>>>> locally > > >>>>>>>>> in > > >>>>>>>>>> a > > >>>>>>>>>>>> matter of minutes) > > >>>>>>>>>>>> - We can get rid of TC bot and enforce green TC by > integrating > > >>>> TC > > >>>>>>>>> build > > >>>>>>>>>>>> results with GitHub PRs (the same way Travis is currently > > >>>>>> integrated > > >>>>>>>>>> to PR > > >>>>>>>>>>>> check). We should restrict PR merge without a TC check > > >>>>>>>>>>>> - We will still have to re-write all tests, but only once. > If > > >>>> we > > >>>>>> try > > >>>>>>>>> to > > >>>>>>>>>>>> modify the old codebase, we would need to modify all the > tests > > >>>>> for > > >>>>>>>>>> every > > >>>>>>>>>>>> major change (public API change, configuration change) > > >>>>>>>>>>>> - We will have fewer conflicts when working together. For > > >>>>> example, > > >>>>>> I > > >>>>>>>>>>>> cannot imagine how one would merge two changes of getting > rid > > >>>> of > > >>>>>>>>>>>> IgniteFuture and changes in replication protocol, for > example > > >>>>>>>>>>>> > > >>>>>>>>>>>> Technically, I would suggest creating a new repository for > > >>>> Ignite > > >>>>>> 3.0 > > >>>>>>>>>>>> (perhaps, a new clean branch, but a new repo looks nicer to > > me) > > >>>>>> and a > > >>>>>>>>>> new > > >>>>>>>>>>>> Ignite 3.0 TeamCity project. > > >>>>>>>>>>>> > > >>>>>>>>>>>> While it may seem quite radical, I do believe that this > > >>>> approach > > >>>>>> will > > >>>>>>>>>> give > > >>>>>>>>>>>> us more benefits than trying to make such major changes in > the > > >>>>>>>>> existing > > >>>>>>>>>>>> codebase. If needed, let's schedule a discord chat like > before > > >>>> to > > >>>>>>>>>> discuss > > >>>>>>>>>>>> this. > > >>>>>>>>>>>> > > >>>>>>>>>>>> WDYT? > > >>>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Best regards, > > >>>> Andrey V. Mashenkov > > >>>> > > >> > > >> > > > > >
