Downloading additional jars will not 100% work in private networks, so we must provide full distro anyway.
Also, I do not see any problems in downloading all dependencies, because their summary size is not too big. I suggest, in addition to tgz archive for manual deployment, provide Ignite distribution as RPM (or other package manager(s)) for Linux (MSI for Windows) to automate distribution upgrades, instead of reinventing the wheel. пн, 31 авг. 2020 г. в 10:09, Valentin Kulichenko < valentin.kuliche...@gmail.com>: > Hi Nikolay, > > Thanks for your feedback! Absolutely, the IEP is currently in the draft > mode - we will be adding more details going forward. Also, Petr mentioned > that he wants to create a prototype which I think will make things clearer > for all of us. > > As for "download jars from the Internet on a production server", that's not > necessarily what we propose. In case of Maven, artifacts are downloaded > from a specified repository, which doesn't have to be on the Internet. > Companies that have security concerns typically have a mirror/proxy > containing approved artifacts only, already checked for viruses, etc. > Production servers use the proxy rather than public repositories. > > The concerns that you brought up are certainly valid. However, I believe > that using Maven actually addresses them, because Maven (as any other > package/dependency manager for that matter) provides the corresponding > functionality out of the box. > > On the contrary, users who currently download our ZIP package are *forced* > to download modules and dependencies that they do not even need and will > not use. In my experience, this is actually a bigger source of concern. > > Anyway, let us add more details to the IEP, and we will go from there. > > P.S. I'm also not sure I agree with your assessment of the embedded mode > and thick clients, but that's probably a different discussion. Please feel > free to create a new thread if you would like to discuss this. > > -Val > > On Sat, Aug 29, 2020 at 1:46 AM Nikolay Izhikov <nizhi...@apache.org> > wrote: > > > Also, guys. > > > > Can we make one step back and add some requirements to the IEP. > > How we want to install and upgrade Ignite? > > What specific issues we want to solve? > > > > With this input we can be on the same page during solution discussion. > > > > > 29 авг. 2020 г., в 11:41, Nikolay Izhikov <nizhikov....@gmail.com> > > написал(а): > > > > > > Hello, Val. > > > > > > Sorry, but I sill don’t understand how «download jars from the internet > > on production server» approach helps us to solve the issues you describe. > > > > > > Moreover, with this enhancement you want to add more dependencies to > > Ignite and create some security vulnerabilities: > > > * Maven tool - new dependency. I doubt that and of the production > > server have it > > > * Internet maven repository - many production servers restrict > > outgoing internet connection to any addresses. > > > It’s hard to understand why distributed DB should have > the > > ability to connect to some kind of address on the Internet. > > > * All Ignite distributions can be compromised with the hack of > > some random maven repo or with the malicious proxy. > > > * Note, that Ignite installation or upgrade procedure can become > > dead just because of some Maven repo down. > > > > > > > > > I have 2 concerns: > > > > > > 1. We shouldn’t download any jars from the internet during production > > deployment or upgrade procedure. > > > 2. The IEP description, for now, is quite small for such important > > things as a way we distribute and upgrade Ignite. > > > Petr, can you, please, make IEP more specific. > > > I think we should add the following details: > > > * Description of the typical Ignite deployment procedure > > including folder structure(config, persistence, .m2 and other folders) > > > * This also should include examples of the bash > > commands if any required. > > > * Description of the typical upgrade(downgrade?) > procedure > > > * This also should include examples of the bash > > commands if any required. > > > * Description of «single tool responsible for all > > operations». > > > * all commands > > > * all operations > > > * all parameters > > > > > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively > > forces (or at least motivates) users to put everything related to Ignite > > under IGNITE_HOME (configs, additional JARs, etc.). > > > > > > AFAIU this issue doesn’t relate to the way Ignite distributed. > > > We can and should fix this outside of this IEP. > > > > > >> 2. Modules are enabled/disabled manually by moving folders around… > this > > is error-prone, > > > > > > Makes sense. > > > Let’s fix this without «download jar from the internet»? > > > We can have some kind of config with the enabled modules. > > > > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if > > you look at how this ZIP is built, you will notice that we don't actually > > include all transitive dependencies, > > > > > > We need to fix this. > > > Let’s include all the required JARs to the distribution. > > > > > >> which means that an embedded node that uses Maven can have a different > > classpath. > > > > > > Embedded Ignite node is an error-prone design decision in the first > > place. > > > We can’t fix it with the «download jars from the internet» approach. > > > We should eliminate the client node as a thing and use a thin clients > > instead of it. > > > > > >> 4. ignite.sh allows manual modifications (there are several "Uncomment > > this to..." lines). This complicates upgrades as well. > > > > > > Makes sense. > > > Let’s fix this. > > > > > >> The whole purpose of the IEP is to fix the above, automate and > simplify > > operations for the users. The general idea is to create a single tool > > responsible for all those operations and use Maven under the hood to > > include/exclude modules. > > > > > > For now, IEP lacks the description of this tool. > > > > > >> 28 авг. 2020 г., в 22:35, Valentin Kulichenko < > > valentin.kuliche...@gmail.com> написал(а): > > >> > > >> Hi Nikolay, > > >> > > >> Here are some of the issues that I see with the current binary ZIP > > package > > >> that we create for every release. > > >> > > >> 1. Most paths are resolved relative to IGNITE_HOME, which effectively > > >> forces (or at least motivates) users to put everything related to > Ignite > > >> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user > > >> downloads and unzips a new version, they have to somehow merge the > > content > > >> together - that's poor upgradability. > > >> 2. Modules are enabled/disabled manually by moving folders around. Not > > only > > >> this is error-prone, but it also adds more issues to upgradability - > you > > >> have to apply these steps for every version you download. > > >> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if > > you > > >> look at how this ZIP is built, you will notice that we don't actually > > >> include all transitive dependencies, which means that an embedded node > > that > > >> uses Maven can have a different classpath. It seems that we've been > > lucky > > >> so far as this hasn't caused any issues, but I still think the > approach > > is > > >> wrong. The behavior should be consistent. > > >> 4. ignite.sh allows manual modifications (there are several "Uncomment > > this > > >> to..." lines). This complicates upgrades as well. > > >> > > >> The whole purpose of the IEP is to fix the above, automate and > > >> simplify operations for the users. The general idea is to create a > > single > > >> tool responsible for all those operations and use Maven under the hood > > to > > >> include/exclude modules. > > >> > > >> Exact commands and steps are still under discussion though. It looks > > like > > >> Petr is going to create a prototype that can be used as a starting > > point, > > >> but if you have any suggestions or ideas in the meantime, please share > > them > > >> with us. > > >> > > >> -Val > > >> > > >> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <nizhi...@apache.org > > > > >> wrote: > > >> > > >>> Hello, Igniters. > > >>> > > >>> Not sure if I understand the essence of the proposal at the moment. > > >>> Can you, please, describe the advantages of the proposed way from the > > user > > >>> perspective? > > >>> > > >>> How the typical DevOps pipeline should look like with this > enhancement? > > >>> > > >>> How I as a user can create a fully functional installation package of > > the > > >>> Ignite? > > >>> AFAIK downloading some artifacts from the internet straight to the > > >>> production server is a security anti-pattern. > > >>> > > >>> > > >>>> 28 авг. 2020 г., в 01:59, Denis Magda <dma...@apache.org> > написал(а): > > >>>> > > >>>> Petr, thanks, > > >>>> > > >>>> There is also a collection of modules located in our extensions > > >>> repository: > > >>>> https://github.com/apache/ignite-extensions > > >>>> > > >>>> @Saikat Maitra <saikat.mai...@gmail.com> is migrating all our > > existing > > >>>> integrations to that repository and, once this is done, the > extensions > > >>> will > > >>>> be released to Maven separately. Is it reasonable to expand the > scope > > of > > >>>> the IEP-52 and contemplate on how to install those extensions? > > >>>> > > >>>> - > > >>>> Denis > > >>>> > > >>>> > > >>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko < > > >>>> valentin.kuliche...@gmail.com> wrote: > > >>>> > > >>>>> Hi Petr, > > >>>>> > > >>>>> The proposal makes sense to me - thanks for starting the > discussion. > > >>>>> Current installation and upgrade procedures involve a lot of manual > > >>> steps > > >>>>> and quite error-prone, we need to automate and simplify them as > much > > as > > >>>>> possible. > > >>>>> > > >>>>> I believe we eventually should end up with a unified command-line > > tool > > >>>>> (ignitectl?) that will incorporate all the operations > (enable/disable > > >>>>> modules, start/stop, configuration updates, activation, baseline, > > etc.). > > >>>>> This IEP is a step in this direction. > > >>>>> > > >>>>> Looking forward to testing a prototype :) > > >>>>> > > >>>>> -Val > > >>>>> > > >>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr.wei...@gmail.com> > > >>> wrote: > > >>>>> > > >>>>>> Hi, Igniters! > > >>>>>> > > >>>>>> > > >>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced > > >>> delivery > > >>>>>> and upgrade processes [1]. > > >>>>>> The key idea is to make main binary as slim as possible > (delivering > > >>> every > > >>>>>> optional component by demand only) while providing way to run each > > new > > >>>>>> version seamlessly without much of the efforts migrating data and > > >>>>>> configuration between upgrades. > > >>>>>> > > >>>>>> I plan to create prototype based on current release of Apache > Ignite > > >>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> [1] > > >>>>>> > > >>>>> > > >>> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958 > > >>>>>> > > >>>>>> > > >>>>> > > >>> > > >>> > > > > > > > > -- Best regards, Alexei Scherbakov
