Hello, Val.
Sorry, but I sill don’t understand how «download jars from the internet on
production server» approach helps us to solve the issues you describe.
Moreover, with this enhancement you want to add more dependencies to Ignite and
create some security vulnerabilities:
* Maven tool - new dependency. I doubt that and of the production
server have it
* Internet maven repository - many production servers restrict outgoing
internet connection to any addresses.
It’s hard to understand why distributed DB should have the
ability to connect to some kind of address on the Internet.
* All Ignite distributions can be compromised with the hack of some
random maven repo or with the malicious proxy.
* Note, that Ignite installation or upgrade procedure can become dead
just because of some Maven repo down.
I have 2 concerns:
1. We shouldn’t download any jars from the internet during production
deployment or upgrade procedure.
2. The IEP description, for now, is quite small for such important things as a
way we distribute and upgrade Ignite.
Petr, can you, please, make IEP more specific.
I think we should add the following details:
* Description of the typical Ignite deployment procedure
including folder structure(config, persistence, .m2 and other folders)
* This also should include examples of the bash
commands if any required.
* Description of the typical upgrade(downgrade?) procedure
* This also should include examples of the bash
commands if any required.
* Description of «single tool responsible for all operations».
* all commands
* all operations
* all parameters
> 1. Most paths are resolved relative to IGNITE_HOME, which effectively forces
> (or at least motivates) users to put everything related to Ignite under
> IGNITE_HOME (configs, additional JARs, etc.).
AFAIU this issue doesn’t relate to the way Ignite distributed.
We can and should fix this outside of this IEP.
> 2. Modules are enabled/disabled manually by moving folders around… this is
> error-prone,
Makes sense.
Let’s fix this without «download jar from the internet»?
We can have some kind of config with the enabled modules.
> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you
> look at how this ZIP is built, you will notice that we don't actually include
> all transitive dependencies,
We need to fix this.
Let’s include all the required JARs to the distribution.
> which means that an embedded node that uses Maven can have a different
> classpath.
Embedded Ignite node is an error-prone design decision in the first place.
We can’t fix it with the «download jars from the internet» approach.
We should eliminate the client node as a thing and use a thin clients instead
of it.
> 4. ignite.sh allows manual modifications (there are several "Uncomment this
> to..." lines). This complicates upgrades as well.
Makes sense.
Let’s fix this.
> The whole purpose of the IEP is to fix the above, automate and simplify
> operations for the users. The general idea is to create a single tool
> responsible for all those operations and use Maven under the hood to
> include/exclude modules.
For now, IEP lacks the description of this tool.
> 28 авг. 2020 г., в 22:35, Valentin Kulichenko <valentin.kuliche...@gmail.com>
> написал(а):
>
> Hi Nikolay,
>
> Here are some of the issues that I see with the current binary ZIP package
> that we create for every release.
>
> 1. Most paths are resolved relative to IGNITE_HOME, which effectively
> forces (or at least motivates) users to put everything related to Ignite
> under IGNITE_HOME (configs, additional JARs, etc.). Now, when a user
> downloads and unzips a new version, they have to somehow merge the content
> together - that's poor upgradability.
> 2. Modules are enabled/disabled manually by moving folders around. Not only
> this is error-prone, but it also adds more issues to upgradability - you
> have to apply these steps for every version you download.
> 3. Standalone nodes use JARs prepackaged in the ZIP file. However, if you
> look at how this ZIP is built, you will notice that we don't actually
> include all transitive dependencies, which means that an embedded node that
> uses Maven can have a different classpath. It seems that we've been lucky
> so far as this hasn't caused any issues, but I still think the approach is
> wrong. The behavior should be consistent.
> 4. ignite.sh allows manual modifications (there are several "Uncomment this
> to..." lines). This complicates upgrades as well.
>
> The whole purpose of the IEP is to fix the above, automate and
> simplify operations for the users. The general idea is to create a single
> tool responsible for all those operations and use Maven under the hood to
> include/exclude modules.
>
> Exact commands and steps are still under discussion though. It looks like
> Petr is going to create a prototype that can be used as a starting point,
> but if you have any suggestions or ideas in the meantime, please share them
> with us.
>
> -Val
>
> On Fri, Aug 28, 2020 at 12:31 AM Nikolay Izhikov <nizhi...@apache.org>
> wrote:
>
>> Hello, Igniters.
>>
>> Not sure if I understand the essence of the proposal at the moment.
>> Can you, please, describe the advantages of the proposed way from the user
>> perspective?
>>
>> How the typical DevOps pipeline should look like with this enhancement?
>>
>> How I as a user can create a fully functional installation package of the
>> Ignite?
>> AFAIK downloading some artifacts from the internet straight to the
>> production server is a security anti-pattern.
>>
>>
>>> 28 авг. 2020 г., в 01:59, Denis Magda <dma...@apache.org> написал(а):
>>>
>>> Petr, thanks,
>>>
>>> There is also a collection of modules located in our extensions
>> repository:
>>> https://github.com/apache/ignite-extensions
>>>
>>> @Saikat Maitra <saikat.mai...@gmail.com> is migrating all our existing
>>> integrations to that repository and, once this is done, the extensions
>> will
>>> be released to Maven separately. Is it reasonable to expand the scope of
>>> the IEP-52 and contemplate on how to install those extensions?
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Thu, Aug 27, 2020 at 3:40 PM Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com> wrote:
>>>
>>>> Hi Petr,
>>>>
>>>> The proposal makes sense to me - thanks for starting the discussion.
>>>> Current installation and upgrade procedures involve a lot of manual
>> steps
>>>> and quite error-prone, we need to automate and simplify them as much as
>>>> possible.
>>>>
>>>> I believe we eventually should end up with a unified command-line tool
>>>> (ignitectl?) that will incorporate all the operations (enable/disable
>>>> modules, start/stop, configuration updates, activation, baseline, etc.).
>>>> This IEP is a step in this direction.
>>>>
>>>> Looking forward to testing a prototype :)
>>>>
>>>> -Val
>>>>
>>>> On Thu, Aug 27, 2020 at 2:17 AM Petr Ivanov <mr.wei...@gmail.com>
>> wrote:
>>>>
>>>>> Hi, Igniters!
>>>>>
>>>>>
>>>>> For Apache Ignite 3.0 I would like to propose vision of enhanced
>> delivery
>>>>> and upgrade processes [1].
>>>>> The key idea is to make main binary as slim as possible (delivering
>> every
>>>>> optional component by demand only) while providing way to run each new
>>>>> version seamlessly without much of the efforts migrating data and
>>>>> configuration between upgrades.
>>>>>
>>>>> I plan to create prototype based on current release of Apache Ignite
>>>>> (2.8.1 or 2.9.0 if it will be released soon) later in September.
>>>>>
>>>>>
>>>>>
>>>>> [1]
>>>>>
>>>>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>>>>>
>>>>>
>>>>
>>
>>
