Re: [DISCUSS] Ignite 3.0 development approach

[Carbone, Adam]

I don't believe Java 7 was LTS, and I hope that others will have upgraded long 
before that. If that is the release timeframe for 3.0, then I suppose that 
would makes sense, I would still doubt that people would be going newer than 
java 11, just my opinion of what I'm seeing.

Regards
~adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 12/15/20, 4:25 AM, "Ilya Kasnacheev"  wrote:

Hello!

I guess Ignite 3.0 will be ready for production use somewhere in 2022,
realistically. By that time, Java 8 will be long enough out of support so
that most companies will actually forbid its use, WRT vulnerabilities et
all.

After all we have managed to upgrade from Java 7 to Java 8 and only got a
minor amount of complaints.

Regards,
-- 
Ilya Kasnacheev


пн, 14 дек. 2020 г. в 19:06, Carbone, Adam :

> So just one bit to consider... Are there features that you need to use in
> these newer versions of java? Or are we just updating to stay current? The
> reason I ask is that there are still lots of people in an enterprise space
> that are beholden to having to support legacy JAVAEE supported 
applications
> on Websphere, Weblogic, Redhat, etc... and the organizations that use 
those
> platforms are slow to move... Most of them are still on Java8
>
> So as a platform I think a strong consideration needs to be towards having
> the broadest possible support profile until it becomes an impediment to
> doing things that the platform needs. So far I haven't seen huge things in
> the newer versions of java that are must haves ( a few exceptions are
> things that would be really nice to take advantage of ).
>
> I think that apache commons has taken the right approach by staying on
> java 8 giving the largest possible user base.
>
> Even standardizing on java 11 would have to make us reconsider Ignite as
> the platform we are using, we are not so invested in it as of now, even
> though we have big plans to leverage it. Just because we aren't sure when
> we are going to be able to upgrade from java8. It has support through 
2022,
> so I imagine that is when we will be discussing that.
>
> Regards
>
> ~Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 11/24/20, 7:38 AM, "Alexey Zinoviev"  wrote:
>
> Java 15 is better, VarHandles, ForeignMemory access and so on.
>
> In both cases, I support the Java version 11 and higher for the
> development!
>
> вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov <
> andrey.mashen...@gmail.com>:
>
> > Let's add maven plugins  for sanity checks at the early stage.
> > I've created a ticket for this [1].
> >
> > Also, I've found initial pom.xml has a target version Java 8.
> > Do we intend to move to Java 11 (or may be next LTS) and drop Java 8
> in
> > Ignite 3.0?
> >
> > [1]
> 
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
> >
> > On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Folks,
> > >
> > > I went ahead and created the repository [1]. I also configured a
> TeamCity
> > > project [2] that runs all available JUnit tests on every PR
> creation or
> > > update. It also sends the status update to GitHub so that it's
> reflected
> > in
> > > the PR itself so that we can do merges directly from GitHub. Basic
> steps
> > to
> > > make a change are described on the Wiki page [3].
> > >
> > > Let me know if you have any questions.
> > >
> > > [1]
> 
https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
> > > [2]
> 
https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
> > > [3]
> > >
> > >
  

Re: [DISCUSS] Ignite 3.0 development approach

[Carbone, Adam]

So just one bit to consider... Are there features that you need to use in these 
newer versions of java? Or are we just updating to stay current? The reason I 
ask is that there are still lots of people in an enterprise space that are 
beholden to having to support legacy JAVAEE supported applications on 
Websphere, Weblogic, Redhat, etc... and the organizations that use those 
platforms are slow to move... Most of them are still on Java8

So as a platform I think a strong consideration needs to be towards having the 
broadest possible support profile until it becomes an impediment to doing 
things that the platform needs. So far I haven't seen huge things in the newer 
versions of java that are must haves ( a few exceptions are things that would 
be really nice to take advantage of ). 

I think that apache commons has taken the right approach by staying on java 8 
giving the largest possible user base. 

Even standardizing on java 11 would have to make us reconsider Ignite as the 
platform we are using, we are not so invested in it as of now, even though we 
have big plans to leverage it. Just because we aren't sure when we are going to 
be able to upgrade from java8. It has support through 2022, so I imagine that 
is when we will be discussing that. 

Regards

~Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 11/24/20, 7:38 AM, "Alexey Zinoviev"  wrote:

Java 15 is better, VarHandles, ForeignMemory access and so on.

In both cases, I support the Java version 11 and higher for the development!

вт, 24 нояб. 2020 г. в 15:21, Andrey Mashenkov :

> Let's add maven plugins  for sanity checks at the early stage.
> I've created a ticket for this [1].
>
> Also, I've found initial pom.xml has a target version Java 8.
> Do we intend to move to Java 11 (or may be next LTS) and drop Java 8 in
> Ignite 3.0?
>
> [1] 
https://urldefense.com/v3/__https://issues.apache.org/jira/browse/IGNITE-13751__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKAO2Ejs8$
 
>
> On Tue, Nov 24, 2020 at 5:40 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Folks,
> >
> > I went ahead and created the repository [1]. I also configured a 
TeamCity
> > project [2] that runs all available JUnit tests on every PR creation or
> > update. It also sends the status update to GitHub so that it's reflected
> in
> > the PR itself so that we can do merges directly from GitHub. Basic steps
> to
> > make a change are described on the Wiki page [3].
> >
> > Let me know if you have any questions.
> >
> > [1] 
https://urldefense.com/v3/__https://github.com/apache/ignite-3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKIq24lxF$
 
> > [2] 
https://urldefense.com/v3/__https://ci.ignite.apache.org/project/ignite3__;!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKFGL_oJx$
 
> > [3]
> >
> >
> 
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0*ApacheIgnite3.0-DevelopmentProcess__;Kysj!!O3mv9RujDHg!37ujwREhL1l-B3DmRXix6yaN1dE1KgH1Tx_tSl0eLZe4x1y0NnUlF4MzW5FeKNhWzQ0s$
 
> >
> > -Val
> >
> > On Wed, Nov 18, 2020 at 4:24 PM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Thanks, guys. It looks like we are much closer to the consensus now. I
> > > totally on board with the plan, but I would also like to address the
> > > short-term needs. As I've already mentioned earlier, there are several
> > > active IEPs, but we still don't have even a preliminary technical
> process
> > > for working on these IEPs. I believe this might be frustrating for the
> > > folks who would like to commit code.
> > >
> > > The scope we agreed on is quite big, and it will surely take
> significant
> > > time to implement all the changes and stabilize them. Therefore, it's
> > clear
> > > to me that we will have to maintain 2.x and 3.x in parallel for quite
> > some
> > > time - this needs to be addressed somehow. I'm convinced that having a
> > > separate repo is the ONLY way to do that, and so far, I haven't heard
> any
> > > clear alternatives or reasons why we shouldn't do this.
> > >
> > > That said, I'm inclined to proceed with this in the next few days - I
> > will
> > > create a repo and describe the process (which we, of course, can
> discuss
> > > and modify going forward). Let's, at the very least, try and see where
> it
> > > leads us.
> > >
> > > If someone has any concrete alternative options on how to we can
> maintain
> > > two major versions in parallel, let's have another voice discussion
> this
> > > 

Re: [DISCUSSION] Renaming Ignite's product category

[Carbone, Adam]

Good Afternoon Denis, 

We came across ignite in very similar ways to everyone else. We had our 
existing application that we were looking to ramp up and scale inside 
Kubernetes. It is an internal ML Platform that was developed largely as a 
monolithic application, we wanted work on it to be more cloud native. In order 
to accomplish that we needed to be able to split the separate services, they 
make heavy use of Spark so we liked the Shared Dataframes that ignite provided, 
we also stumbled across some of the Job Scheduling, as well as the Distributed 
locking... So when we saw that we might get all of those from a single platform 
instead of having to build/integrate with multiple parts we were intrigued. So 
far we have done a basic integration, next steps will be to really work on the 
shared spark dataframes.

Regards
~Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 9/24/20, 2:55 PM, "Denis Magda"  wrote:

Adam,

Like your way of thinking. It makes sense to me.

For me, a computing platform comprises two essential components - a storage
engine and compute APIs (not only compute grid, but also streaming, SQL,
etc.). Under this definition, Ignite fits the "compute platform" category
for sure, but the problem (and the reason why I started this discussion
thread) is that the "in-memory computing platform" is not a ubiquitous
product category. At the same time, we all know that Ignite is used to
speed things up by storing and executing in memory. And if you need to make
software faster, you usually search for something scalable or in-memory
(in-memory caches, in-memory, NoSQL, distributed databases). Basically,
most of the folks search for "caches" and "in-memory database" which is
related to Ignite and only the tip of the iceberg searches for "in-memory
computing platforms" and "data grids".

Btw, how has your story started with Ignite? How did you find out about it?

-
Denis


On Wed, Sep 23, 2020 at 1:24 PM Carbone, Adam 
wrote:

> Good Evening Denis,
>
> I’m sure there are more out there as well,  I would consider platforms
> that the entire applications but that all the execution of code happens
> exclusively on the platform, and most of the applications are written to
> run in, not connected to the platform.
>
> Hmmm by this criteria k8s could possible fit the bill…
>
> Spark might even be considered a compute grid as well but it is not a
> generic compute framework and people don’t usually run there whole
> applications inside.
>
> What is the vision for the platform? That might help in this discussion,
> set your category with the direction you are heading, and what you are
> trying to achieve.
>
> Regards
>
>
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
    >
>
>
>
>
>
> *From: *Denis Magda 
> *Date: *Wednesday, September 23, 2020 at 4:14 PM
> *To: *dev , "Carbone, Adam" <
> adam.carb...@bottomline.com>
> *Cc: *"u...@ignite.apache.org" 
> *Subject: *Re: [DISCUSSION] Renaming Ignite's product category
>
>
>
> Adam,
>
>
>
> You defined GigaSpaces as a true in-memory computing platform. What is the
> true platform for you?
>
>
>
>
> -
>
> Denis
>
>
>
>
>
> On Fri, Sep 18, 2020 at 7:02 AM Carbone, Adam 

> wrote:
>
> So when I came across Ignite It was described as an In Memory Data Grid
>
> So one way to look at this is who do you fashion as Ignite competing
> against?
>
> Are competing against Redis, Aerospike - In Memory Databases
>
> Or are you more competing with
>
> Gigaspaces - True In memory Compute platform
>
> And then you have like of
>
> Hazelcast that started as a Distributed Hash and have gained some
> features...
>
> On thing that I think is a differentiator that isn't being highlighted but
> Is  unique feature to Ignited, and the primary reason we ended up here; 
The
> integration with spark and it's distributed/shared Datasets/Dataframes.
>
> I don't know for me the In Memory Data Grid I think fits what Ignite is...
>
> Regards
>
> ~Adam
>
> Adam Carbone |

Re: [DISCUSSION] Renaming Ignite's product category

[Carbone, Adam]

Good Evening Denis,

I’m sure there are more out there as well,  I would consider platforms that the 
entire applications but that all the execution of code happens exclusively on 
the platform, and most of the applications are written to run in, not connected 
to the platform.

Hmmm by this criteria k8s could possible fit the bill…

Spark might even be considered a compute grid as well but it is not a generic 
compute framework and people don’t usually run there whole applications inside.

What is the vision for the platform? That might help in this discussion, set 
your category with the direction you are heading, and what you are trying to 
achieve.

Regards

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com



From: Denis Magda 
Date: Wednesday, September 23, 2020 at 4:14 PM
To: dev , "Carbone, Adam" 
Cc: "u...@ignite.apache.org" 
Subject: Re: [DISCUSSION] Renaming Ignite's product category

Adam,

You defined GigaSpaces as a true in-memory computing platform. What is the true 
platform for you?


-
Denis


On Fri, Sep 18, 2020 at 7:02 AM Carbone, Adam 
mailto:adam.carb...@bottomline.com>> wrote:
So when I came across Ignite It was described as an In Memory Data Grid

So one way to look at this is who do you fashion as Ignite competing against?

Are competing against Redis, Aerospike - In Memory Databases

Or are you more competing with

Gigaspaces - True In memory Compute platform

And then you have like of

Hazelcast that started as a Distributed Hash and have gained some features...

On thing that I think is a differentiator that isn't being highlighted but Is  
unique feature to Ignited, and the primary reason we ended up here; The 
integration with spark and it's distributed/shared Datasets/Dataframes.

I don't know for me the In Memory Data Grid I think fits what Ignite is...

Regards

~Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com<http://www.bottomline.com>



On 9/17/20, 11:45 AM, "Glenn Wiebe" 
mailto:glenn.wi...@gridgain.com>> wrote:

I agree with Stephen about "database" devaluing what Ignite can do (though
it probably hits the majority of existing use cases). I tend to go with
"massively distributed storage and compute platform"

I know, I didn't take sides, I just have both.

Cheers,
  Glenn

On Thu., Sep. 17, 2020, 7:04 a.m. Stephen Darlington, <
stephen.darling...@gridgain.com<mailto:stephen.darling...@gridgain.com>> 
wrote:

> I think this is a great question. Explaining what Ignite does is always a
> challenge, so having a useful “tag line” would be very valuable.
>
> I’m not sure what the answer is but I think calling it a “database”
> devalues all the compute facilities. "Computing platform” may be too vague
> but it at least says that we do more than “just” store data.
>
> On 17 Sep 2020, at 06:29, Valentin Kulichenko <
> valentin.kuliche...@gmail.com<mailto:valentin.kuliche...@gmail.com>> 
wrote:
>
> My vote is for the "distributed memory-first database". It clearly states
> that Ignite is a database (which is true at this point), while still
> emphasizing the in-memory computing power endorsed by the platform.
>
> The "in-memory computing platform" is an ambiguous term and doesn't really
> reflect what Ignite is, especially in its current state.
>
> -Val
>
> On Wed, Sep 16, 2020 at 3:53 PM Denis Magda 
mailto:dma...@apache.org>> wrote:
>
>> Igniters,
>>
>> Throughout the history of our project, we could see how the addition of
>> certain features required us to reassess the project's name and category.
>>
>> Before Ignite joined the ASF, it supported only compute APIs resembling
>> the
>> MapReduce engine of Hadoop. Those days, it was fair to define Ignite as 
"a
>> distributed in-memory computing engine". Next, at the time of the project
>> donation, it already included key-value/SQL/transactional APIs, was used
>> as
>> a distributed cache, and significantly outgrew the "in-memory computing
>> engine" use case. That's how the project transitioned to the product
>> category of in-memory caches and we started to name it as an "in-memory
>> data grid" or "in-memory computing platform" to differentiate from
>> classical caching products such as Memcached and Redis.
>>
>> Nowadays, the project outgrew its caching use case, and the 
classification
  

Re: [DISCUSSION] Renaming Ignite's product category

[Carbone, Adam]

So when I came across Ignite It was described as an In Memory Data Grid 

So one way to look at this is who do you fashion as Ignite competing against?

Are competing against Redis, Aerospike - In Memory Databases

Or are you more competing with 

Gigaspaces - True In memory Compute platform

And then you have like of 

Hazelcast that started as a Distributed Hash and have gained some features...

On thing that I think is a differentiator that isn't being highlighted but Is  
unique feature to Ignited, and the primary reason we ended up here; The 
integration with spark and it's distributed/shared Datasets/Dataframes. 

I don't know for me the In Memory Data Grid I think fits what Ignite is... 

Regards

~Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 9/17/20, 11:45 AM, "Glenn Wiebe"  wrote:

I agree with Stephen about "database" devaluing what Ignite can do (though
it probably hits the majority of existing use cases). I tend to go with
"massively distributed storage and compute platform"

I know, I didn't take sides, I just have both.

Cheers,
  Glenn

On Thu., Sep. 17, 2020, 7:04 a.m. Stephen Darlington, <
stephen.darling...@gridgain.com> wrote:

> I think this is a great question. Explaining what Ignite does is always a
> challenge, so having a useful “tag line” would be very valuable.
>
> I’m not sure what the answer is but I think calling it a “database”
> devalues all the compute facilities. "Computing platform” may be too vague
> but it at least says that we do more than “just” store data.
>
> On 17 Sep 2020, at 06:29, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> My vote is for the "distributed memory-first database". It clearly states
> that Ignite is a database (which is true at this point), while still
> emphasizing the in-memory computing power endorsed by the platform.
>
> The "in-memory computing platform" is an ambiguous term and doesn't really
> reflect what Ignite is, especially in its current state.
>
> -Val
>
> On Wed, Sep 16, 2020 at 3:53 PM Denis Magda  wrote:
>
>> Igniters,
>>
>> Throughout the history of our project, we could see how the addition of
>> certain features required us to reassess the project's name and category.
>>
>> Before Ignite joined the ASF, it supported only compute APIs resembling
>> the
>> MapReduce engine of Hadoop. Those days, it was fair to define Ignite as 
"a
>> distributed in-memory computing engine". Next, at the time of the project
>> donation, it already included key-value/SQL/transactional APIs, was used
>> as
>> a distributed cache, and significantly outgrew the "in-memory computing
>> engine" use case. That's how the project transitioned to the product
>> category of in-memory caches and we started to name it as an "in-memory
>> data grid" or "in-memory computing platform" to differentiate from
>> classical caching products such as Memcached and Redis.
>>
>> Nowadays, the project outgrew its caching use case, and the 
classification
>> of Ignite as an "in-memory data grid" or "in-memory computing platform"
>> doesn't sound accurate. We rebuilt our storage engine by replacing a
>> typical key-value engine with a B-tree engine that spans across memory 
and
>> disk tiers. And it's not surprising to see more deployments of Ignite as 
a
>> database on its own. So, it feels like we need to reconsider Ignite
>> positioning again so that a) application developers can discover it 
easily
>> via search engines and b) the project can stand out from in-memory
>> projects
>> with intersecting capabilities.
>>
>> To the point, I'm suggesting to reposition Ignite in one of the following
>> ways:
>>
>>1. Ignite is a "distributed X database". We are indeed a distributed
>>partitioned database where X can be "multi-tiered" or "memory-first" 
to
>>emphasize that we are more than an in-memory database.
>>2. Keep defining Ignite as "an in-memory computing platform" but name
>>our storage engine uniquely as "IgniteDB" to highlight that the
>> platform is
>>powered by a "distributed multi-tiered/memory-first database".
>>
>> What is your thinking?
>>
>>
>> (Also, regardless of a selected name, Ignite still will be used as a 
cache
>> and grid, and we're not going to stop appealing to those use cases. But
>> those are just use cases while Ignite has to figure out its new identity
>> ... again).
>>
>>
>> -
>> Denis
>>
>
>
>

Re: Apache Ignite 3.0

[Carbone, Adam]

If you want to make is simpler to have the components that you want, but have 
that be immutable at install time you could take an approach similar to the way 
spring does it with their initializer ( https://start.spring.io/ )  as an 
example... Basically the concept being something that produces a set of 
configurations that are used to define what the environment looks like ( these 
could be k8s objects ) they could be spring-configuration objects? They could 
be something that you develop all upon ignite ( probably wouldn’t recommend 
this approach )  there seems to be plenty of these types of things already 

Example
* https://spring.io/guides/gs/centralized-configuration 
* 
https://cocoon.apache.org/subprojects/configuration/spring-configurator/index.html

And I'm by no means saying to use spring these are just examples that I came 
across

I'm thinking the outcome needs to be a platform config of source ( that can be 
checked in for those doing gitops ) and maybe ends up as a config map for those 
doing k8s, and some other fashion for those doing something else. 

Honestly I am not deep enough into the internals of ignite to know how this 
might work for the platform, was more describing what I'm seeing from a bigger 
picture trend 

Regards 

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 8/13/20, 7:55 PM, "Valentin Kulichenko"  
wrote:

Hi Ilya,

Can you please describe your vision of how it should work?

Let's say, I want to set up a cluster of several standalone server nodes
with a couple of optional modules enabled. What are my steps?

-Val

On Thu, Aug 13, 2020 at 6:03 AM Carbone, Adam 
wrote:

> Good Morning from the EastCoast
>
> I have to agree that the larger industry is tending towards immutability,
> and that you build once and test, then you promote/migrate that immutable
> binary object, be is a library or a docker image etc... however there are
> still patterns that allow you to determine at install/or deployment time (
> helm as an example, you choose based on your values what the package
> installs/provides ) It just isn't decided at runtime but install and often
> in a gitops type world that is determined by configuration as code. I 
think
> run time is difficult to manage especially in our increasingly
> containerized world.
>
> Regards.
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 8/13/20, 8:01 AM, "Ilya Kasnacheev"  wrote:
>
> Hello!
>
> On the contrary, I would suggest that apache2 way was outdated even at
> times when apache was all rage.
>
> Now the nginx approach is prevalent: on devops phase, assemble a 
custom
> bundle with all plugins included, store it somewhere, and ship it to
> production as a whole to remove any on-the-fly uncertainty from
> production.
>
> This is what docker does, but also maven, which downloads dependencies
> during build. You do not need to download anything in runtime, except
> for
> experimental deployments. You need to be all set before runtime 
starts.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> ср, 12 авг. 2020 г. в 09:48, Petr Ivanov :
>
> > Hi, Val.
> >
> > > On 12 Aug 2020, at 01:31, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> > >
> > > Hi Petr,
> > >
> > > I agree -- we should better modularize the platform. The current
> way if
> > very error-prone, especially during upgrades -- any changes made
> within
> > IGNITE_HOME (configs, scripts, modules, etc.) must be merged with a
> new
> > version of the package. There is no standard way of doing this.
> > >
> > > However, I'm a bit concerned with your suggestion regarding custom
> > dependency management. Can you please elaborate on how you think it
> should
> > work? Are there tools we can reuse for this purpose? I would try to
> avoid
> > reinventing the wheel.
> >
> > I see it as a a2enmod | 2dismod analog of Apache2.
> >
> > We build and store Apache Ignite and its modules as separate 
binaries
> > (binary per module) then use custom script th

Re: Apache Ignite 3.0

[Carbone, Adam]

Good Morning from the EastCoast 

I have to agree that the larger industry is tending towards immutability, and 
that you build once and test, then you promote/migrate that immutable binary 
object, be is a library or a docker image etc... however there are still 
patterns that allow you to determine at install/or deployment time ( helm as an 
example, you choose based on your values what the package installs/provides ) 
It just isn't decided at runtime but install and often in a gitops type world 
that is determined by configuration as code. I think run time is difficult to 
manage especially in our increasingly containerized world. 

Regards.

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 8/13/20, 8:01 AM, "Ilya Kasnacheev"  wrote:

Hello!

On the contrary, I would suggest that apache2 way was outdated even at
times when apache was all rage.

Now the nginx approach is prevalent: on devops phase, assemble a custom
bundle with all plugins included, store it somewhere, and ship it to
production as a whole to remove any on-the-fly uncertainty from production.

This is what docker does, but also maven, which downloads dependencies
during build. You do not need to download anything in runtime, except for
experimental deployments. You need to be all set before runtime starts.

Regards,
-- 
Ilya Kasnacheev


ср, 12 авг. 2020 г. в 09:48, Petr Ivanov :

> Hi, Val.
>
> > On 12 Aug 2020, at 01:31, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > Hi Petr,
> >
> > I agree -- we should better modularize the platform. The current way if
> very error-prone, especially during upgrades -- any changes made within
> IGNITE_HOME (configs, scripts, modules, etc.) must be merged with a new
> version of the package. There is no standard way of doing this.
> >
> > However, I'm a bit concerned with your suggestion regarding custom
> dependency management. Can you please elaborate on how you think it should
> work? Are there tools we can reuse for this purpose? I would try to avoid
> reinventing the wheel.
>
> I see it as a a2enmod | 2dismod analog of Apache2.
>
> We build and store Apache Ignite and its modules as separate binaries
> (binary per module) then use custom script that will know where to 
download
> necessary module. Or possibly use modified ignite.sh to specify required
> optional libs in run command while ignite.sh will download everything
> missing from known storage.
>
> The whole idea is in storing everything remotely and download on demand,
> not have all libs locally from the start.
>
>
> >
> > -Val
> >
> > On Sun, Aug 9, 2020 at 11:25 PM Petr Ivanov  > wrote:
> > Hi, Val!
> > Thanks for your efforts on this endeavour!
> >
> >
> > I would like to suggest deliveries changes in Apache Ignite 3.0:
> >  — modularised  binary delivery — single minimal binary for starting
> Ignite and all other modules and parts of the project (benchmarks,
> examples, etc.) packed in their own binary which can be added via custom
> dependency management tool (i.e. modules.sh)
> >  — same distribution for RPM and DEB packages but with modules packed as
> separate ones (PHP for example)
> >  — separate thin client release cycle with custom versioning
> > Possibly, we can we add additional section to the document you
> introduced for this part.
> >
> > Also, it seems that full JDK11 support (including building) would be a
> huge milestone and a sign of healthy modern project that tends to be on 
the
> verge of mainstream technologies and not the stockpile of legacy leftovers
> (fully support Iliya in removing all that was deprecated and/or marked as
> unused anymore).
> >
> >
> > > On 8 Aug 2020, at 02:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com >
> wrote:
> > >
> > > Igniters,
> > >
> > > I've created the page:
> > > 
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0__;Kys!!O3mv9RujDHg!2GlQzPzSAyxjW5tzyIzjaVVuR5_U_s65MCFLww8yIHRMzDqSrm5C2nkXYQErpm9uWJo_$
  <
> 
https://urldefense.com/v3/__https://cwiki.apache.org/confluence/display/IGNITE/Apache*Ignite*3.0__;Kys!!O3mv9RujDHg!2GlQzPzSAyxjW5tzyIzjaVVuR5_U_s65MCFLww8yIHRMzDqSrm5C2nkXYQErpm9uWJo_$
 >
> > >
> > > That's not everything I have in mind, but I believe there is already a
> lot
> > > to talk about :)
> > >
> > > Please take a look let me know if you have any concerns, objections, 
or
> > > questions. Once we reach the consensus on the proposed changes, I will
  

Re: Ignite XGBoost support

[Carbone, Adam]

You mentioned "I’m currently working to integrate the ignite distributed 
dataframes", what dataframes are you referring to? Could you share a link to 
docs, for example? We have no official term "the ignite distributed dataframes".

I guess the term that the community uses is shared dataframes
https://apacheignite-fs.readme.io/docs/ignite-data-frame

As far as XGBoost, seems as though the platform isn’t really working on 
training, just inference. My teammate did say he “needs to investigate Ignite 
for distributed training for both XGBoost as well as TF (tensorflow). And, I 
have been fiddling around with it” so I will keep you informed.

Regards

~Adam


From: Alexey Zinoviev 
Date: Thursday, April 9, 2020 at 5:59 AM
To: "Carbone, Adam" 
Cc: "dev@ignite.apache.org" 
Subject: Re: Ignite XGBoost support

Morning!
I'm going to publish the roadmap for Ignite ML 2.9 for wide discussion at the 
end of April when the dates of 2.9 will be finished.

But I suppose that we will closer to the this point "the plan to rely on models 
trained elsewhere and then imported into the platform for scoring?", I mean 
distributed inference, hope to increase the amount of integrated models and 
model formats.

You mentioned "I’m currently working to integrate the ignite distributed 
dataframes", what dataframes are you referring to? Could you share a link to 
docs, for example? We have no official term "the ignite distributed dataframes".

If you get some results about integration with XGBoost, please let me know!

Sincerely yours,
 Alexey

пт, 27 мар. 2020 г. в 16:29, Carbone, Adam 
mailto:adam.carb...@bottomline.com>>:
Good Morning Alexey,

Let me first answer your questions.

1. Are you a member of XGBoost project and have a permission to commit the 
XGBoost project? (in many cases the collaboration involves changes in both 
integrated frameworks)
No I am not personally, nor is our organization.
2. What are the primitives or integration points are accessible in XGBoost? 
Could you share a paper/article/link to give me a chance to read more?
Not sure that we the person on my team doing this work, has that level of 
understanding yet. Like I mentioned in the previous email we were about to 
embark on this when we saw the 2.8 announcement come out and decided to look 
further at what the level of support was.
3. What is planned architecture with native C++ libraries? Could you share it 
with me and Ignite community?
I will only be able to share the higher level on this ( again if it makes sense 
we could do a deeper dive with the developers that are working on this 
directly), but currently our Tensorflow Neural Network modeling is exposed via 
some internal webservices written in C++ wrapping the tensorflow libraries. 
This is  called within our job scheduling/runner framework. These webservices 
run on different images within our overall system. We were looking to do 
something similar around XGBoost prior to seeing it come up in the announcement.



So it looks like you are using MLeap to import and support external models we 
have looked at the same approach ourselves. From what you mentioned it seems 
that there are currently no intentions to add distributed training of any 
external Algorithms to the platform, are you developing your own algorithms? Or 
is the plan to rely on models trained elsewhere and then imported into the 
platform for scoring? Just interested in the ways that we may be able to 
leverage the platform or help contribute, we are looking to use other features 
of ignite so leveraging additional features over time seems like the right 
approach. I’m currently working to integrate the ignite distributed dataframes.

Regards

Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com<http://www.bottomline.com>



From: Alexey Zinoviev mailto:zaleslaw@gmail.com>>
Date: Friday, March 27, 2020 at 1:58 AM
To: "Carbone, Adam" 
mailto:adam.carb...@bottomline.com>>
Cc: "dev@ignite.apache.org<mailto:dev@ignite.apache.org>" 
mailto:dev@ignite.apache.org>>
Subject: Re: Ignite XGBoost support

Morning, Adam, Denis!

Let me describe the current status

1. https://issues.apache.org/jira/browse/IGNITE-10810 is related to MLeap not 
to XGBoost. This is the right ticket for XGBoost 
https://issues.apache.org/jira/browse/IGNITE-10289
2. Currently, we have no plans to add XGBoost or any external ML library for 
distributed training (inference could be supported now with a few limitations, 
see XGBoost or H2O examples)
3. We have models storage and partitioned dataset primitives to keep the data 
with MapReduce-like operations, but each algorithm should be implemented as a 
sequence of MR operations manually (we have no MR code generation here)

I have a few questions, could 

Re: Ignite XGBoost support

[Carbone, Adam]

Good afternoon Denis, 

Nice to meet you, Hello to you too Alexey. So I'm not sure if it will be me or 
another member on our team, but I wanted to start the discussion.  We are 
investigating/integrating ignite into our ML platform. In addition We have 
already done a separate tensor flow implementation for Neural Network using the 
C++ libraries. And we were about to take the same approach for XGBoost, when we 
saw the 2.8 announcement. So before we went that route I wanted to do a more 
proper investigations as to where things were, and where they might head.

Regards

Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 3/26/20, 5:20 PM, "Denis Magda"  wrote:

Hi Adam, thanks for starting the thread. The contributions are
highly appreciated and we'll be glad to see you among our contributors,
especially, if it helps to make our ML library stronger.

But first things first, let me introduce you to @Alexey Zinoviev
 who is our main ML maintainer.

-
Denis


On Thu, Mar 26, 2020 at 1:49 PM Carbone, Adam 
wrote:

> Good Afternoon All
>
> I was asked to forward this here by Denis Magda. I see in the 2.8 release
> that you implemented importing of XGBoost models for distributed inference
> =>
> 
https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
> there any plans to add distributed training, We are at a cross roads of
> building on top of the C++ libraries an XGBoost solution, but if this is 
on
> the roadmap maybe we will go the ignite direction vs the pure C++, and
> maybe we might even be able to help and contribute.
    >
> Regards
>
> Adam Carbone
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>

Ignite XGBoost support

[Carbone, Adam]

Good Afternoon All

I was asked to forward this here by Denis Magda. I see in the 2.8 release that 
you implemented importing of XGBoost models for distributed inference => 
https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
 there any plans to add distributed training, We are at a cross roads of 
building on top of the C++ libraries an XGBoost solution, but if this is on the 
roadmap maybe we will go the ignite direction vs the pure C++, and maybe we 
might even be able to help and contribute.

Regards

Adam Carbone

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com