Re: Ignite XGBoost support

[Alexey Zinoviev]

Morning, Adam, Denis!

Let me describe the current status

1. https://issues.apache.org/jira/browse/IGNITE-10810 is related to MLeap
not to XGBoost. This is the right ticket for XGBoost
https://issues.apache.org/jira/browse/IGNITE-10289
2. Currently, we have no plans to add XGBoost or any external ML library
for distributed training (inference could be supported now with a few
limitations, see XGBoost or H2O examples)
3. We have models storage and partitioned dataset primitives to keep the
data with MapReduce-like operations, but each algorithm should be
implemented as a sequence of MR operations manually (we have no MR code
generation here)

I have a few questions, could you please answer them?

1. Are you a member of XGBoost project and have a permission to commit the
XGBoost project? (in many cases the collaboration involves changes in both
integrated frameworks)
2. What are the primitives or integration points are accessible in XGBoost?
Could you share a paper/article/link to give me a chance to read more?
3. What is planned architecture with native C++ libraries? Could you share
it with me and Ignite community?

P.S. I need to go deeper to understand what capabilities of Ignite ML could
be used to become the platform for distributed training, you answers will
be helpful.

Sincerely yours,
  Alexey Zinoviev

пт, 27 мар. 2020 г. в 01:04, Carbone, Adam :

> Good afternoon Denis,
>
> Nice to meet you, Hello to you too Alexey. So I'm not sure if it will be
> me or another member on our team, but I wanted to start the discussion.  We
> are investigating/integrating ignite into our ML platform. In addition We
> have already done a separate tensor flow implementation for Neural Network
> using the C++ libraries. And we were about to take the same approach for
> XGBoost, when we saw the 2.8 announcement. So before we went that route I
> wanted to do a more proper investigations as to where things were, and
> where they might head.
>
> Regards
>
> Adam
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>
> On 3/26/20, 5:20 PM, "Denis Magda"  wrote:
>
> Hi Adam, thanks for starting the thread. The contributions are
> highly appreciated and we'll be glad to see you among our contributors,
> especially, if it helps to make our ML library stronger.
>
> But first things first, let me introduce you to @Alexey Zinoviev
>  who is our main ML maintainer.
>
> -
> Denis
>
>
> On Thu, Mar 26, 2020 at 1:49 PM Carbone, Adam <
> adam.carb...@bottomline.com>
> wrote:
>
> > Good Afternoon All
> >
> > I was asked to forward this here by Denis Magda. I see in the 2.8
> release
> > that you implemented importing of XGBoost models for distributed
> inference
> > =>
> >
> https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
> > there any plans to add distributed training, We are at a cross roads
> of
> > building on top of the C++ libraries an XGBoost solution, but if
> this is on
> > the roadmap maybe we will go the ignite direction vs the pure C++,
> and
> > maybe we might even be able to help and contribute.
> >
> > Regards
> >
> > Adam Carbone
> >
> > Adam Carbone | Director of Innovation – Intelligent Platform Team |
> > Bottomline Technologies
> > Office: 603-501-6446 | Mobile: 603-570-8418
> > www.bottomline.com
> >
> >
> >
>
>
>

GridKernalContextImpl.isDaemon is calling System.Properties frequently

[Sunny Chan, CLSA]

Hello,

I was performing some tuning using JFR with our application, and I have found 
that in many of the cache get it makes a lot of calls to get System Properties, 
which is a synchronized call. Upon further examination of the code I have found 
that we can potentially optimized the code to cache the results.

I have raised a bug report (https://issues.apache.org/jira/browse/IGNITE-12842) 
and proposed a patch (https://github.com/apache/ignite/pull/7579). Please 
review this and let me know whether this is acceptable

Sunny Chan
Senior Lead Engineer, Executive Services
D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

[:1. Social Media Icons:CLSA_Social Media 
Icons_linkedin.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_twitter.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_youtube.png][:1.
 Social Media Icons:CLSA_Social Media 
Icons_facebook.png]

clsa.com
Insights. Liquidity. Capital.

[CLSA_RGB]

A CITIC Securities Company

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

[jira] [Created] (IGNITE-12842) Cache "IGNITE_DAEMON" system properties in isDaemon call

[Sunny Chan (Jira)]

Sunny Chan created IGNITE-12842:
---

 Summary: Cache "IGNITE_DAEMON" system properties in isDaemon call
 Key: IGNITE-12842
 URL: https://issues.apache.org/jira/browse/IGNITE-12842
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.7.6
Reporter: Sunny Chan


In our performance tuning exercise using JFR, we have observed that when 
accessing the cache, Ignite will repeatedly called System.getProperty in 
GridKernalContextImpl.isDaemon()

!isDaemon.jpg! This is generated with the earlier version of Ignite, and when I 
examine the latest code it also behaving the same way



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Ignite XGBoost support

[Carbone, Adam]

Good afternoon Denis, 

Nice to meet you, Hello to you too Alexey. So I'm not sure if it will be me or 
another member on our team, but I wanted to start the discussion.  We are 
investigating/integrating ignite into our ML platform. In addition We have 
already done a separate tensor flow implementation for Neural Network using the 
C++ libraries. And we were about to take the same approach for XGBoost, when we 
saw the 2.8 announcement. So before we went that route I wanted to do a more 
proper investigations as to where things were, and where they might head.

Regards

Adam

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com
 
 

On 3/26/20, 5:20 PM, "Denis Magda"  wrote:

Hi Adam, thanks for starting the thread. The contributions are
highly appreciated and we'll be glad to see you among our contributors,
especially, if it helps to make our ML library stronger.

But first things first, let me introduce you to @Alexey Zinoviev
 who is our main ML maintainer.

-
Denis


On Thu, Mar 26, 2020 at 1:49 PM Carbone, Adam 
wrote:

> Good Afternoon All
>
> I was asked to forward this here by Denis Magda. I see in the 2.8 release
> that you implemented importing of XGBoost models for distributed inference
> =>
> 
https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
> there any plans to add distributed training, We are at a cross roads of
> building on top of the C++ libraries an XGBoost solution, but if this is 
on
> the roadmap maybe we will go the ignite direction vs the pure C++, and
> maybe we might even be able to help and contribute.
>
> Regards
>
> Adam Carbone
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>

Re: Ignite XGBoost support

[Denis Magda]

Hi Adam, thanks for starting the thread. The contributions are
highly appreciated and we'll be glad to see you among our contributors,
especially, if it helps to make our ML library stronger.

But first things first, let me introduce you to @Alexey Zinoviev
 who is our main ML maintainer.

-
Denis


On Thu, Mar 26, 2020 at 1:49 PM Carbone, Adam 
wrote:

> Good Afternoon All
>
> I was asked to forward this here by Denis Magda. I see in the 2.8 release
> that you implemented importing of XGBoost models for distributed inference
> =>
> https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
> there any plans to add distributed training, We are at a cross roads of
> building on top of the C++ libraries an XGBoost solution, but if this is on
> the roadmap maybe we will go the ignite direction vs the pure C++, and
> maybe we might even be able to help and contribute.
>
> Regards
>
> Adam Carbone
>
> Adam Carbone | Director of Innovation – Intelligent Platform Team |
> Bottomline Technologies
> Office: 603-501-6446 | Mobile: 603-570-8418
> www.bottomline.com
>
>
>

Ignite XGBoost support

[Carbone, Adam]

Good Afternoon All

I was asked to forward this here by Denis Magda. I see in the 2.8 release that 
you implemented importing of XGBoost models for distributed inference => 
https://issues.apache.org/jira/browse/IGNITE-10810?focusedCommentId=16728718=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16728718Is
 there any plans to add distributed training, We are at a cross roads of 
building on top of the C++ libraries an XGBoost solution, but if this is on the 
roadmap maybe we will go the ignite direction vs the pure C++, and maybe we 
might even be able to help and contribute.

Regards

Adam Carbone

Adam Carbone | Director of Innovation – Intelligent Platform Team | Bottomline 
Technologies
Office: 603-501-6446 | Mobile: 603-570-8418
www.bottomline.com

Re: Thin client: compute support

[Denis Magda]

>
> Deployment API definitely needed as one of the next steps. Currently, we
> are talking only about the first step (execution of already deployed
> tasks).
> Also, I'm not sure about automatic redeploy and peer-class-loading for thin
> clients, I think it's better to have more control here and provide API to
> explicitly deploy classes or jar files. WDYT?


Alex, agree that automatic redeployment is better suited for the management
APIs. How about adding this capability to our command-line tool
(control.sh, or visor cmd, or one new holistic tool).

-
Denis


On Wed, Mar 25, 2020 at 1:04 PM Alex Plehanov 
wrote:

> Pavel,
>
> 1. Actually it can be solved on the client-side (and already solved in PoC
> implementation). But I agreed it brings extra complexity for client-side
> implementation, will try to provide such guarantees on the server-side.
> 2. ComputeTask has also "reduce" step which is executed on the initiator
> node. Binary-rest client implementation, for example, has such affinity
> methods (to execute the task by name). I'm ok with removing it. At least if
> someone will need it we can implement it again at any time in the future
> without protocol change.
> I've fixed IEP.
>
> Denis,
>
> Deployment API definitely needed as one of the next steps. Currently, we
> are talking only about the first step (execution of already deployed
> tasks).
> Also, I'm not sure about automatic redeploy and peer-class-loading for thin
> clients, I think it's better to have more control here and provide API to
> explicitly deploy classes or jar files. WDYT?
>
> ср, 25 мар. 2020 г. в 21:17, Denis Magda :
>
> > Alex, thanks for preparing the outline.
> >
> > I'd like us to discuss an approach for compute tasks update with no
> > downtimes on the servers' end. For instance, let's assume that a
> > Python/C++/Node.JS developer requested to update a compute task he called
> > from the app. Should we introduce some system level API to the binary
> > protocol that can take a jar file (or class) and redeploy it
> automatically
> > with the usage of peer-class-loading?
> >
> > -
> > Denis
> >
> >
> > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov 
> > wrote:
> >
> > > Hello guys.
> > >
> > > I've implemented PoC and created IEP [1] for thin client compute grid
> > > functionality. Please have a look.
> > >
> > > [1]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-42+Thin+client%3A+compute+support
> > >
> > > пт, 24 янв. 2020 г. в 16:56, Alex Plehanov :
> > >
> > > > We've discussed thin client compute protocol with Pavel Tupitsyn and
> > Igor
> > > > Sapego and come to the conclusion that approach with two-way requests
> > > > should be used: client generates taskId and send a request to the
> > server
> > > to
> > > > execute a task. The server responds that the request has been
> accepted.
> > > > After task has finished the server notifies the client (send a
> request
> > > > without waiting for a response). The client can cancel the task by
> > > sending
> > > > a corresponding request to the server.
> > > >
> > > > Also, a node list should be passed (optionally) with a request to
> limit
> > > > nodes to execute the task.
> > > >
> > > > I will create IEP and file detailed protocol changes shortly.
> > > >
> > > > вт, 21 янв. 2020 г. в 18:46, Alex Plehanov  >:
> > > >
> > > >> Igor, thanks for the reply.
> > > >>
> > > >> > Approach with taskId will require a lot of changes in protocol and
> > > thus
> > > >> more "heavy" for implementation
> > > >> Do you mean approach with server notifications mechanism? Yes, it
> will
> > > >> require a lot of changes. But in most recent messages we've
> discussed
> > > with
> > > >> Pavel approach without server notifications mechanism. This approach
> > > have
> > > >> the same complexity and performance as an approach with requestId.
> > > >>
> > > >> > But such clients as Python, Node.js, PHP, Go most probably won't
> > have
> > > >> support for this API, at least for now.
> > > >> Without a server notifications mechanism, there will be no breaking
> > > >> changes in the protocol, so client implementation can just skip this
> > > >> feature and protocol version and implement the next one.
> > > >>
> > > >> > Or never.
> > > >> I think it still useful to execute java compute tasks from non-java
> > thin
> > > >> clients. Also, we can provide some out-of-the-box java tasks, for
> > > example
> > > >> ExecutePythonScriptTask with python compute implementation, which
> can
> > > run
> > > >> python script on server node.
> > > >>
> > > >> > So, maybe it's a good time for us to change our backward
> > compatibility
> > > >> mechanism from protocol versioning to feature masks?
> > > >> I like the idea with feature masks, but it will force us to support
> > both
> > > >> backward compatibility mechanisms, protocol versioning and feature
> > > masks.
> > > >>
> > > >> пн, 20 янв. 2020 г. в 20:34, Pavel Tupitsyn :
> > > >>
> > > >>> Huge +1 from me for 

Re: FetchSize of the JdbcStore

[Nikolay Izhikov]

Hello, Ivan.

Yes, this is the same issue.
I will proceed with this improvement if there is no objections.

чт, 26 мар. 2020 г., 19:40 Ivan Pavlukhin :

> Hi Nikolay,
>
> There is a very similar ticket [1]. Perhaps it could be somehow useful.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-11894
>
> Best regards,
> Ivan Pavlukhin
>
> чт, 26 мар. 2020 г. в 17:54, Nikolay Izhikov :
> >
> > Hello, Igniters.
> >
> > Right now there is no way for the user to set fetchSize of the JdbcStore
> > This leads to the poor performance of the cache load with the 3-d party
> persistence [1]
> >
> > I propose to add JdbcDialect#setFetchSize method [2]
> >
> > Thoughts?
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-12826
> > [2] https://github.com/apache/ignite/pull/7577/files
>

Re: FetchSize of the JdbcStore

[Ivan Pavlukhin]

Hi Nikolay,

There is a very similar ticket [1]. Perhaps it could be somehow useful.

[1] https://issues.apache.org/jira/browse/IGNITE-11894

Best regards,
Ivan Pavlukhin

чт, 26 мар. 2020 г. в 17:54, Nikolay Izhikov :
>
> Hello, Igniters.
>
> Right now there is no way for the user to set fetchSize of the JdbcStore
> This leads to the poor performance of the cache load with the 3-d party 
> persistence [1]
>
> I propose to add JdbcDialect#setFetchSize method [2]
>
> Thoughts?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12826
> [2] https://github.com/apache/ignite/pull/7577/files

Re: Thin client: compute support

[Pavel Tupitsyn]

> will try to provide such guarantees on the server-side
Thanks. I think it is better to do once in the server code than N times in
every client.

On Wed, Mar 25, 2020 at 11:04 PM Alex Plehanov 
wrote:

> Pavel,
>
> 1. Actually it can be solved on the client-side (and already solved in PoC
> implementation). But I agreed it brings extra complexity for client-side
> implementation, will try to provide such guarantees on the server-side.
> 2. ComputeTask has also "reduce" step which is executed on the initiator
> node. Binary-rest client implementation, for example, has such affinity
> methods (to execute the task by name). I'm ok with removing it. At least if
> someone will need it we can implement it again at any time in the future
> without protocol change.
> I've fixed IEP.
>
> Denis,
>
> Deployment API definitely needed as one of the next steps. Currently, we
> are talking only about the first step (execution of already deployed
> tasks).
> Also, I'm not sure about automatic redeploy and peer-class-loading for thin
> clients, I think it's better to have more control here and provide API to
> explicitly deploy classes or jar files. WDYT?
>
> ср, 25 мар. 2020 г. в 21:17, Denis Magda :
>
> > Alex, thanks for preparing the outline.
> >
> > I'd like us to discuss an approach for compute tasks update with no
> > downtimes on the servers' end. For instance, let's assume that a
> > Python/C++/Node.JS developer requested to update a compute task he called
> > from the app. Should we introduce some system level API to the binary
> > protocol that can take a jar file (or class) and redeploy it
> automatically
> > with the usage of peer-class-loading?
> >
> > -
> > Denis
> >
> >
> > On Wed, Mar 25, 2020 at 5:47 AM Alex Plehanov 
> > wrote:
> >
> > > Hello guys.
> > >
> > > I've implemented PoC and created IEP [1] for thin client compute grid
> > > functionality. Please have a look.
> > >
> > > [1]:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-42+Thin+client%3A+compute+support
> > >
> > > пт, 24 янв. 2020 г. в 16:56, Alex Plehanov :
> > >
> > > > We've discussed thin client compute protocol with Pavel Tupitsyn and
> > Igor
> > > > Sapego and come to the conclusion that approach with two-way requests
> > > > should be used: client generates taskId and send a request to the
> > server
> > > to
> > > > execute a task. The server responds that the request has been
> accepted.
> > > > After task has finished the server notifies the client (send a
> request
> > > > without waiting for a response). The client can cancel the task by
> > > sending
> > > > a corresponding request to the server.
> > > >
> > > > Also, a node list should be passed (optionally) with a request to
> limit
> > > > nodes to execute the task.
> > > >
> > > > I will create IEP and file detailed protocol changes shortly.
> > > >
> > > > вт, 21 янв. 2020 г. в 18:46, Alex Plehanov  >:
> > > >
> > > >> Igor, thanks for the reply.
> > > >>
> > > >> > Approach with taskId will require a lot of changes in protocol and
> > > thus
> > > >> more "heavy" for implementation
> > > >> Do you mean approach with server notifications mechanism? Yes, it
> will
> > > >> require a lot of changes. But in most recent messages we've
> discussed
> > > with
> > > >> Pavel approach without server notifications mechanism. This approach
> > > have
> > > >> the same complexity and performance as an approach with requestId.
> > > >>
> > > >> > But such clients as Python, Node.js, PHP, Go most probably won't
> > have
> > > >> support for this API, at least for now.
> > > >> Without a server notifications mechanism, there will be no breaking
> > > >> changes in the protocol, so client implementation can just skip this
> > > >> feature and protocol version and implement the next one.
> > > >>
> > > >> > Or never.
> > > >> I think it still useful to execute java compute tasks from non-java
> > thin
> > > >> clients. Also, we can provide some out-of-the-box java tasks, for
> > > example
> > > >> ExecutePythonScriptTask with python compute implementation, which
> can
> > > run
> > > >> python script on server node.
> > > >>
> > > >> > So, maybe it's a good time for us to change our backward
> > compatibility
> > > >> mechanism from protocol versioning to feature masks?
> > > >> I like the idea with feature masks, but it will force us to support
> > both
> > > >> backward compatibility mechanisms, protocol versioning and feature
> > > masks.
> > > >>
> > > >> пн, 20 янв. 2020 г. в 20:34, Pavel Tupitsyn :
> > > >>
> > > >>> Huge +1 from me for Feature Masks.
> > > >>> I think this should be our top priority for thin client protocol,
> > since
> > > >>> it
> > > >>> simplifies change management a lot.
> > > >>>
> > > >>> On Mon, Jan 20, 2020 at 8:21 PM Igor Sapego 
> > > wrote:
> > > >>>
> > > >>> > Sorry for the late reply.
> > > >>> >
> > > >>> > Approach with taskId will require a lot of changes in protocol
> and
> > > thus
> > > >>> > more "heavy" for 

Re: Security Subject of thin client on remote nodes

[Ivan Rakov]

Denis,

I'll review your PR. If this issue is a subject to be included in 2.8.1 in
emergency mode, I'm ok with the current API changes.
Please think about driving creating IEP on security API overhaul prior to
2.9. I believe that you are the most suitable Ignite community member to
drive this activity. I'd love to share some ideas as well.

On Tue, Mar 24, 2020 at 2:04 PM Denis Garus  wrote:

> Hi, guys!
>
>
> I agree that we should rework the security API, but it can take a long
> time.
>
> And currently, our users have certain impediments that are blockers for
> their job.
>
> I think we have to fix bugs that IEP-41 [1] contains as soon as possible to
> support our users.
>
>  From my point of view, IEP-41 is the best place to track bug fixing.
>
>
>
>1.
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-41%3A+Security+Context+of+thin+client+on+remote+nodes
>
>
> вт, 24 мар. 2020 г. в 12:26, Ivan Rakov :
>
> > Alexey,
> >
> > That can be another version of our plan. If everyone agrees that
> > SecurityContext and SecuritySubject should be merged, such fix of thin
> > clients' issue will bring us closer to the final solution.
> > Denis, what do you think?
> >
> > On Tue, Mar 24, 2020 at 10:38 AM Alexei Scherbakov <
> > alexey.scherbak...@gmail.com> wrote:
> >
> > > Why can't we start gradually changing security API right now ?
> > > I see no point in delaying with.
> > > All changes will go to next 2.9 release anyway.
> > >
> > > My proposal:
> > > 1. Get rid of security context. Doing this will bring security API to
> > more
> > > or less consistent state.
> > > 2. Remove IEP-41 because it's no longer needed because of change [1]
> > > 3. Propose an IEP to make security API avoid using internals.
> > >
> > >
> > >
> > > пн, 23 мар. 2020 г. в 19:53, Denis Garus :
> > >
> > > > Hello, Alexei, Ivan!
> > > >
> > > > >> Seems like security API is indeed a bit over-engineered
> > > >
> > > > Nobody has doubt we should do a reworking of GridSecurityProcessor.
> > > > But this point is outside of scope thin client's problem that we are
> > > > solving.
> > > > I think we can create new IEP that will accumulate all ideas of
> > Ignite's
> > > > security improvements.
> > > >
> > > > >> Presence of the separate #securityContext(UUID) highlights that
> user
> > > > indeed should care
> > > > >> about propagation of thin clients' contexts between the cluster
> > nodes.
> > > >
> > > > I agree with Ivan. I've implemented both variants,
> > > > and I like one with #securityContext(UUID) more.
> > > >
> > > > Could you please take a look at PR [1] for the issue [2]?
> > > >
> > > > 1. https://github.com/apache/ignite/pull/7523
> > > > 2. https://issues.apache.org/jira/browse/IGNITE-12759
> > > >
> > > > пн, 23 мар. 2020 г. в 11:45, Ivan Rakov :
> > > >
> > > > > Alex, Denis,
> > > > >
> > > > > Seems like security API is indeed a bit over-engineered.
> > > > >
> > > > > Let's get rid of SecurityContext and use SecuritySubject instead.
> > > > > > SecurityContext is just a POJO wrapper over
> > > > > > SecuritySubject's
> > > > > > org.apache.ignite.plugin.security.SecuritySubject#permissions.
> > > > > > It's functionality can be easily moved to SecuritySubject.
> > > > >
> > > > > I totally agree. Both subject and context are implemented by plugin
> > > > > provider, and I don't see any reason to keep both abstractions,
> > > > especially
> > > > > if we are going to get rid of transferring subject in node
> attributes
> > > > > (argument that subject is more lightweight won't work anymore).
> > > > >
> > > > > Also, there's kind of mess in node authentication logic. There are
> at
> > > > least
> > > > > two components responsible for it: DiscoverySpiNodeAuthenticator
> > (which
> > > > is
> > > > > forcibly set by GridDiscoveryManager, but in fact public) and
> > > > > GridSecurityProcessor (which performs actual node auth logic, but
> > > > private).
> > > > > I also don't understand why we need both
> > > > > #authenticate(AuthenticationContext) and
> > #authenticateNode(ClusterNode,
> > > > > SecurityCredentials) methods while it's possible to set explicit
> > > > > SecuritySubjectType.REMOTE_NODE in AuthenticationContext (this is
> > > > arguable;
> > > > > perhaps there are strong reasons).
> > > > >
> > > > > Finally, areas of responsibility between IgniteSecurity and
> > > > > GridSecurityProcessor are kind of mixed. As far as I understand,
> the
> > > > first
> > > > > is responsible for Ignite-internal management of security logic
> > > (keeping
> > > > > thread-local context, caching security contexts, etc; we don't
> expect
> > > > > IgniteSecurity to be replaced by plugin provider) and the latter is
> > > > > responsible for user-custom authentication / authorization logic.
> To
> > be
> > > > > honest, it took plenty of time to figure this out for me.
> > > > >
> > > > > From my point of view, we should make GridSecurityProcessor
> interface
> > > > > public, rename it (it requires plenty 

FetchSize of the JdbcStore

[Nikolay Izhikov]

Hello, Igniters.

Right now there is no way for the user to set fetchSize of the JdbcStore
This leads to the poor performance of the cache load with the 3-d party 
persistence [1]

I propose to add JdbcDialect#setFetchSize method [2]

Thoughts?

[1] https://issues.apache.org/jira/browse/IGNITE-12826
[2] https://github.com/apache/ignite/pull/7577/files

[jira] [Created] (IGNITE-12841) @Override must be on the same line as a method

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created IGNITE-12841:


 Summary: @Override must be on the same line as a method
 Key: IGNITE-12841
 URL: https://issues.apache.org/jira/browse/IGNITE-12841
 Project: Ignite
  Issue Type: Bug
Reporter: Nikolay Izhikov


Right now, there are many places where codestyle broken.

{noformat}
/** {@inheritDoc} */
@Override
public boolean registerClassName(byte platformId, int typeId, String 
clsName) throws IgniteCheckedException {
return registerClassName(platformId, typeId, clsName, false);
}

{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Improvement in control.sh configuration.

[Oleg Ostanin]

I'm currently working on ticket
https://issues.apache.org/jira/browse/IGNITE-12832

It's not exactly what had been suggested but it will allow to pass some
custom properties to the client configuration and to pass parameters without
length limitations using .properties file.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-12840) Leaking H2 internals when querying from pyignite

[Ilya Kasnacheev (Jira)]

Ilya Kasnacheev created IGNITE-12840:


 Summary: Leaking H2 internals when querying from pyignite
 Key: IGNITE-12840
 URL: https://issues.apache.org/jira/browse/IGNITE-12840
 Project: Ignite
  Issue Type: Bug
  Components: clients, python
Affects Versions: 2.8
Reporter: Ilya Kasnacheev
 Fix For: 2.8.1


I am sharing a slightly modified reproducer from userlist.

To run:

Start a node (with JVM_OPTS=-Xmx512m to see it most clearly).
Run a .sql file with sqlline (!run)
python3 test_insert_in_cache.py
in parallel terminal: python3 read_cache_insert_update_table.py

Then you should observe slowing down and long GC pauses on server node. jmap 
will collect ever-increasing heap dumps.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Who can update TC Bot version.

[Zhenya Stanilovsky]

Dmitriy, could you plz ?


  
>We can manage that here.
>
>
>> On 24 Mar 2020, at 13:29, Dmitriy Pavlov < dpav...@apache.org > wrote:
>>
>> Hi,
>>
>> I could potentially update this version in the source code, but I'm not
>> able to deploy new version to the server (since it is now sponsored by
>> GrigGain, we need someone who will deploy).
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пн, 23 мар. 2020 г. в 08:59, Zhenya Stanilovsky < arzamas...@mail.ru >:
>>
>>> Igniters, 2.8 version already released, but looks like TC bot [1] [2]
>>> still using 2.7.6 ver. , who can update it ?
>>> Thanks !
>>>
>>> [1]  https://github.com/apache/ignite-teamcity-bot
>>> [2]  https://mtcga.gridgain.com/
>>>
>>>
>>> 
 
 
 
 

Re: Server Node comes down with : (err) Failed to notify listener: GridDhtTxPrepareFuture Error

[Denis Mekhanikov]

Thanks for the report!

The issue here is that a remote filter for a continuous query is loaded
using peer class loading, and other classes that this remote filter depends
on can be lazily loaded during its work.
Loading every dependency class involves going to the node where the
originating class was loading from, and asking that node to send missing
classes over the network.
The issues begin when this node is not in the cluster anymore, and the
continuous query wasn't undeployed yet.
A server sends a request for a class to a node that is not available, but
wasn't kicked out of the topology yet, since a failure detection timeout
hasn't elapsed yet.
It leads to a NoClasDefFound exception that you observe in the logs.

The biggest issue here is that this exception triggers a failure handler
that makes the whole node go down.
I would expect that only one request would fail, but not the whole node.

As a temporary solution you can stop relying on peer class loading for
continuous queries and provide the code of remote filters to the classpath
of server nodes.
This way no lazy class loading will be performed over the network since
they will all be available locally.

Denis

пт, 13 мар. 2020 г. в 20:39, VeenaMithare :

> Raised this jira :
> https://issues.apache.org/jira/browse/IGNITE-12784
>
> Observed in 2.7.6. Unable to easily test in 2.8.0 because of other issues.
> One of them being -
>
> http://apache-ignite-users.70518.x6.nabble.com/2-8-0-JDBC-Thin-Client-Unable-to-load-the-tables-via-DBeaver-td31681.html
> Please note this happens
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[jira] [Created] (IGNITE-12839) IGNITE-12789 broke WALRecordSerializationTest

[Ivan Bessonov (Jira)]

Ivan Bessonov created IGNITE-12839:
--

 Summary: IGNITE-12789 broke WALRecordSerializationTest
 Key: IGNITE-12839
 URL: https://issues.apache.org/jira/browse/IGNITE-12839
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


[https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-3192056576753991319=%3Cdefault%3E=testDetails]

 

Sorry, too bad that I skipped it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)