[jira] [Created] (IGNITE-12719) Allow users to configuring Ignite Thread Pool's Core thread count/max thread count/etc

[Sunny Chan (Jira)]

Sunny Chan created IGNITE-12719:
---

 Summary: Allow users to configuring Ignite Thread Pool's Core 
thread count/max thread count/etc
 Key: IGNITE-12719
 URL: https://issues.apache.org/jira/browse/IGNITE-12719
 Project: Ignite
  Issue Type: Bug
 Environment: We are running Ignite cluster on bare metal on a 
relatively high core count machine (4x10 cores 20 threads), and looking some of 
the thread pool initialization code: 

{{(IgnitionEx.java)}}

{{sysExecSvc = *new* IgniteThreadPoolExecutor(}}{{                          
"sys", }}{{cfg.getIgniteInstanceName(),}}{{    
cfg.getSystemThreadPoolSize(),}}{{    
cfg.getSystemThreadPoolSize(),}}{{    
*_DFLT_THREAD_KEEP_ALIVE_TIME_*, }}{{*new* LinkedBlockingQueue(),}}{{ 
         GridIoPolicy.*_SYSTEM_POOL_*);}}

Notice that the core thread pool size is equals to the max thread pool 
settings, which is by default same as the number of CPU cores. And in our 
cases, we won’t be reusing any threads until we have enough request coming in 
to fill 80 threads. Also, we might want to tune the thread keep alive time to 
improve thread reuse.

We would like to propose to change ignite so that users can configure the core 
thread pool size in these Ignite thread pools.
Reporter: Sunny Chan






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

RE: Configuring Ignite Thread Pool's Core thread count/max thread count/etc

[Sunny Chan, CLSA]

Hi Ilya,

For the IgniteConfiguration interface, do we prefer:

1) IgniteConfiguration.setSystemThreadPoolCoreSize(), 
IgniteConfiguration.setSystemThreadPoolTimeOut(),IgniteConfiguration.setPublicThreadPoolCoreSize(),
 etc.
2) IgniteConfiguration.setSystemThreadPool(ThreadPoolConfiguration), 
IgniteConfiguration.setThreadPoolCoreSize(ThreadPoolConfiguration) and then we 
have ThreadPoolConfiguration.setCoreSize(), etc

Thanks.

-Original Message-
From: Ilya Kasnacheev  
Sent: Wednesday, February 26, 2020 6:10 PM
To: dev 
Subject: Re: Configuring Ignite Thread Pool's Core thread count/max thread 
count/etc

Hello!

We recommend exposing IgniteConfiguration for user to modify. You never know in 
advance what kind of tuning will be needed, and don't want to play chinese 
whispers game with your users.

Regards,
--
Ilya Kasnacheev


ср, 26 февр. 2020 г. в 13:07, Sunny Chan, CLSA :

> Hello,
>
>
>
> We are running Ignite cluster on bare metal on a relatively high core
> count machine (4x10 cores 20 threads), and looking some of the thread pool
> initialization code:
>
>
>
> (IgnitionEx.java)
>
> sysExecSvc = *new* IgniteThreadPoolExecutor(
>
> "sys",
>
> cfg.getIgniteInstanceName(),
>
> cfg.getSystemThreadPoolSize(),
>
> cfg.getSystemThreadPoolSize(),
>
> *DFLT_THREAD_KEEP_ALIVE_TIME*,
>
> *new* LinkedBlockingQueue(),
>
> GridIoPolicy.*SYSTEM_POOL*);
>
>
>
> Notice that the core thread pool size is equals to the max thread pool
> settings, which is by default same as the number of CPU cores. And in our
> cases, we won’t be reusing any threads until we have enough request coming
> in to fill 80 threads. Also, we might want to tune the thread keep alive
> time to improve thread reuse.
>
>
>
> We would like to propose to change ignite so that users can configure the
> core thread pool size in these Ignite thread pools. What is the best way to
> expose these parameters for user to modify?
>
>
>
> Would the ignite dev team prefer exposing individual core thread size and
> others (ie. cfg.get/setSystemThreadPoolCoreSize(),
> cfg.get/setSystemThreadPoolKeepAliveTime(), ..) or should we use a thread
> pool configuration object? (e.g.
> cfg.getSystemThreadPoolConfiguration(ThreadPoolConfiguration config) where
> ThreadPoolConfiguration has get and set methods for core thread pool size,
> etc)?
>
>
>
> *Sunny Chan*
>
> *Senior Lead Engineer, Executive Services*
>
> D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
>
> 5/F, One Island East, 18 Westlands Road, Island East, Hong Kong
>
>
>
> [image: :1. Social Media Icons:CLSA_Social Media Icons_linkedin.png]
>   >[image: :1. Social Media
> Icons:CLSA_Social Media Icons_twitter.png]
>   >[image: :1. Social Media
> Icons:CLSA_Social Media Icons_youtube.png]
>   >[image: :1.
> Social Media Icons:CLSA_Social Media Icons_facebook.png]
>   >
>
>
>
> *clsa.com* 
>
> *Insights. Liquidity. Capital. *
>
>
>
> [image: CLSA_RGB] 
>
>
>
> *A CITIC Securities Company*
>
>
>
> The content of this communication is intended for the recipient and is
> subject to CLSA Legal and Regulatory Notices.
> These can be viewed at https://www.clsa.com/disclaimer.html or sent to
> you upon request.
> Please consider before printing. CLSA is ISO14001 certified and committed
> to reducing its impact on the environment.
>
The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

Re: Subscription Request

[Denis Magda]

Duplicate message, responded in a similar thread.

-
Denis


On Wed, Feb 26, 2020 at 8:04 AM Javad Alimohammadi <
bs.alimohamm...@gmail.com> wrote:

> Hello to everybody. I would be glad to be part of your community.
>
> *My Areas of Expertise *
> Core Java, Concurrent Programming, Performance Optimization, Distributed
> System, SQL
>
> *How Could I Help*
> Fix bugs
> Develop a new feature
> Fix documentation
>
> *My Jir Account *
> j-alimohammadi
>

Re: Subscription request

[Denis Magda]

Hello Javad,

Welcome to the community, and thanks for sharing your areas of expertise
with interests! You were added to the contributors' list in JIRA.

Please check the "Pick a Ticket" section on this page [1] selecting a task
you'd like to start with. Our community members have refreshed the list
recently adding the most urgent tickets we need help with. All those
tickets are good for newcomers to the community.

Btw, do you have any experience with Ignite?

[1] https://ignite.apache.org/community/contribute.html

-
Denis


On Wed, Feb 26, 2020 at 12:03 PM Javad Alimohammadi <
bs.alimohamm...@gmail.com> wrote:

> Hello to everybody. I would like to be part of your community.
>
> *My Areas of Expertise *
> Core Java, Concurrent Programming, Performance Optimization, Distributed
> System, SQL
>
> *How Could I Help*
> Fix bugs
> Develop a new feature
> Fix documentation
>
> *My Jir Account *
> j-alimohammadi
>

Subscription request

[Javad Alimohammadi]

Hello to everybody. I would like to be part of your community.

*My Areas of Expertise *
Core Java, Concurrent Programming, Performance Optimization, Distributed
System, SQL

*How Could I Help*
Fix bugs
Develop a new feature
Fix documentation

*My Jir Account *
j-alimohammadi

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Pavel Tupitsyn]

Maxim,

Fixes are in ignite-2.8 now, and builds have passed [1].
I think we can proceed.

Thank you and sorry for the broken build.

[1]
https://ci.ignite.apache.org/buildConfiguration/ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages/5083539?buildTab=overview

On Wed, Feb 26, 2020 at 8:46 PM Pavel Tupitsyn  wrote:

> I'm running a final check in branch ignite-2.8-dotnet-build-fix, should be
> done soon.
> After that I'll merge the changes into ignite-2.8 (and backport to
> master), and we can proceed.
>
> I had to fix both TC project and the build script.
> There were a few regressions introduced by various recent changes.
>
> > Can you also provide some details - which exactly artefacts should I
> check on staging resource after the [1] suite execution?
> We are supposed to verify that NuGet packages are valid by installing them
> and starting Ignite.
> However, we added an automatic check recently: `Run NuGet verification
> script` build step.
> So the only thing to check is that new package appears on MyGet [1]
> (scroll down to see versions and dates)
>
> [1]
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
>
>
> On Wed, Feb 26, 2020 at 7:42 PM Maxim Muzafarov  wrote:
>
>> Pavel,
>>
>>
>> Thank you for your help.
>> Please, let me know when I can proceed with a vote preparation.
>>
>>
>> Can you also provide some details - which exactly artefacts should I
>> check on staging resource after the [1] suite execution?
>>
>> [1]
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
>>
>> On Wed, 26 Feb 2020 at 10:46, Pavel Tupitsyn 
>> wrote:
>> >
>> > Maxim, I did a quick fix for the script, but it did not work.
>> > Investigating further, will get back to you later today.
>> >
>> > On Tue, Feb 25, 2020 at 10:10 PM Maxim Muzafarov 
>> wrote:
>> >>
>> >> Pavel,
>> >>
>> >>
>> >> Can you assist me with preparing NuGet staging according to the
>> >> release steps [1]? I've double-checked everything related to the build
>> >> but suite [2] for preparing nuget package still fails (sorry for
>> >> running it multiple times).
>> >>
>> >> I see some issues in the log which may be related to the file copy in
>> >> the build script which recently changed [4].
>> >> I'm still digging in, but anyway can you take a look and help?
>> >>
>> >>
>> >> Step 2:
>> >> Copy-Item : Cannot find path
>> >>
>> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core.dll'because
>> >> it does not exist.
>> >>
>> >> Step 3:
>> >> Cannot find path
>> >>
>> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core\bin\Release\Apache.Ignite.Core.dll'
>> >> because it does not exist.
>> >> At
>> C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\build.ps1:283
>> >> char:47
>> >>
>> >> pack: invalid arguments.
>> >>
>> >>
>> >> [1]
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-4.4.2.PrepareNuGetstaging
>> >> [2]
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
>> >> [3]
>> https://ci.ignite.apache.org/viewLog.html?buildId=5080553=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=buildLog
>> >> [4] https://issues.apache.org/jira/browse/IGNITE-12604
>> >>
>> >> On Fri, 21 Feb 2020 at 10:36, Maxim Muzafarov 
>> wrote:
>> >> >
>> >> > Denis,
>> >> >
>> >> >
>> >> > Currently, we have no blockers. I'm preparing the build.
>> >> >
>> >> > On Thu, 20 Feb 2020 at 21:10, Denis Magda  wrote:
>> >> > >
>> >> > > Folks,
>> >> > >
>> >> > > Is there anything else apart from the open documentation tickets
>> that
>> >> > > prevent us from starting the release vote? I think that it should
>> take
>> >> > > around two weeks to run the release through the vote and announce
>> it. The
>> >> > > top doc changes should be finished throughout that time already.
>> >> > >
>> >> > > -
>> >> > > Denis
>> >> > >
>> >> > >
>> >> > > On Wed, Feb 19, 2020 at 9:55 AM Maxim Muzafarov 
>> wrote:
>> >> > >
>> >> > > > Ilya,
>> >> > > >
>> >> > > >
>> >> > > > I think we must accept only blocker issues to the release branch.
>> >> > > >
>> >> > > > My previous experience tells me that even a small change which
>> seems
>> >> > > > absolutely easy and clear can break everything. So, let's move
>> this
>> >> > > > issue [1] to the next release. Currently, it doesn't look like a
>> >> > > > blocker.
>> >> > > >
>> >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12672
>> >> > > >
>> >> > > > On Tue, 18 Feb 2020 at 13:51, Maxim Muzafarov 
>> wrote:
>> >> > > > >
>> >> > > > > Igniters,
>> >> > > > >
>> >> > > > >
>> >> > > > > I've prepared the issue [1] and PR [2] with removing @deprecate
>> >> > > > > annotation on DataRegionMetrics and adding @IgniteExperimental
>> to the
>> >> > > > > new metrics API.
>> >> > > > > Can anyone review my changes?
>> >> > > > >
>> >> > > > >
>> >> > > > > [1] 

Node.JS, PHP, Python API references for Ignite 2.8 release

[Denis Magda]

Igor Sapego, Igniters,

I've been working on Ignite website improvements that will be introduced
and contributed later and found that we still don't generate API references
for Node.JS, PHP, Python. The APIs are not even shipped withing binary
packages.

Do our scripts for build/release generate those APIs? Can we start
uploading them to the website under this path?
https://svn.apache.org/repos/asf/ignite/site/trunk/releases

-
Denis

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Pavel Tupitsyn]

I'm running a final check in branch ignite-2.8-dotnet-build-fix, should be
done soon.
After that I'll merge the changes into ignite-2.8 (and backport to master),
and we can proceed.

I had to fix both TC project and the build script.
There were a few regressions introduced by various recent changes.

> Can you also provide some details - which exactly artefacts should I
check on staging resource after the [1] suite execution?
We are supposed to verify that NuGet packages are valid by installing them
and starting Ignite.
However, we added an automatic check recently: `Run NuGet verification
script` build step.
So the only thing to check is that new package appears on MyGet [1] (scroll
down to see versions and dates)

[1]
https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite


On Wed, Feb 26, 2020 at 7:42 PM Maxim Muzafarov  wrote:

> Pavel,
>
>
> Thank you for your help.
> Please, let me know when I can proceed with a vote preparation.
>
>
> Can you also provide some details - which exactly artefacts should I
> check on staging resource after the [1] suite execution?
>
> [1]
> https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
>
> On Wed, 26 Feb 2020 at 10:46, Pavel Tupitsyn  wrote:
> >
> > Maxim, I did a quick fix for the script, but it did not work.
> > Investigating further, will get back to you later today.
> >
> > On Tue, Feb 25, 2020 at 10:10 PM Maxim Muzafarov 
> wrote:
> >>
> >> Pavel,
> >>
> >>
> >> Can you assist me with preparing NuGet staging according to the
> >> release steps [1]? I've double-checked everything related to the build
> >> but suite [2] for preparing nuget package still fails (sorry for
> >> running it multiple times).
> >>
> >> I see some issues in the log which may be related to the file copy in
> >> the build script which recently changed [4].
> >> I'm still digging in, but anyway can you take a look and help?
> >>
> >>
> >> Step 2:
> >> Copy-Item : Cannot find path
> >>
> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core.dll'because
> >> it does not exist.
> >>
> >> Step 3:
> >> Cannot find path
> >>
> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core\bin\Release\Apache.Ignite.Core.dll'
> >> because it does not exist.
> >> At
> C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\build.ps1:283
> >> char:47
> >>
> >> pack: invalid arguments.
> >>
> >>
> >> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-4.4.2.PrepareNuGetstaging
> >> [2]
> https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
> >> [3]
> https://ci.ignite.apache.org/viewLog.html?buildId=5080553=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=buildLog
> >> [4] https://issues.apache.org/jira/browse/IGNITE-12604
> >>
> >> On Fri, 21 Feb 2020 at 10:36, Maxim Muzafarov 
> wrote:
> >> >
> >> > Denis,
> >> >
> >> >
> >> > Currently, we have no blockers. I'm preparing the build.
> >> >
> >> > On Thu, 20 Feb 2020 at 21:10, Denis Magda  wrote:
> >> > >
> >> > > Folks,
> >> > >
> >> > > Is there anything else apart from the open documentation tickets
> that
> >> > > prevent us from starting the release vote? I think that it should
> take
> >> > > around two weeks to run the release through the vote and announce
> it. The
> >> > > top doc changes should be finished throughout that time already.
> >> > >
> >> > > -
> >> > > Denis
> >> > >
> >> > >
> >> > > On Wed, Feb 19, 2020 at 9:55 AM Maxim Muzafarov 
> wrote:
> >> > >
> >> > > > Ilya,
> >> > > >
> >> > > >
> >> > > > I think we must accept only blocker issues to the release branch.
> >> > > >
> >> > > > My previous experience tells me that even a small change which
> seems
> >> > > > absolutely easy and clear can break everything. So, let's move
> this
> >> > > > issue [1] to the next release. Currently, it doesn't look like a
> >> > > > blocker.
> >> > > >
> >> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12672
> >> > > >
> >> > > > On Tue, 18 Feb 2020 at 13:51, Maxim Muzafarov 
> wrote:
> >> > > > >
> >> > > > > Igniters,
> >> > > > >
> >> > > > >
> >> > > > > I've prepared the issue [1] and PR [2] with removing @deprecate
> >> > > > > annotation on DataRegionMetrics and adding @IgniteExperimental
> to the
> >> > > > > new metrics API.
> >> > > > > Can anyone review my changes?
> >> > > > >
> >> > > > >
> >> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12690
> >> > > > > [2] https://github.com/apache/ignite/pull/7440
> >> > > > >
> >> > > > > On Tue, 18 Feb 2020 at 13:42, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> >> > > > wrote:
> >> > > > > >
> >> > > > > > Hello!
> >> > > > > >
> >> > > > > > I have just merged a fix for embarrassing issue where you
> could UPDATE
> >> > > > > > entries with Spring Data, but not "Update" or "update" them.
> >> > > > > >
> >> > > > > > I 

Re: Apache Ignite 2.8 RELEASE [Time, Scope, Manager]

[Maxim Muzafarov]

Pavel,


Thank you for your help.
Please, let me know when I can proceed with a vote preparation.


Can you also provide some details - which exactly artefacts should I
check on staging resource after the [1] suite execution?

[1] 
https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages

On Wed, 26 Feb 2020 at 10:46, Pavel Tupitsyn  wrote:
>
> Maxim, I did a quick fix for the script, but it did not work.
> Investigating further, will get back to you later today.
>
> On Tue, Feb 25, 2020 at 10:10 PM Maxim Muzafarov  wrote:
>>
>> Pavel,
>>
>>
>> Can you assist me with preparing NuGet staging according to the
>> release steps [1]? I've double-checked everything related to the build
>> but suite [2] for preparing nuget package still fails (sorry for
>> running it multiple times).
>>
>> I see some issues in the log which may be related to the file copy in
>> the build script which recently changed [4].
>> I'm still digging in, but anyway can you take a look and help?
>>
>>
>> Step 2:
>> Copy-Item : Cannot find path
>> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core.dll'because
>> it does not exist.
>>
>> Step 3:
>> Cannot find path
>> 'C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\Apache.Ignite.Core\bin\Release\Apache.Ignite.Core.dll'
>> because it does not exist.
>> At 
>> C:\BuildAgent\work\3722fcb3466a49e6\src\modules\platforms\dotnet\build.ps1:283
>> char:47
>>
>> pack: invalid arguments.
>>
>>
>> [1] 
>> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-4.4.2.PrepareNuGetstaging
>> [2] 
>> https://ci.ignite.apache.org/viewType.html?buildTypeId=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages
>> [3] 
>> https://ci.ignite.apache.org/viewLog.html?buildId=5080553=ApacheIgniteReleaseJava8_PrepareVote3BuildNuGetPackages=buildLog
>> [4] https://issues.apache.org/jira/browse/IGNITE-12604
>>
>> On Fri, 21 Feb 2020 at 10:36, Maxim Muzafarov  wrote:
>> >
>> > Denis,
>> >
>> >
>> > Currently, we have no blockers. I'm preparing the build.
>> >
>> > On Thu, 20 Feb 2020 at 21:10, Denis Magda  wrote:
>> > >
>> > > Folks,
>> > >
>> > > Is there anything else apart from the open documentation tickets that
>> > > prevent us from starting the release vote? I think that it should take
>> > > around two weeks to run the release through the vote and announce it. The
>> > > top doc changes should be finished throughout that time already.
>> > >
>> > > -
>> > > Denis
>> > >
>> > >
>> > > On Wed, Feb 19, 2020 at 9:55 AM Maxim Muzafarov  
>> > > wrote:
>> > >
>> > > > Ilya,
>> > > >
>> > > >
>> > > > I think we must accept only blocker issues to the release branch.
>> > > >
>> > > > My previous experience tells me that even a small change which seems
>> > > > absolutely easy and clear can break everything. So, let's move this
>> > > > issue [1] to the next release. Currently, it doesn't look like a
>> > > > blocker.
>> > > >
>> > > > [1] https://issues.apache.org/jira/browse/IGNITE-12672
>> > > >
>> > > > On Tue, 18 Feb 2020 at 13:51, Maxim Muzafarov  
>> > > > wrote:
>> > > > >
>> > > > > Igniters,
>> > > > >
>> > > > >
>> > > > > I've prepared the issue [1] and PR [2] with removing @deprecate
>> > > > > annotation on DataRegionMetrics and adding @IgniteExperimental to the
>> > > > > new metrics API.
>> > > > > Can anyone review my changes?
>> > > > >
>> > > > >
>> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-12690
>> > > > > [2] https://github.com/apache/ignite/pull/7440
>> > > > >
>> > > > > On Tue, 18 Feb 2020 at 13:42, Ilya Kasnacheev 
>> > > > > 
>> > > > wrote:
>> > > > > >
>> > > > > > Hello!
>> > > > > >
>> > > > > > I have just merged a fix for embarrassing issue where you could 
>> > > > > > UPDATE
>> > > > > > entries with Spring Data, but not "Update" or "update" them.
>> > > > > >
>> > > > > > I suggest adding this fix to the scope of 2.8, since Spring Data is
>> > > > popular
>> > > > > > and it does not in any way affect code outside of its modules.
>> > > > > >
>> > > > > > https://issues.apache.org/jira/browse/IGNITE-12672
>> > > > > >
>> > > > > > WDYT?
>> > > > > >
>> > > > > > Regards,
>> > > > > > --
>> > > > > > Ilya Kasnacheev
>> > > > > >
>> > > > > >
>> > > > > > пн, 17 февр. 2020 г. в 12:44, Maxim Muzafarov :
>> > > > > >
>> > > > > > > Alexey,
>> > > > > > >
>> > > > > > >
>> > > > > > > Yes. I will remove @deprecation according to the vote results and
>> > > > will
>> > > > > > > go further with the release steps [1] since there no blockers 
>> > > > > > > left.
>> > > > > > >
>> > > > > > >
>> > > > > > > [1]
>> > > > https://cwiki.apache.org/confluence/display/IGNITE/Release+Process
>> > > > > > >
>> > > > > > > On Mon, 17 Feb 2020 at 11:48, Alexey Goncharuk
>> > > > > > >  wrote:
>> > > > > > > >
>> > > > > > > > Folks,
>> > > > > > > >
>> > > > > > > > I have merged IGNITE-12650 (mark MVCC as experimental) to 
>> > > > > > > > master
>> > > 

Task is ready for review

[Artsiom Panko]

Hello everyone. Task 12610 (https://issues.apache.org/jira/browse/IGNITE-12610) 
is ready for review

Subscription Request

[Javad Alimohammadi]

Hello to everybody. I would be glad to be part of your community.

*My Areas of Expertise *
Core Java, Concurrent Programming, Performance Optimization, Distributed
System, SQL

*How Could I Help*
Fix bugs
Develop a new feature
Fix documentation

*My Jir Account *
j-alimohammadi

[jira] [Created] (IGNITE-12718) Python Thin Client: add an ability to specify keyfile password

[Andrey Kuznetsov (Jira)]

Andrey Kuznetsov created IGNITE-12718:
-

 Summary: Python Thin Client: add an ability to specify keyfile 
password
 Key: IGNITE-12718
 URL: https://issues.apache.org/jira/browse/IGNITE-12718
 Project: Ignite
  Issue Type: Improvement
  Components: thin client
Affects Versions: 2.8
Reporter: Andrey Kuznetsov


In pyignite, there is no way to specify password for keyfile being used to 
establish TLS connection to Ignite cluster. If keyfile is encrypted, then 
OpenSSL library prompts for password interactively.

In order to add configurable password, one can set up explicit {{SSLContext}} 
instead of {{ssl.wrap_socket}} call.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (IGNITE-12717) SQL: indexing DB object creation refactoring

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-12717:
-

 Summary: SQL: indexing DB object creation refactoring
 Key: IGNITE-12717
 URL: https://issues.apache.org/jira/browse/IGNITE-12717
 Project: Ignite
  Issue Type: Task
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


Do refactoring children of the H2 IndexBase to simplify the H2 upgrade.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Configuring Ignite Thread Pool's Core thread count/max thread count/etc

[Ilya Kasnacheev]

Hello!

We recommend exposing IgniteConfiguration for user to modify. You never
know in advance what kind of tuning will be needed, and don't want to play
chinese whispers game with your users.

Regards,
-- 
Ilya Kasnacheev


ср, 26 февр. 2020 г. в 13:07, Sunny Chan, CLSA :

> Hello,
>
>
>
> We are running Ignite cluster on bare metal on a relatively high core
> count machine (4x10 cores 20 threads), and looking some of the thread pool
> initialization code:
>
>
>
> (IgnitionEx.java)
>
> sysExecSvc = *new* IgniteThreadPoolExecutor(
>
> "sys",
>
> cfg.getIgniteInstanceName(),
>
> cfg.getSystemThreadPoolSize(),
>
> cfg.getSystemThreadPoolSize(),
>
> *DFLT_THREAD_KEEP_ALIVE_TIME*,
>
> *new* LinkedBlockingQueue(),
>
> GridIoPolicy.*SYSTEM_POOL*);
>
>
>
> Notice that the core thread pool size is equals to the max thread pool
> settings, which is by default same as the number of CPU cores. And in our
> cases, we won’t be reusing any threads until we have enough request coming
> in to fill 80 threads. Also, we might want to tune the thread keep alive
> time to improve thread reuse.
>
>
>
> We would like to propose to change ignite so that users can configure the
> core thread pool size in these Ignite thread pools. What is the best way to
> expose these parameters for user to modify?
>
>
>
> Would the ignite dev team prefer exposing individual core thread size and
> others (ie. cfg.get/setSystemThreadPoolCoreSize(),
> cfg.get/setSystemThreadPoolKeepAliveTime(), ..) or should we use a thread
> pool configuration object? (e.g.
> cfg.getSystemThreadPoolConfiguration(ThreadPoolConfiguration config) where
> ThreadPoolConfiguration has get and set methods for core thread pool size,
> etc)?
>
>
>
> *Sunny Chan*
>
> *Senior Lead Engineer, Executive Services*
>
> D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
>
> 5/F, One Island East, 18 Westlands Road, Island East, Hong Kong
>
>
>
> [image: :1. Social Media Icons:CLSA_Social Media Icons_linkedin.png]
> [image: :1. Social Media
> Icons:CLSA_Social Media Icons_twitter.png]
> [image: :1. Social Media
> Icons:CLSA_Social Media Icons_youtube.png]
> [image: :1.
> Social Media Icons:CLSA_Social Media Icons_facebook.png]
> 
>
>
>
> *clsa.com* 
>
> *Insights. Liquidity. Capital. *
>
>
>
> [image: CLSA_RGB] 
>
>
>
> *A CITIC Securities Company*
>
>
>
> The content of this communication is intended for the recipient and is
> subject to CLSA Legal and Regulatory Notices.
> These can be viewed at https://www.clsa.com/disclaimer.html or sent to
> you upon request.
> Please consider before printing. CLSA is ISO14001 certified and committed
> to reducing its impact on the environment.
>

Configuring Ignite Thread Pool's Core thread count/max thread count/etc

[Sunny Chan, CLSA]

Hello,

We are running Ignite cluster on bare metal on a relatively high core count 
machine (4x10 cores 20 threads), and looking some of the thread pool 
initialization code:

(IgnitionEx.java)
sysExecSvc = new IgniteThreadPoolExecutor(
"sys",
cfg.getIgniteInstanceName(),
cfg.getSystemThreadPoolSize(),
cfg.getSystemThreadPoolSize(),
DFLT_THREAD_KEEP_ALIVE_TIME,
new LinkedBlockingQueue(),
GridIoPolicy.SYSTEM_POOL);

Notice that the core thread pool size is equals to the max thread pool 
settings, which is by default same as the number of CPU cores. And in our 
cases, we won't be reusing any threads until we have enough request coming in 
to fill 80 threads. Also, we might want to tune the thread keep alive time to 
improve thread reuse.

We would like to propose to change ignite so that users can configure the core 
thread pool size in these Ignite thread pools. What is the best way to expose 
these parameters for user to modify?

Would the ignite dev team prefer exposing individual core thread size and 
others (ie. cfg.get/setSystemThreadPoolCoreSize(), 
cfg.get/setSystemThreadPoolKeepAliveTime(), ..) or should we use a thread pool 
configuration object? (e.g. 
cfg.getSystemThreadPoolConfiguration(ThreadPoolConfiguration config) where 
ThreadPoolConfiguration has get and set methods for core thread pool size, etc)?

Sunny Chan
Senior Lead Engineer, Executive Services
D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

[:1. Social Media Icons:CLSA_Social Media 
Icons_linkedin.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_twitter.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_youtube.png][:1.
 Social Media Icons:CLSA_Social Media 
Icons_facebook.png]

clsa.com
Insights. Liquidity. Capital.

[CLSA_RGB]

A CITIC Securities Company

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

Re: Security Subject of thin client on remote nodes

[Mikhail Petrov]

Hi, Alexei.

The ticket [1] describes the general problem for all thin client 
authorizations. Its particular case is described in the mentioned in [1] 
ticket [2] on the example of a JDBC client  with the reproducer attached.


[1] https://issues.apache.org/jira/browse/IGNITE-12589
[2] https://issues.apache.org/jira/browse/IGNITE-12579

On 26.02.2020 11:47, Alexei Scherbakov wrote:

Denis Garus,

It is forbidden to remove any public IGNITE attribute without proper
deprecation steps.

I have read the thread and still do not clearly see the problem. The subject id
is not required to be a node id.

The referenced in the thread ticket [1] do not provide any reproducers.

Can you properly demonstrate a broken scenario ?

[1] https://issues.apache.org/jira/browse/IGNITE-12589

пт, 21 февр. 2020 г. в 13:35, Andrey Kuznetsov :


Hi, guys!

The change suggested by Denis looks robust to me: it covers security
subject handling by all kinds of clients/nodes at once. As for
ATTR_SECURITY_SUBJECT_V2 attribute, it is really better to move it to
plugin implementations to support backward compatibility with peer nodes of
older versions. Obviously, cluster with security disabled will not suffer
from attribute removal. Ignite core should know nothing about the specific
way of security context propagation.

Denis, could you please create Jira issue for your change?

чт, 20 февр. 2020 г. в 17:01, Denis Garus :


I just transmitted security subjects for rest requests.

SecurityContext has an unlimited size so we can get significant overhead.
And we do not solve problems with other thin clients.


If you remove ATTR_SECURITY_SUBJECT_V2, it breaks compatibility between

old
versions and new.

I suggest removing ATTR_SECURITY_SUBJECT_V2 from Ignite's codebase, but

for

compatibility, it can be used by a security plugin like in PoC.

чт, 20 февр. 2020 г. в 16:47, Maksim Stepachev <

maksim.stepac...@gmail.com

:
Yes, I said about it at 07.19.



http://apache-ignite-developers.2346864.n4.nabble.com/Improvements-for-new-security-approach-td42698.html#a42708

And in my solution, I just transmitted security subjects for rest

requests.

If you remove ATTR_SECURITY_SUBJECT_V2, it breaks compatibility between

old

versions and new.

чт, 20 февр. 2020 г. в 15:56, Denis Garus :


Hi, Igniters!


At present, a security subject id is assumed to be node id.

But when we are dealing with thin client, JDBC or REST subject id is

random

UUID. In this case, we cannot get the subject information on a remote

node,

and we get problems like these [1], [2].

To fix the problem, we should spread the client session to the whole
cluster.


I want to suggest a solution to the problem.


First, we should get subject information using GridSecurityProcessor.

How GridSecurityProcessor will retrieve a subject data, it is up to

plugin

developers.


Second, we should get rid of the assumption that a subject id is a

node

id

and remove the ATTR_SECURITY_SUBJECT_V2 attribute.


I have prepared PoC PR [3] that:

- places the existing logic of spreading security context to
GridSecurityProcessor;

- uses GridSecurityProcessor to get SecurityContext.



1.



http://apache-ignite-developers.2346864.n4.nabble.com/JDBC-thin-client-incorrect-security-context-td45929.html

2. https://issues.apache.org/jira/browse/IGNITE-12589
3. https://github.com/apache/ignite/pull/7375



--
Best regards,
   Andrey Kuznetsov.

Re: Let's make BinaryObjectImpl and CacheKeyObject Comparable

[Ilya Kasnacheev]

Hello!

I think they also should do that, but as phase II.

Regards,
-- 
Ilya Kasnacheev


ср, 26 февр. 2020 г. в 11:26, Ivan Pavlukhin :

> And what about non-java platforms?
>
> Best regards,
> Ivan Pavlukhin
>
> пт, 21 февр. 2020 г. в 14:22, Ilya Kasnacheev :
> >
> > Hello!
> >
> > I don't see how we can force users to implement Comparable for
> BinaryObject
> > keys, since BinaryObject implementation is not provided by users but by
> us.
> >
> > I think that we, however, could force it for composite non-BinaryObject
> > keys.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 21 февр. 2020 г. в 00:21, Denis Magda :
> >
> > > Hi Ilya,
> > >
> > > We can oblige users to implement Comparable if they use BinaryObject
> keys.
> > > Ignite can print out a warning if BinaryObject keys passed to putAll
> > > methods don't do that.
> > >
> > > I also wonder how a similar task was solved for Ignite INSERTs. Our
> engine
> > > should use BinaryObjects for compound primary keys and insert them at
> > > patches. That implementation can suggest us some hints.
> > >
> > > -
> > > Denis
> > >
> > >
> > > On Thu, Feb 20, 2020 at 6:53 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > Since we have merged
> https://issues.apache.org/jira/browse/IGNITE-6804
> > > we
> > > > have to face an embarrassing fact that BinaryObject is not
> Comparable,
> > > > i.e., when you do cache.withKeepBinary().putAll(), there are no
> obvious
> > > > ways to not get a deadlock (or at least warning) here.
> > > >
> > > > One can use LinkedHashMap, but they will have to sort BinaryObject's
> on
> > > > their side, which is not trivial.
> > > >
> > > > So my proposal is to make BinaryObjectImpl and KeyCacheObject (?)
> > > > Comparable, by their binary representation. We can't add this
> constraint
> > > to
> > > > BinaryObject since it is a public interface (can we), but we can do
> that
> > > > for the implementation types. What do you think?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > >
>

Re: Security Subject of thin client on remote nodes

[Alexei Scherbakov]

Denis Garus,

It is forbidden to remove any public IGNITE attribute without proper
deprecation steps.

I have read the thread and still do not clearly see the problem. The subject id
is not required to be a node id.

The referenced in the thread ticket [1] do not provide any reproducers.

Can you properly demonstrate a broken scenario ?

[1] https://issues.apache.org/jira/browse/IGNITE-12589

пт, 21 февр. 2020 г. в 13:35, Andrey Kuznetsov :

> Hi, guys!
>
> The change suggested by Denis looks robust to me: it covers security
> subject handling by all kinds of clients/nodes at once. As for
> ATTR_SECURITY_SUBJECT_V2 attribute, it is really better to move it to
> plugin implementations to support backward compatibility with peer nodes of
> older versions. Obviously, cluster with security disabled will not suffer
> from attribute removal. Ignite core should know nothing about the specific
> way of security context propagation.
>
> Denis, could you please create Jira issue for your change?
>
> чт, 20 февр. 2020 г. в 17:01, Denis Garus :
>
> > > I just transmitted security subjects for rest requests.
> >
> > SecurityContext has an unlimited size so we can get significant overhead.
> > And we do not solve problems with other thin clients.
> >
> > >If you remove ATTR_SECURITY_SUBJECT_V2, it breaks compatibility between
> > old
> > versions and new.
> >
> > I suggest removing ATTR_SECURITY_SUBJECT_V2 from Ignite's codebase, but
> for
> > compatibility, it can be used by a security plugin like in PoC.
> >
> > чт, 20 февр. 2020 г. в 16:47, Maksim Stepachev <
> maksim.stepac...@gmail.com
> > >:
> >
> > > Yes, I said about it at 07.19.
> > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/Improvements-for-new-security-approach-td42698.html#a42708
> > > And in my solution, I just transmitted security subjects for rest
> > requests.
> > >
> > > If you remove ATTR_SECURITY_SUBJECT_V2, it breaks compatibility between
> > old
> > > versions and new.
> > >
> > > чт, 20 февр. 2020 г. в 15:56, Denis Garus :
> > >
> > > > Hi, Igniters!
> > > >
> > > >
> > > > At present, a security subject id is assumed to be node id.
> > > >
> > > > But when we are dealing with thin client, JDBC or REST subject id is
> > > random
> > > > UUID. In this case, we cannot get the subject information on a remote
> > > node,
> > > > and we get problems like these [1], [2].
> > > >
> > > > To fix the problem, we should spread the client session to the whole
> > > > cluster.
> > > >
> > > >
> > > > I want to suggest a solution to the problem.
> > > >
> > > >
> > > > First, we should get subject information using GridSecurityProcessor.
> > > >
> > > > How GridSecurityProcessor will retrieve a subject data, it is up to
> > > plugin
> > > > developers.
> > > >
> > > >
> > > > Second, we should get rid of the assumption that a subject id is a
> node
> > > id
> > > > and remove the ATTR_SECURITY_SUBJECT_V2 attribute.
> > > >
> > > >
> > > > I have prepared PoC PR [3] that:
> > > >
> > > > - places the existing logic of spreading security context to
> > > > GridSecurityProcessor;
> > > >
> > > > - uses GridSecurityProcessor to get SecurityContext.
> > > >
> > > >
> > > >
> > > >1.
> > > >
> > > >
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/JDBC-thin-client-incorrect-security-context-td45929.html
> > > >2. https://issues.apache.org/jira/browse/IGNITE-12589
> > > >3. https://github.com/apache/ignite/pull/7375
> > > >
> > >
> >
>
>
> --
> Best regards,
>   Andrey Kuznetsov.
>


-- 

Best regards,
Alexei Scherbakov

Re: Let's make BinaryObjectImpl and CacheKeyObject Comparable

[Ivan Pavlukhin]

And what about non-java platforms?

Best regards,
Ivan Pavlukhin

пт, 21 февр. 2020 г. в 14:22, Ilya Kasnacheev :
>
> Hello!
>
> I don't see how we can force users to implement Comparable for BinaryObject
> keys, since BinaryObject implementation is not provided by users but by us.
>
> I think that we, however, could force it for composite non-BinaryObject
> keys.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 21 февр. 2020 г. в 00:21, Denis Magda :
>
> > Hi Ilya,
> >
> > We can oblige users to implement Comparable if they use BinaryObject keys.
> > Ignite can print out a warning if BinaryObject keys passed to putAll
> > methods don't do that.
> >
> > I also wonder how a similar task was solved for Ignite INSERTs. Our engine
> > should use BinaryObjects for compound primary keys and insert them at
> > patches. That implementation can suggest us some hints.
> >
> > -
> > Denis
> >
> >
> > On Thu, Feb 20, 2020 at 6:53 AM Ilya Kasnacheev  > >
> > wrote:
> >
> > > Hello!
> > >
> > > Since we have merged https://issues.apache.org/jira/browse/IGNITE-6804
> > we
> > > have to face an embarrassing fact that BinaryObject is not Comparable,
> > > i.e., when you do cache.withKeepBinary().putAll(), there are no obvious
> > > ways to not get a deadlock (or at least warning) here.
> > >
> > > One can use LinkedHashMap, but they will have to sort BinaryObject's on
> > > their side, which is not trivial.
> > >
> > > So my proposal is to make BinaryObjectImpl and KeyCacheObject (?)
> > > Comparable, by their binary representation. We can't add this constraint
> > to
> > > BinaryObject since it is a public interface (can we), but we can do that
> > > for the implementation types. What do you think?
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> >