Re: SHA-512 for Maven deployment

[Valentin Kulichenko]

Hi Ivan,

Thanks for your response. I've looked into the PGP plugin, and
unfortunately it looks like it only can create signatures, but not
checksums.

-Val

On Sun, Dec 27, 2020 at 11:54 PM Ivan Bessonov 
wrote:

> Hi,
>
> I've never done this before, but it seems like we need maven-gpg-plugin for
> it [1].
>
> Algorithm configuration would look like this:
> 
> --digest-algo=SHA512
> 
>
> Maybe this will help.
>
> [1]
>
> http://maven.apache.org/plugins-archives/maven-gpg-plugin-LATEST/sign-mojo.html
>
> пн, 28 дек. 2020 г. в 01:25, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Igniters,
> >
> > I've been preparing the 3.0.0-alpha1 release and got confused about the
> > requirements for checksums in Maven deployments. The Apache instruction
> [1]
> > states that MD5 is deprecated and SHA1 should be avoided in favor of
> > SHA-256 or SHA-512. However, it looks like we are still using the
> MD5/SHA1
> > combination (at least that's what the staging for 2.9.1 [2] contains).
> >
> > On top of that, I can't find an easy way to switch to another checksum -
> > Maven deploy plugin [3] creates MD5 and SHA1 files automatically and
> > doesn't seem to have any options to tweak this behavior.
> >
> > That said, I have two questions:
> >
> >1. Are we required to use SHA512 or MD5/SHA1 is OK for now?
> >2. Is there a painless way to include SHA512 in addition to MD5/SHA1?
> >
> > Can anyone shed some light on this?
> >
> > [1] https://infra.apache.org/release-signing.html#basic-facts
> > [2]
> >
> >
> https://repository.apache.org/content/repositories/orgapacheignite-1490/org/apache/ignite/ignite-core/2.9.1/
> > [3]
> https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html
> >
> > -Val
> >
>
>
> --
> Sincerely yours,
> Ivan Bessonov
>

Re: IEP-54: Schema-first approach for 3.0

[Valentin Kulichenko]

; > > > >
> > > > > > > > > > > > > > > > On Tue, Nov 24, 2020 at 1:34 PM Pavel
> Tupitsyn
> > <
> > > > > > > > > > > > ptupit...@apache.org
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Igniters,
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > I think we should support unsigned data
> > types:
> > > > > > > > > > > > > > > > > uByte, uShort, uInt, uLong
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Java does not have them, but many other
> > > languages
> > > > > do,
> > > > > > > > > > > > > > > > > and with the growing number of thin clients
> > > this
> > > > is
> > > > > > > > > > important.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > For example, in current Ignite.NET
> > > implementation
> > > > > we
> > > > > > > > store
> > > > > > > > > > > > unsigned
> > > > > > > > > > > > > > > > values
> > > > > > > > > > > > > > > > > as signed internally,
> > > > > > > > > > > > > > > > > but this is a huge pain when it comes to
> > > > metadata,
> > > > > > > binary
> > > > > > > > > > > > objects,
> > > > > > > > > > > > > > etc.
> > > > > > > > > > > > > > > > > (it is easy to deserialize int as uint when
> > you
> > > > > have
> > > > > > a
> > > > > > > > > class,
> > > > > > > > > > > but
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > BinaryObject.GetField)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Any objections?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Nov 24, 2020 at 12:28 PM Andrey
> > > > Mashenkov <
> > > > > > > > > > > > > > > > > andrey.mashen...@gmail.com> wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Denis,
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Good point. Both serializers use
> reflection
> > > > API.
> > > > > > > > > > > > > > > > > > However, we will allow users to configure
> > > > static
> > > > > > > schema
> > > > > > > > > > along
> > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > 'strict'
> > > > > > > > > > > > > > > > > > schema mode, we still need to validate
> user
> > > > > classes
> > > > > > > on
> > > > > > > > > > client
> > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > > > > against
> > > > > > > > > > > > > > > > > > the latest schema in the grid  and
> > reflection
> > > > API
> > > > > > is
> > > > > > > > the
> > > > > > > > > > only
> > > &g

Re: Ignite 3 CLI: Custom Maven repos support doesn't seem to work

[Valentin Kulichenko]

Adding correct emails...


On Mon, Dec 28, 2020 at 7:16 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Kirill, Slava,
>
> I think you've been involved in creating this functionality. Could you
> please take a look at this ticket?
> https://issues.apache.org/jira/browse/IGNITE-13927
>
> This is a blocker for the alpha1 release.
>
> -Val
>

Ignite 3 CLI: Custom Maven repos support doesn't seem to work

[Valentin Kulichenko]

Kirill, Slava,

I think you've been involved in creating this functionality. Could you
please take a look at this ticket?
https://issues.apache.org/jira/browse/IGNITE-13927

This is a blocker for the alpha1 release.

-Val

Re: [DISCUSSION] 3.0.0 Alpha release

[Valentin Kulichenko]

Hi Fedor,

Thanks again - you're really helping to improve the quality of the release.

I've fixed everything except for the progress bar issue - should not be a
blocker though, I will move it outside of the alpha if I do not succeed by
the end of tomorrow.

Can you please verify that other issues are fixed (especially
IGNITE-13936)? Here is the new staging:
https://repository.apache.org/content/repositories/orgapacheignite-1500/

-Val

On Wed, Dec 30, 2020 at 5:31 AM Fedor Malchikov 
wrote:

> Hi Valentin,
> I looked at the resources available on staging and found a number of
> additional issues:
> One blocker:
> https://issues.apache.org/jira/browse/IGNITE-13936
> and three minor:
> https://issues.apache.org/jira/browse/IGNITE-13938
> https://issues.apache.org/jira/browse/IGNITE-13937
> https://issues.apache.org/jira/browse/IGNITE-13939
>
> On Mon, Dec 28, 2020 at 10:24 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > I've also noticed that there is functionality allowing to use custom
> Maven
> > repositories. However, it is available only for the 'module add' command,
> > but not for the 'init' command. Created a ticket for this as well:
> > https://issues.apache.org/jira/browse/IGNITE-13923
> >
> > -Val
> >
> > On Mon, Dec 28, 2020 at 11:21 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Hi Fedor,
> > >
> > > Thanks for doing this!
> > >
> > > Regarding IGNITE-13921: in the first version, the tool created the
> > folders
> > > in the current directory, which created unpredictable behavior, so we
> > > switched to home as default. However, using the folder where the tool
> is
> > > located instead of the home folder seems fair - I've included the
> ticket
> > > into the alpha1. Also, note that in the next versions, there will be an
> > > option to customize the paths, so the user will not have to rely on
> > default
> > > values.
> > >
> > > If you don't mind, I suggest to postpone other issues for further
> > > development - they do not look so critical.
> > >
> > > -Val
> > >
> > > On Mon, Dec 28, 2020 at 9:45 AM Fedor Malchikov <
> fmalchi...@gridgain.com
> > >
> > > wrote:
> > >
> > >> Hello everyone,
> > >> I smoke tested the alpha version of the CLI tool on ubuntu and
> windows.
> > So
> > >> I found the following issues:
> > >> https://issues.apache.org/jira/browse/IGNITE-13914
> > >> https://issues.apache.org/jira/browse/IGNITE-13924
> > >> https://issues.apache.org/jira/browse/IGNITE-13921
> > >> https://issues.apache.org/jira/browse/IGNITE-13922
> > >> https://issues.apache.org/jira/browse/IGNITE-13920
> > >>
> > >>
> > >> On Sun, Dec 27, 2020 at 2:18 AM Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com> wrote:
> > >>
> > >> > Folks,
> > >> >
> > >> > I've created a version in Jira and a Wiki page for the alpha
> release:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0.0-alpha1
> > >> >
> > >> > Based on the scope described earlier, all code changes are
> completed -
> > >> only
> > >> > documentation is left. Code freeze starts now - only bug fixes are
> > >> > permitted.
> > >> >
> > >> > I'm planning to start the vote next week, presumably on Tuesday.
> > >> >
> > >> > -Val
> > >> >
> > >> > On Tue, Dec 22, 2020 at 9:56 AM Valentin Kulichenko <
> > >> > valentin.kuliche...@gmail.com> wrote:
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > > Yes, you are correct - no tables, caches, compute or any other
> major
> > >> > > features will be available.
> > >> > >
> > >> > > Alpha releases are generally produced in the middle of the
> > development
> > >> > > process for internal testing. In my view, that's exactly what the
> > >> > upcoming
> > >> > > release is, but let me know if you have a better naming in mind.
> > >> > >
> > >> > > -Val
> > >> > >
> > >> > > On Tue, Dec 22, 2020 at 1:56 AM Ivan Pavlukhin <
> vololo...@gmail.com
> > >
> &g

[VOTE] Release Apache Ignite 3.0.0-alpha1 RC1

[Valentin Kulichenko]

Dear Community,

The release candidate is uploaded here:
https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc1/
Maven staging:
https://repository.apache.org/content/repositories/orgapacheignite-1502/
Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc1

For more information on the purpose and the scope of the release, see this
discussion:
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html

Included Jira tickets:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1

DEVNOTES:
https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc1/DEVNOTES.md

The vote is formal, see voting guidelines:
https://www.apache.org/foundation/voting.html

+1 - accept Apache Ignite 3.0.0-alpha1 RC1
0 - don't care either way
-1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC1 (explain why)

See notes on how to verify release here:
https://www.apache.org/info/verification.html

This vote will be open for 72 hours and will close on January 7th at 12:00
am UTC:
https://www.timeanddate.com/countdown/generic?iso=20210106T16=224=cursive

-Val

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC2

[Valentin Kulichenko]

Hi Ilya,

Please see my comments below.

-Val

On Tue, Jan 5, 2021 at 2:09 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> -1 (binding)
>
> After doing mvn clean install, the modules/cli/target/ignite executable
> file is malformed, it would not run.
>
> When running it with zsh, the following error is seen:
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% ./ignite
> ./ignite: 17: set: Illegal option -o pipefail
>
> When running it with another popular shell, fish, the following error is
> seen:
> ikasnacheev@ikasnacheev-ThinkPad-P51s ~/D/a/m/c/target> ./ignite
> Failed to execute process './ignite'. Reason:
> exec: Exec format error
> The file './ignite' is marked as an executable but could not be run by the
> operating system.
>
> The reason for this, the file starts with Apache license, and then the
> #!/usr/bin/env marker is seen.
> However, by UNIX conventions, the shebang line has to be the very first.
> And only then you can specify the license.
>
> I would have thought that it will finally run as explicit bash script, but
> it won't either:
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% bash ignite
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% echo $?
> 1
>
> It won't even run as "bash ignite" even if you log in to bash first. The
> only way of running it is doing ./ignite while in bash.
>

Which OS are you running on? The tool was quite rigorously tested on bash
under Ubuntu and MacOS. I'm not sure about other shells though, will check.

Either way, does moving the env marker to the top help under all those
shells? If that's the case, the fix is easy and it might make sense to
include it and restart the vote. Please let me know if you have the fix.


> This aside, I don't really understand how we are going to publish this
> release. Will we offer downloads from ignite.apache.org? What if people
> start actually downloading it and see that it basically does nothing?
> Do we have documentation for this release? README.md is a one-liner.
> DEVNOTES.md does not even explain how to run it. There's `./docs' but no
> explanation how to use them. I have tried to go through the "Getting
> started" in the docs but also has failed miserably:
> ~/Downloads/apache-ignite-3.0.0-alpha1-src% mkdir ignite3 && cd ignite3
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% curl -L "
>
> https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download=ignite/3.0.0-alpha1/ignite
> "
> -o ignite && chmod +x ignite
>
>  % Total% Received % Xferd  Average Speed   TimeTime Time
>  Current
> Dload  Upload   Total   SpentLeft
>  Speed
>  0 00 00 0  0  0 --:--:-- --:--:-- --:--:--
> 0
> 100  10170  10170 0   1040  0 --:--:-- --:--:-- --:--:--
>  1040
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% ls
> ignite
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% ./ignite
> ./ignite: 2: Syntax error: newline unexpected
> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% head ignite
> 
>   "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>
> http://www.w3.org/1999/xhtml; lang="en" xml:lang="en">
> 
> Object not found!
> mailto:%5bno%20address%20given%5d; />
> 

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC2

[Valentin Kulichenko]

I've created a ticket for the shell issue:
https://issues.apache.org/jira/browse/IGNITE-13959. Will fix it and restart
the vote.

-Val

On Tue, Jan 5, 2021 at 9:34 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Hi Ilya,
>
> Please see my comments below.
>
> -Val
>
> On Tue, Jan 5, 2021 at 2:09 AM Ilya Kasnacheev 
> wrote:
>
>> Hello!
>>
>> -1 (binding)
>>
>> After doing mvn clean install, the modules/cli/target/ignite executable
>> file is malformed, it would not run.
>>
>> When running it with zsh, the following error is seen:
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% ./ignite
>> ./ignite: 17: set: Illegal option -o pipefail
>>
>> When running it with another popular shell, fish, the following error is
>> seen:
>> ikasnacheev@ikasnacheev-ThinkPad-P51s ~/D/a/m/c/target> ./ignite
>> Failed to execute process './ignite'. Reason:
>> exec: Exec format error
>> The file './ignite' is marked as an executable but could not be run by the
>> operating system.
>>
>> The reason for this, the file starts with Apache license, and then the
>> #!/usr/bin/env marker is seen.
>> However, by UNIX conventions, the shebang line has to be the very first.
>> And only then you can specify the license.
>>
>> I would have thought that it will finally run as explicit bash script, but
>> it won't either:
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% bash ignite
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/modules/cli/target% echo $?
>> 1
>>
>> It won't even run as "bash ignite" even if you log in to bash first. The
>> only way of running it is doing ./ignite while in bash.
>>
>
> Which OS are you running on? The tool was quite rigorously tested on bash
> under Ubuntu and MacOS. I'm not sure about other shells though, will check.
>
> Either way, does moving the env marker to the top help under all those
> shells? If that's the case, the fix is easy and it might make sense to
> include it and restart the vote. Please let me know if you have the fix.
>
>
>> This aside, I don't really understand how we are going to publish this
>> release. Will we offer downloads from ignite.apache.org? What if people
>> start actually downloading it and see that it basically does nothing?
>> Do we have documentation for this release? README.md is a one-liner.
>> DEVNOTES.md does not even explain how to run it. There's `./docs' but no
>> explanation how to use them. I have tried to go through the "Getting
>> started" in the docs but also has failed miserably:
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src% mkdir ignite3 && cd ignite3
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% curl -L "
>>
>> https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download=ignite/3.0.0-alpha1/ignite
>> "
>> -o ignite && chmod +x ignite
>>
>>  % Total% Received % Xferd  Average Speed   TimeTime Time
>>  Current
>> Dload  Upload   Total   SpentLeft
>>  Speed
>>  0 00 00 0  0  0 --:--:-- --:--:-- --:--:--
>> 0
>> 100  10170  10170 0   1040  0 --:--:-- --:--:-- --:--:--
>>  1040
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% ls
>> ignite
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% ./ignite
>> ./ignite: 2: Syntax error: newline unexpected
>> ~/Downloads/apache-ignite-3.0.0-alpha1-src/ignite3% head ignite
>> 
>> >  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd;>
>> http://www.w3.org/1999/xhtml; lang="en" xml:lang="en">
>> 
>> Object not found!
>> mailto:%5bno%20address%20given%5d; />
>> 

[VOTE] Release Apache Ignite 3.0.0-alpha1 RC3

[Valentin Kulichenko]

Dear Community,

The release candidate is uploaded here:
https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc3/
Maven staging:
https://repository.apache.org/content/repositories/orgapacheignite-1504/
Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc3

For more information on the purpose and the scope of the release, see this
discussion:
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html

Included Jira tickets:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1

Fixes included on top of 3.0.0-alpha1 RC2:

   - https://issues.apache.org/jira/browse/IGNITE-13959 (ignite.sh doesn't
   work under 'ksh' or 'fish' shells)
   - Updated DEVNOTES with a smoke test procedure that can be used for a
   release candidate.

Fixes included on top of 3.0.0-alpha1 RC1:

   - https://issues.apache.org/jira/browse/IGNITE-13950 (IgniteCliInterfaceTest
   can fail on Windows)

DEVNOTES:
https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc3/DEVNOTES.md

The vote is formal, see voting guidelines:
https://www.apache.org/foundation/voting.html

+1 - accept Apache Ignite 3.0.0-alpha1 RC3
0 - don't care either way
-1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC3 (explain why)

See notes on how to verify release here:
https://www.apache.org/info/verification.html

This vote will be open for 72 hours and will close on January 8th at 12:00
am UTC:
https://www.timeanddate.com/countdown/generic?iso=20210108T13=224=cursive

-Val

Re: Error Codes

[Valentin Kulichenko]

Mike, Denis,

Having error codes certainly makes sense. Please send the ticket link, and
we'll go from there.

-Val

On Mon, Jan 4, 2021 at 7:50 PM Denis Magda  wrote:

> Do back this idea up of having a glossary of common errors. There is even
> a ticket for that I created a couple of years ago. Can search for it later.
>
> Val, how about the 3.0 suggestion? Let’s introduce error codes.
>
> On Monday, January 4, 2021, Michael Cherkasov 
> wrote:
>
>> Hi Ilya,
>>
>> It's about logs only, I don't think we need this at the API level. Error
>> codes will make the solutions more searchable.
>> Plus we can build troubleshooting guides based on it, it will help us
>> gather information from user list and StackOverflow.
>>
>> Even a solution for trivial cases will be helpful, once I was requested
>> to join the call late evening because ignite failed to copy WAL file and
>> there just was no space on the disk.
>> While the error was obvious for me, it's not obvious for all users.
>>
>> Let's start from something simple, just assign error codes to
>> absolutely all exceptions first. So next year or two user list will full of
>> error codes and solutions for them.
>>
>> Might be it's a change for Ignite 3.0? @Val, I think you can help with
>> this question.
>>
>> Any thoughts/comments?
>>
>> Thanks,
>> Mike.
>>
>> сб, 2 янв. 2021 г. в 12:18, Ilya Kasnacheev :
>>
>>> Hello!
>>>
>>> I don't think there's a direct link between an exception thrown in
>>> depths of Ignite code, and specific error which may be reported to user.
>>>
>>> A notorious example is CorruptedTreeException which is known to be
>>> thrown due to incorrect field type in binary object or bad SQL cast. So we
>>> could document it "If you get IGN13 error this means your persistence is
>>> corrupted beyond repair. This, or you have a typo in your SQL." - of course
>>> it will not help anyone.
>>>
>>> This means we can't get to the desired result by application of 1.
>>>
>>> There's got to be a different plan. First of all, we need to decide
>>> what's our target. Is it log, or is it API?
>>>
>>> Regards,
>>> --
>>> Ilya Kasnacheev
>>>
>>>
>>> пт, 1 янв. 2021 г. в 02:07, Michael Cherkasov <
>>> michael.cherka...@gmail.com>:
>>>
 Hi folks,

 I was thinking how we can simplify Ignite clusters troubleshooting and
 the best of course if the cluster can do self-healing, like transaction
 cancellation if tx blocks exchange or note restart on OOM error. However,
 sometimes those mechanisms don't work well or user interaction is required.
 Not all errors are obvious for users and it's not clear what actions
 required to restore the cluster.
 If you google exceptions or error messages and the results can be
 ambiguous and not certain because different errors can have similar
 exceptions and you need to analyze stack trace to distinguish them. So
 googling isn't a straight and easy process in this case.
 Almost all major DBs have error codes[1][2][3]
 Let's do the same for Ignite, error codes easy to google, so user/dev
 list will be significantly more useful. We can have documentation with an
 error code registry and solutions for the errors.

 To implement this we need to do the following:
 1. all error messages/exceptions must have a unique error code(so, all
 new PR must NOT be accepted if any exceptions/errors don't have error
 codes.)
 2. to avoid error code duplication, all error codes will be stored as
 files under some folder.
 3. those files can be a source of documentation for this error code.

 All this files can be empty, but futher, if exception will apper on
 user list and someone will find solution, first, other people can easialy
 google it by error code, and second, we can build documentation for this
 error code base on user-list thread/stackoverflow/other source.

 Any thoughts?

 [1] Mysql
 https://dev.mysql.com/doc/refman/8.0/en/error-message-elements.html
 [2] OracleDB https://docs.oracle.com/pls/db92/db92.error_search
 [3] PostgreSQL
 https://www.postgresql.org/docs/10/errcodes-appendix.html

 Thanks,
 Mike.

>>>
>
> --
> -
> Denis
>
>

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC1

[Valentin Kulichenko]

I've fixed the issue. Since there are no other votes at this point, I will
restart the vote.

-Val

On Mon, Jan 4, 2021 at 12:43 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> I've identified an issue with tests on Windows:
> https://issues.apache.org/jira/browse/IGNITE-13950. It is purely a test
> issue related to the line separating characters. I don't think it should be
> a blocker for this alpha release though - we can fix it going forward.
>
> Nikita, is this the issue you're having, or there is something else?
>
> -Val
>
> On Mon, Jan 4, 2021 at 11:19 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Nikita,
>>
>> Can you show the error it fails with? Can you also try to exclude tests
>> by running 'mvn clean package -DskipTests'?
>>
>> As for the binaries, we always include them. An Apache release must
>> contain a source package, but it doesn't have to be limited to it. 'ignite'
>> and 'ignite.exe' binaries will be downloadable from the website, and for
>> that, they have to be released.
>>
>> -Val
>>
>> On Mon, Jan 4, 2021 at 10:52 AM Nikita Ivanov 
>> wrote:
>>
>>> -1
>>>
>>> - 'mvn clean package' fails on Win10 (at least)
>>> - binary artifacts in release location (ignite.exe)
>>>
>>> --
>>> Nikita Ivanov
>>>
>>>
>>>
>>> On Sun, Jan 3, 2021 at 4:00 PM Valentin Kulichenko <
>>> valentin.kuliche...@gmail.com> wrote:
>>>
>>> > Dear Community,
>>> >
>>> > The release candidate is uploaded here:
>>> > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc1/
>>> > Maven staging:
>>> >
>>> https://repository.apache.org/content/repositories/orgapacheignite-1502/
>>> > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc1
>>> >
>>> > For more information on the purpose and the scope of the release, see
>>> this
>>> > discussion:
>>> >
>>> >
>>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html
>>> >
>>> > Included Jira tickets:
>>> >
>>> >
>>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1
>>> >
>>> > DEVNOTES:
>>> > https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc1/DEVNOTES.md
>>> >
>>> > The vote is formal, see voting guidelines:
>>> > https://www.apache.org/foundation/voting.html
>>> >
>>> > +1 - accept Apache Ignite 3.0.0-alpha1 RC1
>>> > 0 - don't care either way
>>> > -1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC1 (explain why)
>>> >
>>> > See notes on how to verify release here:
>>> > https://www.apache.org/info/verification.html
>>> >
>>> > This vote will be open for 72 hours and will close on January 7th at
>>> 12:00
>>> > am UTC:
>>> >
>>> >
>>> https://www.timeanddate.com/countdown/generic?iso=20210106T16=224=cursive
>>> >
>>> > -Val
>>> >
>>>
>>

[CANCEL] [VOTE] Release Apache Ignite 3.0.0-alpha1 RC1

[Valentin Kulichenko]

Canceling due to failing build on Windows:
https://issues.apache.org/jira/browse/IGNITE-13950

-Val

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC3

[Valentin Kulichenko]

Vote closed with four "+1" votes:
http://apache-ignite-developers.2346864.n4.nabble.com/RESULT-VOTE-Release-Apache-Ignite-3-0-0-alpha1-RC3-td50964.html

-Val

On Sun, Jan 10, 2021 at 1:10 PM Saikat Maitra 
wrote:

> +1 (binding)
>
> On Sun, Jan 10, 2021 at 3:01 PM Andrey Gura  wrote:
>
> > +1 (binding)
> >
> > On Thu, Jan 7, 2021 at 10:42 PM Vladimir Pligin 
> > wrote:
> > >
> > > +1 (non-binding)
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>

[RESULT] [VOTE] Release Apache Ignite 3.0.0-alpha1 RC3

[Valentin Kulichenko]

Igniters,

Apache Ignite 3.0.0 RC3 has been accepted.

4 "+1" votes received:
   - Denis Magda (binding)
   - Vladimir Pligin (non-binding)
   - Andrey Gura (binding)
   - Saikat Maitra (binding)

No "0" or "-1" votes.

Vote thread:
http://apache-ignite-developers.2346864.n4.nabble.com/VOTE-Release-Apache-Ignite-3-0-0-alpha1-RC3-td50948.html

-Val

Re: SHA-512 for Maven deployment

[Valentin Kulichenko]

Hi Ivan,

No, I haven't found a way yet. SHA1 still works, but I believe we should
consider using better options in future releases.

Do you have any ideas on how to implement this?

-Val

On Wed, Jan 13, 2021 at 8:21 AM Ivan Pavlukhin  wrote:

> Folks,
>
> Were you able to resolve this?
>
> 2020-12-28 22:15 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Hi Ivan,
> >
> > Thanks for your response. I've looked into the PGP plugin, and
> > unfortunately it looks like it only can create signatures, but not
> > checksums.
> >
> > -Val
> >
> > On Sun, Dec 27, 2020 at 11:54 PM Ivan Bessonov 
> > wrote:
> >
> >> Hi,
> >>
> >> I've never done this before, but it seems like we need maven-gpg-plugin
> >> for
> >> it [1].
> >>
> >> Algorithm configuration would look like this:
> >> 
> >> --digest-algo=SHA512
> >> 
> >>
> >> Maybe this will help.
> >>
> >> [1]
> >>
> >>
> http://maven.apache.org/plugins-archives/maven-gpg-plugin-LATEST/sign-mojo.html
> >>
> >> пн, 28 дек. 2020 г. в 01:25, Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com>:
> >>
> >> > Igniters,
> >> >
> >> > I've been preparing the 3.0.0-alpha1 release and got confused about
> the
> >> > requirements for checksums in Maven deployments. The Apache
> instruction
> >> [1]
> >> > states that MD5 is deprecated and SHA1 should be avoided in favor of
> >> > SHA-256 or SHA-512. However, it looks like we are still using the
> >> MD5/SHA1
> >> > combination (at least that's what the staging for 2.9.1 [2] contains).
> >> >
> >> > On top of that, I can't find an easy way to switch to another checksum
> >> > -
> >> > Maven deploy plugin [3] creates MD5 and SHA1 files automatically and
> >> > doesn't seem to have any options to tweak this behavior.
> >> >
> >> > That said, I have two questions:
> >> >
> >> >1. Are we required to use SHA512 or MD5/SHA1 is OK for now?
> >> >2. Is there a painless way to include SHA512 in addition to
> >> > MD5/SHA1?
> >> >
> >> > Can anyone shed some light on this?
> >> >
> >> > [1] https://infra.apache.org/release-signing.html#basic-facts
> >> > [2]
> >> >
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapacheignite-1490/org/apache/ignite/ignite-core/2.9.1/
> >> > [3]
> >> https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html
> >> >
> >> > -Val
> >> >
> >>
> >>
> >> --
> >> Sincerely yours,
> >> Ivan Bessonov
> >>
> >
>
>
> --
>
> Best regards,
> Ivan Pavlukhin
>

Re: [DISCUSSION] 3.0.0 Alpha release

[Valentin Kulichenko]

Hi Ivan,

Yesterday I created an email thread that can be used for this:
-
http://apache-ignite-developers.2346864.n4.nabble.com/Looking-for-feedback-on-the-Ignite-3-0-0-Alpha-td51012.html
-
http://apache-ignite-users.70518.x6.nabble.com/Looking-for-feedback-on-the-Ignite-3-0-0-Alpha-td35089.html

In addition, we are planning to have a virtual meetup at the end of the
month - we can discuss there as well. The meetup will be scheduled soon.

-Val

On Wed, Jan 13, 2021 at 4:25 AM Ivan Pavlukhin  wrote:

> Val, Igniters,
>
> I thought about possible results of such release. How do we expect to
> receive feedback? It would be interesting personally for me to see
> feedback/results summary.
>
> 2021-01-01 2:59 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Igniters,
> >
> > Big thanks to everyone involved in the preparation for this alpha
> release!
> >
> > Throughout this week, the current codebase has been tested on Linux and
> > Windows, all the major issues have been resolved, and the Getting Started
> > guide has been created. We're now ready for the vote - I will start it
> > sometime during the weekend or early next week.
> >
> > Happy New Year everyone!
> >
> > -Val
> >
> > On Thu, Dec 31, 2020 at 3:03 AM Fedor Malchikov  >
> > wrote:
> >
> >> Hi Valentin,
> >>
> >> Yes, of course.
> >> I closed tickets for fixed issues and created a few more new ones.
> >>
> >> On Thu, Dec 31, 2020 at 8:57 AM Valentin Kulichenko <
> >> valentin.kuliche...@gmail.com> wrote:
> >>
> >> > Hi Fedor,
> >> >
> >> > Thanks again - you're really helping to improve the quality of the
> >> release.
> >> >
> >> > I've fixed everything except for the progress bar issue - should not
> be
> >> > a
> >> > blocker though, I will move it outside of the alpha if I do not
> succeed
> >> by
> >> > the end of tomorrow.
> >> >
> >> > Can you please verify that other issues are fixed (especially
> >> > IGNITE-13936)? Here is the new staging:
> >> >
> https://repository.apache.org/content/repositories/orgapacheignite-1500/
> >> >
> >> > -Val
> >> >
> >> > On Wed, Dec 30, 2020 at 5:31 AM Fedor Malchikov
> >> >  >> >
> >> > wrote:
> >> >
> >> > > Hi Valentin,
> >> > > I looked at the resources available on staging and found a number of
> >> > > additional issues:
> >> > > One blocker:
> >> > > https://issues.apache.org/jira/browse/IGNITE-13936
> >> > > and three minor:
> >> > > https://issues.apache.org/jira/browse/IGNITE-13938
> >> > > https://issues.apache.org/jira/browse/IGNITE-13937
> >> > > https://issues.apache.org/jira/browse/IGNITE-13939
> >> > >
> >> > > On Mon, Dec 28, 2020 at 10:24 PM Valentin Kulichenko <
> >> > > valentin.kuliche...@gmail.com> wrote:
> >> > >
> >> > > > I've also noticed that there is functionality allowing to use
> >> > > > custom
> >> > > Maven
> >> > > > repositories. However, it is available only for the 'module add'
> >> > command,
> >> > > > but not for the 'init' command. Created a ticket for this as well:
> >> > > > https://issues.apache.org/jira/browse/IGNITE-13923
> >> > > >
> >> > > > -Val
> >> > > >
> >> > > > On Mon, Dec 28, 2020 at 11:21 AM Valentin Kulichenko <
> >> > > > valentin.kuliche...@gmail.com> wrote:
> >> > > >
> >> > > > > Hi Fedor,
> >> > > > >
> >> > > > > Thanks for doing this!
> >> > > > >
> >> > > > > Regarding IGNITE-13921: in the first version, the tool created
> >> > > > > the
> >> > > > folders
> >> > > > > in the current directory, which created unpredictable behavior,
> >> > > > > so
> >> we
> >> > > > > switched to home as default. However, using the folder where the
> >> tool
> >> > > is
> >> > > > > located instead of the home folder seems fair - I've included
> the
> >> > > ticket
> >> > > > > into the alpha1. Also, note that in the ne

Re: SHA-512 for Maven deployment

[Valentin Kulichenko]

Andrey,

Did you try on the 2.x or 3.x?

I've just tried to do the same in ignite-3, but it didn't work for me. I've
updated the parent pom version to 23 and ran "mvn clean deploy
-Papache-release". The source package is now signed with SHA512, which is
good, but there was no effect on the JAR artifacts. As a matter of fact, I
don't see any checksum files for them. My guess is that by default they are
generated by the deploy plugin, during the upload to Maven. Here is the
resulting staging (still MD5 and SHA1):
https://repository.apache.org/content/repositories/orgapacheignite-1505/

Does it behave in the same way for you?

-Val

On Thu, Jan 14, 2021 at 3:30 AM Andrey Mashenkov 
wrote:

> I've made "mvn clean install" with enabled "apache-release" profile and see
> *.sha-512 checksum files in target directories.
> So, upgrading to the latest apache parent looks sufficient.
>
>
> On Thu, Jan 14, 2021 at 12:30 PM Petr Ivanov  wrote:
>
> > Is seems that parent is already updated in
> > https://issues.apache.org/jira/browse/IGNITE-13987 <
> > https://issues.apache.org/jira/browse/IGNITE-13987>
> >
> >
> >
> > > On 14 Jan 2021, at 01:57, Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> > >
> > > Andrey,
> > >
> > > This sounds even better. Can you create a ticket for this change?
> > >
> > > -Val
> > >
> > > On Wed, Jan 13, 2021 at 2:34 PM Andrey Mashenkov <
> > andrey.mashen...@gmail.com>
> > > wrote:
> > >
> > >> Val,
> > >>
> > >> I've just found Maven projects use SHA-512.
> > >> I passed through commits and found they just switched to newer parent
> > >> org.apache:apache pom.
> > >> I've compared our current parent pom with the latest available one
> > >> (org.apache:apache:16 vs org.apache:apache:23)
> > >> and then found checksum-maven-plugin was added [1] somewhen in
> between.
> > >>
> > >> So, seems we have to switched to newer apache pom and maybe add
> > >> checksum-maven-plugin
> > >> to our main pom.
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://github.com/apache/maven-apache-parent/commit/a46aa52b4b56d9b7aa62e1b8cbea5ff0af434a
> > >>
> > >> On Wed, Jan 13, 2021 at 10:41 PM Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com> wrote:
> > >>
> > >>> Hi Andrey,
> > >>>
> > >>> This indeed sounds like the cleanest way. I don't know how much
> effort
> > >> that
> > >>> would be though.
> > >>>
> > >>> -Val
> > >>>
> > >>> On Wed, Jan 13, 2021 at 11:01 AM Andrey Mashenkov <
> > >>> andrey.mashen...@gmail.com> wrote:
> > >>>
> > >>>> Maybe, we could donate to maven plugin possibility to switch to
> > >> SHA-512.
> > >>>> Hopefully, a new plugin version will be released before we have any
> > >>> release
> > >>>> candidate.
> > >>>>
> > >>>> Is it looks like a big deal?
> > >>>>
> > >>>> ср, 13 янв. 2021 г., 21:32 Valentin Kulichenko <
> > >>>> valentin.kuliche...@gmail.com>:
> > >>>>
> > >>>>> Hi Ivan,
> > >>>>>
> > >>>>> No, I haven't found a way yet. SHA1 still works, but I believe we
> > >>> should
> > >>>>> consider using better options in future releases.
> > >>>>>
> > >>>>> Do you have any ideas on how to implement this?
> > >>>>>
> > >>>>> -Val
> > >>>>>
> > >>>>> On Wed, Jan 13, 2021 at 8:21 AM Ivan Pavlukhin <
> vololo...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Folks,
> > >>>>>>
> > >>>>>> Were you able to resolve this?
> > >>>>>>
> > >>>>>> 2020-12-28 22:15 GMT+03:00, Valentin Kulichenko <
> > >>>>>> valentin.kuliche...@gmail.com>:
> > >>>>>>> Hi Ivan,
> > >>>>>>>
> > >>>>>>> Thanks for your response. I've looked into the PGP plugin, and
> > >>>>>>> unfortunately it looks like it only can create signatures, bu

Re: Looking for feedback on the Ignite 3.0.0 Alpha

[Valentin Kulichenko]

The meetup has been scheduled, please RSVP here:
https://www.meetup.com/Apache-Ignite-Virtual-Meetup/events/275722317/

-Val

On Wed, Jan 13, 2021 at 11:21 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Link to the Getting Started Guide:
> https://ignite.apache.org/docs/3.0.0-alpha/quick-start/getting-started-guide
>
> -Val
>
> On Tue, Jan 12, 2021 at 7:55 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
>> Igniters,
>>
>> I'm excited to announce that the first alpha build of the Ignite 3 is out
>> and available for download!
>>
>> Ignite 3 is the new project that was initiated by the Ignite community
>> last year. Please refer to this page if you want to learn more:
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
>>
>> The just-released alpha build is a sneak peek into the future of Ignite.
>> It doesn't represent a fully-functional product (no discovery, caches,
>> compute, etc.), but demonstrates major mechanics of how you will interact
>> with Ignite going forward.
>>
>> The main goal of the release is to gather feedback from the community so
>> that we can adjust further development if needed. That said, I would ask
>> and encourage everyone to go through the Getting Started Guide [1] and play
>> with the build. If you have any questions, issues, concerns, wishes,
>> requests, or any other thoughts, please reply directly to this thread. We
>> will carefully accumulate all the feedback and make sure it is considered
>> going forward.
>>
>> Another opportunity to share your feedback will come closer to the end of
>> January when we will have a virtual meetup. I will present a quick demo of
>> the alpha build, after which we will have an open discussion. Please stay
>> tuned - I will send a message here when the meetup is scheduled.
>>
>> -Val
>>
>

Re: Looking for feedback on the Ignite 3.0.0 Alpha

[Valentin Kulichenko]

Link to the Getting Started Guide:
https://ignite.apache.org/docs/3.0.0-alpha/quick-start/getting-started-guide

-Val

On Tue, Jan 12, 2021 at 7:55 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Igniters,
>
> I'm excited to announce that the first alpha build of the Ignite 3 is out
> and available for download!
>
> Ignite 3 is the new project that was initiated by the Ignite community
> last year. Please refer to this page if you want to learn more:
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0
>
> The just-released alpha build is a sneak peek into the future of Ignite.
> It doesn't represent a fully-functional product (no discovery, caches,
> compute, etc.), but demonstrates major mechanics of how you will interact
> with Ignite going forward.
>
> The main goal of the release is to gather feedback from the community so
> that we can adjust further development if needed. That said, I would ask
> and encourage everyone to go through the Getting Started Guide [1] and play
> with the build. If you have any questions, issues, concerns, wishes,
> requests, or any other thoughts, please reply directly to this thread. We
> will carefully accumulate all the feedback and make sure it is considered
> going forward.
>
> Another opportunity to share your feedback will come closer to the end of
> January when we will have a virtual meetup. I will present a quick demo of
> the alpha build, after which we will have an open discussion. Please stay
> tuned - I will send a message here when the meetup is scheduled.
>
> -Val
>

Re: SHA-512 for Maven deployment

[Valentin Kulichenko]

Hi Andrey,

This indeed sounds like the cleanest way. I don't know how much effort that
would be though.

-Val

On Wed, Jan 13, 2021 at 11:01 AM Andrey Mashenkov <
andrey.mashen...@gmail.com> wrote:

> Maybe, we could donate to maven plugin possibility to switch to SHA-512.
> Hopefully, a new plugin version will be released before we have any release
> candidate.
>
> Is it looks like a big deal?
>
> ср, 13 янв. 2021 г., 21:32 Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Hi Ivan,
> >
> > No, I haven't found a way yet. SHA1 still works, but I believe we should
> > consider using better options in future releases.
> >
> > Do you have any ideas on how to implement this?
> >
> > -Val
> >
> > On Wed, Jan 13, 2021 at 8:21 AM Ivan Pavlukhin 
> > wrote:
> >
> > > Folks,
> > >
> > > Were you able to resolve this?
> > >
> > > 2020-12-28 22:15 GMT+03:00, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > > > Hi Ivan,
> > > >
> > > > Thanks for your response. I've looked into the PGP plugin, and
> > > > unfortunately it looks like it only can create signatures, but not
> > > > checksums.
> > > >
> > > > -Val
> > > >
> > > > On Sun, Dec 27, 2020 at 11:54 PM Ivan Bessonov <
> bessonov...@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I've never done this before, but it seems like we need
> > maven-gpg-plugin
> > > >> for
> > > >> it [1].
> > > >>
> > > >> Algorithm configuration would look like this:
> > > >> 
> > > >> --digest-algo=SHA512
> > > >> 
> > > >>
> > > >> Maybe this will help.
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> http://maven.apache.org/plugins-archives/maven-gpg-plugin-LATEST/sign-mojo.html
> > > >>
> > > >> пн, 28 дек. 2020 г. в 01:25, Valentin Kulichenko <
> > > >> valentin.kuliche...@gmail.com>:
> > > >>
> > > >> > Igniters,
> > > >> >
> > > >> > I've been preparing the 3.0.0-alpha1 release and got confused
> about
> > > the
> > > >> > requirements for checksums in Maven deployments. The Apache
> > > instruction
> > > >> [1]
> > > >> > states that MD5 is deprecated and SHA1 should be avoided in favor
> of
> > > >> > SHA-256 or SHA-512. However, it looks like we are still using the
> > > >> MD5/SHA1
> > > >> > combination (at least that's what the staging for 2.9.1 [2]
> > contains).
> > > >> >
> > > >> > On top of that, I can't find an easy way to switch to another
> > checksum
> > > >> > -
> > > >> > Maven deploy plugin [3] creates MD5 and SHA1 files automatically
> and
> > > >> > doesn't seem to have any options to tweak this behavior.
> > > >> >
> > > >> > That said, I have two questions:
> > > >> >
> > > >> >1. Are we required to use SHA512 or MD5/SHA1 is OK for now?
> > > >> >2. Is there a painless way to include SHA512 in addition to
> > > >> > MD5/SHA1?
> > > >> >
> > > >> > Can anyone shed some light on this?
> > > >> >
> > > >> > [1] https://infra.apache.org/release-signing.html#basic-facts
> > > >> > [2]
> > > >> >
> > > >> >
> > > >>
> > >
> >
> https://repository.apache.org/content/repositories/orgapacheignite-1490/org/apache/ignite/ignite-core/2.9.1/
> > > >> > [3]
> > > >>
> https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html
> > > >> >
> > > >> > -Val
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >> Sincerely yours,
> > > >> Ivan Bessonov
> > > >>
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Ivan Pavlukhin
> > >
> >
>

Re: SHA-512 for Maven deployment

[Valentin Kulichenko]

Andrey,

This sounds even better. Can you create a ticket for this change?

-Val

On Wed, Jan 13, 2021 at 2:34 PM Andrey Mashenkov 
wrote:

> Val,
>
> I've just found Maven projects use SHA-512.
> I passed through commits and found they just switched to newer parent
> org.apache:apache pom.
> I've compared our current parent pom with the latest available one
> (org.apache:apache:16 vs org.apache:apache:23)
> and then found checksum-maven-plugin was added [1] somewhen in between.
>
> So, seems we have to switched to newer apache pom and maybe add
> checksum-maven-plugin
> to our main pom.
>
> [1]
>
> https://github.com/apache/maven-apache-parent/commit/a46aa52b4b56d9b7aa62e1b8cbea5ff0af434a
>
> On Wed, Jan 13, 2021 at 10:41 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Andrey,
> >
> > This indeed sounds like the cleanest way. I don't know how much effort
> that
> > would be though.
> >
> > -Val
> >
> > On Wed, Jan 13, 2021 at 11:01 AM Andrey Mashenkov <
> > andrey.mashen...@gmail.com> wrote:
> >
> > > Maybe, we could donate to maven plugin possibility to switch to
> SHA-512.
> > > Hopefully, a new plugin version will be released before we have any
> > release
> > > candidate.
> > >
> > > Is it looks like a big deal?
> > >
> > > ср, 13 янв. 2021 г., 21:32 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com>:
> > >
> > > > Hi Ivan,
> > > >
> > > > No, I haven't found a way yet. SHA1 still works, but I believe we
> > should
> > > > consider using better options in future releases.
> > > >
> > > > Do you have any ideas on how to implement this?
> > > >
> > > > -Val
> > > >
> > > > On Wed, Jan 13, 2021 at 8:21 AM Ivan Pavlukhin 
> > > > wrote:
> > > >
> > > > > Folks,
> > > > >
> > > > > Were you able to resolve this?
> > > > >
> > > > > 2020-12-28 22:15 GMT+03:00, Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com>:
> > > > > > Hi Ivan,
> > > > > >
> > > > > > Thanks for your response. I've looked into the PGP plugin, and
> > > > > > unfortunately it looks like it only can create signatures, but
> not
> > > > > > checksums.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Sun, Dec 27, 2020 at 11:54 PM Ivan Bessonov <
> > > bessonov...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi,
> > > > > >>
> > > > > >> I've never done this before, but it seems like we need
> > > > maven-gpg-plugin
> > > > > >> for
> > > > > >> it [1].
> > > > > >>
> > > > > >> Algorithm configuration would look like this:
> > > > > >> 
> > > > > >> --digest-algo=SHA512
> > > > > >> 
> > > > > >>
> > > > > >> Maybe this will help.
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> http://maven.apache.org/plugins-archives/maven-gpg-plugin-LATEST/sign-mojo.html
> > > > > >>
> > > > > >> пн, 28 дек. 2020 г. в 01:25, Valentin Kulichenko <
> > > > > >> valentin.kuliche...@gmail.com>:
> > > > > >>
> > > > > >> > Igniters,
> > > > > >> >
> > > > > >> > I've been preparing the 3.0.0-alpha1 release and got confused
> > > about
> > > > > the
> > > > > >> > requirements for checksums in Maven deployments. The Apache
> > > > > instruction
> > > > > >> [1]
> > > > > >> > states that MD5 is deprecated and SHA1 should be avoided in
> > favor
> > > of
> > > > > >> > SHA-256 or SHA-512. However, it looks like we are still using
> > the
> > > > > >> MD5/SHA1
> > > > > >> > combination (at least that's what the staging for 2.9.1 [2]
> > > > contains).
> > > > > >> >
> > > > > >> > On top of that, I can't find an easy way to switch to another
> > > > checksum
> > > > > >> > -
> > > > > >> > Maven deploy plugin [3] creates MD5 and SHA1 files
> automatically
> > > and
> > > > > >> > doesn't seem to have any options to tweak this behavior.
> > > > > >> >
> > > > > >> > That said, I have two questions:
> > > > > >> >
> > > > > >> >1. Are we required to use SHA512 or MD5/SHA1 is OK for now?
> > > > > >> >2. Is there a painless way to include SHA512 in addition to
> > > > > >> > MD5/SHA1?
> > > > > >> >
> > > > > >> > Can anyone shed some light on this?
> > > > > >> >
> > > > > >> > [1] https://infra.apache.org/release-signing.html#basic-facts
> > > > > >> > [2]
> > > > > >> >
> > > > > >> >
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://repository.apache.org/content/repositories/orgapacheignite-1490/org/apache/ignite/ignite-core/2.9.1/
> > > > > >> > [3]
> > > > > >>
> > > https://maven.apache.org/plugins/maven-deploy-plugin/deploy-mojo.html
> > > > > >> >
> > > > > >> > -Val
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Sincerely yours,
> > > > > >> Ivan Bessonov
> > > > > >>
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Best regards,
> > > > > Ivan Pavlukhin
> > > > >
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrey V. Mashenkov
>

[ANNOUNCE] Apache Ignite 3.0.0-alpha1 Released

[Valentin Kulichenko]

The Apache Ignite Community is pleased to announce the release of
Apache Ignite 3.0.0-alpha1.

Apache Ignite [1] is a distributed database for in-memory speed and
high-performance computing.

Last year, the Ignite community initiated the "Ignite 3" project [2]. The
goal of the project is to modernize Ignite and fix various usability issues
that have been accumulated through the years of its existence. The project
introduces several changes related to architecture, API, and tooling - all
those changes will be incorporated into the next major release (Ignite
3.0.0).

Ignite 3.0.0-alpha1 provides a sneak peek into the future of Apache Ignite.
This alpha build does not represent a functional product - it does not
incorporate cluster discovery, data storage, compute capabilities, etc.
However, it demonstrates the major mechanics of how you will interact with
Ignite in the future.

Some of the major highlights are:
- Single-file download (instead of downloading a huge ZIP file that has
hundreds of files and a complicated structure, you get a single script, the
CLI tool, which you can use for all further operations).
- Ability to install core Ignite artifacts and external dependencies via
Maven.
- Ability to manage locally running nodes (start, stop, inspect).
- Ability to connect to a cluster, acquire the current configuration, and
update some of the dynamic parameters.

The best way to get a good feel for the alpha build is to review the
Getting Started Guide:
https://ignite.apache.org/docs/3.0.0-alpha/quick-start/getting-started-guide

Downloads are available here:
https://ignite.apache.org/download.cgi

Please feel free to reach out to the Apache Ignite community in case you
have any questions: https://ignite.apache.org/community/resources.html#ask

[1] https://ignite.apache.org
[2] https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0

Regards,
Valentin Kulichenko
Apache Ignite PMC

Looking for feedback on the Ignite 3.0.0 Alpha

[Valentin Kulichenko]

Igniters,

I'm excited to announce that the first alpha build of the Ignite 3 is out
and available for download!

Ignite 3 is the new project that was initiated by the Ignite community last
year. Please refer to this page if you want to learn more:
https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+3.0

The just-released alpha build is a sneak peek into the future of Ignite. It
doesn't represent a fully-functional product (no discovery, caches,
compute, etc.), but demonstrates major mechanics of how you will interact
with Ignite going forward.

The main goal of the release is to gather feedback from the community so
that we can adjust further development if needed. That said, I would ask
and encourage everyone to go through the Getting Started Guide [1] and play
with the build. If you have any questions, issues, concerns, wishes,
requests, or any other thoughts, please reply directly to this thread. We
will carefully accumulate all the feedback and make sure it is considered
going forward.

Another opportunity to share your feedback will come closer to the end of
January when we will have a virtual meetup. I will present a quick demo of
the alpha build, after which we will have an open discussion. Please stay
tuned - I will send a message here when the meetup is scheduled.

-Val

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC1

[Valentin Kulichenko]

Nikita,

Can you show the error it fails with? Can you also try to exclude tests by
running 'mvn clean package -DskipTests'?

As for the binaries, we always include them. An Apache release must contain
a source package, but it doesn't have to be limited to it. 'ignite' and
'ignite.exe' binaries will be downloadable from the website, and for that,
they have to be released.

-Val

On Mon, Jan 4, 2021 at 10:52 AM Nikita Ivanov  wrote:

> -1
>
> - 'mvn clean package' fails on Win10 (at least)
> - binary artifacts in release location (ignite.exe)
>
> --
> Nikita Ivanov
>
>
>
> On Sun, Jan 3, 2021 at 4:00 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Dear Community,
> >
> > The release candidate is uploaded here:
> > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc1/
> > Maven staging:
> > https://repository.apache.org/content/repositories/orgapacheignite-1502/
> > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc1
> >
> > For more information on the purpose and the scope of the release, see
> this
> > discussion:
> >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html
> >
> > Included Jira tickets:
> >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1
> >
> > DEVNOTES:
> > https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc1/DEVNOTES.md
> >
> > The vote is formal, see voting guidelines:
> > https://www.apache.org/foundation/voting.html
> >
> > +1 - accept Apache Ignite 3.0.0-alpha1 RC1
> > 0 - don't care either way
> > -1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC1 (explain why)
> >
> > See notes on how to verify release here:
> > https://www.apache.org/info/verification.html
> >
> > This vote will be open for 72 hours and will close on January 7th at
> 12:00
> > am UTC:
> >
> >
> https://www.timeanddate.com/countdown/generic?iso=20210106T16=224=cursive
> >
> > -Val
> >
>

Re: [VOTE] Release Apache Ignite 3.0.0-alpha1 RC1

[Valentin Kulichenko]

I've identified an issue with tests on Windows:
https://issues.apache.org/jira/browse/IGNITE-13950. It is purely a test
issue related to the line separating characters. I don't think it should be
a blocker for this alpha release though - we can fix it going forward.

Nikita, is this the issue you're having, or there is something else?

-Val

On Mon, Jan 4, 2021 at 11:19 AM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Nikita,
>
> Can you show the error it fails with? Can you also try to exclude tests by
> running 'mvn clean package -DskipTests'?
>
> As for the binaries, we always include them. An Apache release must
> contain a source package, but it doesn't have to be limited to it. 'ignite'
> and 'ignite.exe' binaries will be downloadable from the website, and for
> that, they have to be released.
>
> -Val
>
> On Mon, Jan 4, 2021 at 10:52 AM Nikita Ivanov  wrote:
>
>> -1
>>
>> - 'mvn clean package' fails on Win10 (at least)
>> - binary artifacts in release location (ignite.exe)
>>
>> --
>> Nikita Ivanov
>>
>>
>>
>> On Sun, Jan 3, 2021 at 4:00 PM Valentin Kulichenko <
>> valentin.kuliche...@gmail.com> wrote:
>>
>> > Dear Community,
>> >
>> > The release candidate is uploaded here:
>> > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc1/
>> > Maven staging:
>> >
>> https://repository.apache.org/content/repositories/orgapacheignite-1502/
>> > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc1
>> >
>> > For more information on the purpose and the scope of the release, see
>> this
>> > discussion:
>> >
>> >
>> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html
>> >
>> > Included Jira tickets:
>> >
>> >
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1
>> >
>> > DEVNOTES:
>> > https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc1/DEVNOTES.md
>> >
>> > The vote is formal, see voting guidelines:
>> > https://www.apache.org/foundation/voting.html
>> >
>> > +1 - accept Apache Ignite 3.0.0-alpha1 RC1
>> > 0 - don't care either way
>> > -1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC1 (explain why)
>> >
>> > See notes on how to verify release here:
>> > https://www.apache.org/info/verification.html
>> >
>> > This vote will be open for 72 hours and will close on January 7th at
>> 12:00
>> > am UTC:
>> >
>> >
>> https://www.timeanddate.com/countdown/generic?iso=20210106T16=224=cursive
>> >
>> > -Val
>> >
>>
>

Re: [DISCUSSION] 3.0.0 Alpha release

[Valentin Kulichenko]

Igniters,

Big thanks to everyone involved in the preparation for this alpha release!

Throughout this week, the current codebase has been tested on Linux and
Windows, all the major issues have been resolved, and the Getting Started
guide has been created. We're now ready for the vote - I will start it
sometime during the weekend or early next week.

Happy New Year everyone!

-Val

On Thu, Dec 31, 2020 at 3:03 AM Fedor Malchikov 
wrote:

> Hi Valentin,
>
> Yes, of course.
> I closed tickets for fixed issues and created a few more new ones.
>
> On Thu, Dec 31, 2020 at 8:57 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Fedor,
> >
> > Thanks again - you're really helping to improve the quality of the
> release.
> >
> > I've fixed everything except for the progress bar issue - should not be a
> > blocker though, I will move it outside of the alpha if I do not succeed
> by
> > the end of tomorrow.
> >
> > Can you please verify that other issues are fixed (especially
> > IGNITE-13936)? Here is the new staging:
> > https://repository.apache.org/content/repositories/orgapacheignite-1500/
> >
> > -Val
> >
> > On Wed, Dec 30, 2020 at 5:31 AM Fedor Malchikov  >
> > wrote:
> >
> > > Hi Valentin,
> > > I looked at the resources available on staging and found a number of
> > > additional issues:
> > > One blocker:
> > > https://issues.apache.org/jira/browse/IGNITE-13936
> > > and three minor:
> > > https://issues.apache.org/jira/browse/IGNITE-13938
> > > https://issues.apache.org/jira/browse/IGNITE-13937
> > > https://issues.apache.org/jira/browse/IGNITE-13939
> > >
> > > On Mon, Dec 28, 2020 at 10:24 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > I've also noticed that there is functionality allowing to use custom
> > > Maven
> > > > repositories. However, it is available only for the 'module add'
> > command,
> > > > but not for the 'init' command. Created a ticket for this as well:
> > > > https://issues.apache.org/jira/browse/IGNITE-13923
> > > >
> > > > -Val
> > > >
> > > > On Mon, Dec 28, 2020 at 11:21 AM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Hi Fedor,
> > > > >
> > > > > Thanks for doing this!
> > > > >
> > > > > Regarding IGNITE-13921: in the first version, the tool created the
> > > > folders
> > > > > in the current directory, which created unpredictable behavior, so
> we
> > > > > switched to home as default. However, using the folder where the
> tool
> > > is
> > > > > located instead of the home folder seems fair - I've included the
> > > ticket
> > > > > into the alpha1. Also, note that in the next versions, there will
> be
> > an
> > > > > option to customize the paths, so the user will not have to rely on
> > > > default
> > > > > values.
> > > > >
> > > > > If you don't mind, I suggest to postpone other issues for further
> > > > > development - they do not look so critical.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Dec 28, 2020 at 9:45 AM Fedor Malchikov <
> > > fmalchi...@gridgain.com
> > > > >
> > > > > wrote:
> > > > >
> > > > >> Hello everyone,
> > > > >> I smoke tested the alpha version of the CLI tool on ubuntu and
> > > windows.
> > > > So
> > > > >> I found the following issues:
> > > > >> https://issues.apache.org/jira/browse/IGNITE-13914
> > > > >> https://issues.apache.org/jira/browse/IGNITE-13924
> > > > >> https://issues.apache.org/jira/browse/IGNITE-13921
> > > > >> https://issues.apache.org/jira/browse/IGNITE-13922
> > > > >> https://issues.apache.org/jira/browse/IGNITE-13920
> > > > >>
> > > > >>
> > > > >> On Sun, Dec 27, 2020 at 2:18 AM Valentin Kulichenko <
> > > > >> valentin.kuliche...@gmail.com> wrote:
> > > > >>
> > > > >> > Folks,
> > > > >> >
> > > > >> > I've created a version in Jira and a Wiki page for the alpha
> > > release:
> 

[VOTE] Release Apache Ignite 3.0.0-alpha1 RC2

[Valentin Kulichenko]

Dear Community,

The release candidate is uploaded here:
https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha1-rc2/
Maven staging:
https://repository.apache.org/content/repositories/orgapacheignite-1503/
Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha1-rc2

For more information on the purpose and the scope of the release, see this
discussion:
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSSION-3-0-0-Alpha-release-td50744.html

Included Jira tickets:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%203.0.0-alpha1

Fixes included on top of 3.0.0-alpha1 RC1:

   - https://issues.apache.org/jira/browse/IGNITE-13950
   (IgniteCliInterfaceTest can fail on Windows)

DEVNOTES:
https://github.com/apache/ignite-3/blob/3.0.0-alpha1-rc2/DEVNOTES.md


The vote is formal, see voting guidelines:
https://www.apache.org/foundation/voting.html

+1 - accept Apache Ignite 3.0.0-alpha1 RC2
0 - don't care either way
-1 - DO NOT accept Apache Ignite 3.0.0-alpha1 RC2 (explain why)

See notes on how to verify release here:
https://www.apache.org/info/verification.html

This vote will be open for 72 hours and will close on January 8th at 12:00
am UTC:
https://www.timeanddate.com/countdown/generic?iso=20210107T16=224=cursive

-Val

Re: [DISCUSSION] Fail on non-colocated join

[Valentin Kulichenko]

I think it should be the first option - just throw a clear exception if a
query with a non-colocated join is executed without the flag set.
Explicitly turning this mode on seems wrong, because in most cases this is
not what a user intends to do.

We also should make sure to run performance tests. As far as I know, this
was not implemented initially because the assumption was that performing
such a check on every single query might have a significant negative
impact, especially for OLTP patterns that require low latency and high
throughput.

-Val

On Tue, Feb 2, 2021 at 7:18 AM Max Timonin  wrote:

> Hi, Igniters!
>
> Last week I investigated a bug [1]. It's about an incorrect result for
> non-colocated joins. For such joins it's required to set up the
> "distributedJoin" flag, or try to make joined tables colocated. It is
> covered in docs [2]. But it's not obvious and some users don't read that or
> forget about that. In result there are wrong results for some queries that
> are pretty hard to debug.
>
> There is a ticket [3] with a comment, where it's suggested to add a check
> for such joins. I tried to implement it and found a place where it's
> possible to put this check. But there is an open question what this check
> should do. Currently I see 2 ways for that:
> 1. Forbid non-colocated joins that aren't marked with the distributedJoin
> flag, and throw an exception.
> 2. Check every query for such joins and implicitly setup a distributedJoin
> flag for them.
>
> Both solutions may break compatibility, but is this compatibility OK?
>
> Igniters, what do you think?
>
> [1] https://issues.apache.org/jira/browse/IGNITE-12847
> [2]
>
> https://ignite.apache.org/docs/latest/SQL/distributed-joins#distributed-joins
> [3] https://issues.apache.org/jira/browse/IGNITE-13019
>

Re: [DISCUSSION] Code style. Variable abbrevations

[Valentin Kulichenko]

Konstantin, thanks for chiming in.

That's exactly my concern. Overformalization is generally not a good thing.
Someone used "mess" to abbreviate "message"? That is surely not a good
coding style, but that's what code reviews are for. I believe that our
committers are more than capable of identifying such issues.

At the same time, with the current rules, we are *forced* to use
abbreviations like "locAddrGrpMgr", whether we like it or not. All it does
is makes it harder to contribute to Ignite, without providing any clear
value.

-Val

On Thu, Jun 10, 2021 at 9:46 AM Konstantin Orlov 
wrote:

> +1 for making this optional
>
> Common abbreviation rules is good for sure, but sometimes I getting sick
> of all those locAddrGrpMgr.
>
>
> --
> Regards,
> Konstantin Orlov
>
>
>
>
> > On 7 Jun 2021, at 14:31, Nikolay Izhikov  wrote:
> >
> > Hello, Anton, Alexei.
> >
> > Thanks for the feedback.
> >
> > Personally, I’m pretty happy current abbreviation rules too.
> > Let see what we can do to make our codebase even more consistent.
> >
> >> 7 июня 2021 г., в 13:23, Alexei Scherbakov <
> alexey.scherbak...@gmail.com> написал(а):
> >>
> >> -1
> >> Common abbrevs add quality to the code.
> >>
> >> пн, 7 июн. 2021 г. в 12:38, Anton Vinogradov :
> >>
> >>> -1 here.
> >>> We can fix the code and set up the rule.
> >>>
> >>> This will help to prevent having a weird abbreviation like "mess" (from
> >>> "message") or "ign" (from "Ignite").
> >>> Also, the abbreviations list (hardcoded at IDEA plugin) allows to have
> same
> >>> names across the whole code, this should simplify the reading.
> >>>
> >>> On Sat, Jun 5, 2021 at 10:49 PM Valentin Kulichenko <
> >>> valentin.kuliche...@gmail.com> wrote:
> >>>
> >>>> I also support removing this requirement. It’s not the first time
> someone
> >>>> brings this up, and so far we haven’t been able to fix it. Not worth
> it
> >>> in
> >>>> my view.
> >>>>
> >>>> -Val
> >>>>
> >>>> On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov 
> >>>> wrote:
> >>>>
> >>>>> Hello, guys.
> >>>>>
> >>>>> Thanks for the feedback.
> >>>>>
> >>>>> Dmitry,
> >>>>>
> >>>>>> Manual rule enforcement saves a couple of symbols in code
> >>>>>
> >>>>> I’m talking about automatic check.
> >>>>> We can implement it easily.
> >>>>> So, if we decide to keep this rule all we need is to fix current
> >>>>> violations (several thousand!).
> >>>>> After it, checkstyle will automatically enforce this rule.
> >>>>> As you may know, currently, any PR checked according to our
> checkstyle
> >>>>> rules.
> >>>>> Please, take a look at little green check sign after PR name.
> >>>>>
> >>>>>
> >>>>>> 5 июня 2021 г., в 00:57, Dmitry Pavlov 
> >>>> написал(а):
> >>>>>>
> >>>>>> +1 for removal. Cnt and count both seem to be fine.
> >>>>>>
> >>>>>> Manual rule enforcement saves a couple of symbols in code, but
> >>> requires
> >>>>> some time from both contributor and reviewer.
> >>>>>>
> >>>>>> Sincerely,
> >>>>>> Dmitriy Pavlov
> >>>>>>
> >>>>>> On 2021/06/04 19:18:37, Pavel Tupitsyn 
> wrote:
> >>>>>>> In my opinion, we should remove this rule.
> >>>>>>> Looks like a waste of time. freq or frequency, cnt or count, it is
> >>>> fine
> >>>>>>> either way.
> >>>>>>>
> >>>>>>> On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov <
> nizhi...@apache.org
> >>>>
> >>>>> wrote:
> >>>>>>>
> >>>>>>>> Hello, Igniters.
> >>>>>>>>
> >>>>>>>> Right now, we have the rule to use some predefined list of
> >>>> abbrevation
> >>>>> for
> >>>>>>>> variable names [1].
> >>>>>>>> Some of the revi

Re: [DISCUSSION] Code style. Variable abbrevations

[Valentin Kulichenko]

t;>>>>> about constant's namings.
> >>>>>>
> >>>>>> I suppose that we should define strict and clear rules about
> >>>>>> constants
> >>>>>> naming.
> >>>>>>
> >>>>>> [1] --
> >>>>>
> https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules
> >>>>>>
> >>>>>>
> >>>>>> чт, 17 июн. 2021 г. в 10:10, Konstantin Orlov  >>> :
> >>>>>>
> >>>>>>> Enforced validation doesn't guarantee code consistency. No
> >>>>>>> validation
> >>>> in
> >>>>>>> the world prevent me from typing manually "mess" instead of
> >>>>> "msg"/"message"
> >>>>>>> or "svc" instead of "srvc"/"service" (btw "svc" is more common
> >> imo).
> >>>>>>>
> >>>>>>> And the fact that someone name a variable "count" instead of "cnt"
> >>>>> doesn't
> >>>>>>> make the whole project immature.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Regards,
> >>>>>>> Konstantin Orlov
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> On 17 Jun 2021, at 08:34, Ivan Pavlukhin 
> >>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi Nikolay,
> >>>>>>>>
> >>>>>>>> Thanks, it is rather interesting.
> >>>>>>>>
> >>>>>>>> Do we all agree that using different conventions for different
> >> code
> >>>>>>>> packages does not break "consistency"? Or did I get something
> >>>>>>>> wrong?
> >>>>>>>>
> >>>>>>>> 2021-06-17 7:12 GMT+03:00, Николай Ижиков :
> >>>>>>>>> Hello, Ivan.
> >>>>>>>>>
> >>>>>>>>> We can create checkstyle rule to enforce usage of abbreviations.
> >>>>>>>>> Internal/public code differs by package.
> >>>>>>>>>
> >>>>>>>>> PoC of rule [1]
> >>>>>>>>>
> >>>>>>>>> [1] https://github.com/apache/ignite/pull/9153
> >>>>>>>>>
> >>>>>>>>>> 17 июня 2021 г., в 07:01, Ivan Pavlukhin 
> >>>>>>>>>> написал(а):
> >>>>>>>>>>
> >>>>>>>>>> Nikita,
> >>>>>>>>>>
> >>>>>>>>>> Do you have a plan in your mind how to make Ignite codebase
> >>>>> consistent?
> >>>>>>>>>>
> >>>>>>>>>> AFAIR, we had it intentionally inconsistent for a long time at
> >>>> least
> >>>>>>>>>> for one sake: for internal code we used special conventions
> >> (e.g.
> >>>>>>>>>> abbreviations, shorten getters) and common Java conventions for
> >>>>> public
> >>>>>>>>>> API and examples (e.g. no abbreviations and usual getters).
> >>>>>>>>>>
> >>>>>>>>>> 2021-06-16 23:37 GMT+03:00, Nikita Ivanov  >>> :
> >>>>>>>>>>> Consistency is what makes it easier to contribute to the
> >> project
> >>>> and
> >>>>>>>>>>> attract new members. Consistency implies strong, well defined
> >>>>>>>>>>> and
> >>>>>>>>>>> universally enforced rules. Just because we have some
> >>>>>>>>>>> individuals
> >>>>> who
> >>>>>>>>>>> are lazy or inexperienced - it does not mean that the entire
> >>>> project
> >>>>>>>>>>> should relax the basic-level engineering discipline.
> >>>>>>>>>>>
> >>>>>>>>>>> On a personal node - nothing screams "immaturity"

Re: IGNITE-14812: Statistics

[Valentin Kulichenko]

Dmitry,

As the PMC chair, would you mind contacting legal regarding the matter?
This is not the only example of such code (e.g. [1]), so we should look
into this asap.

[1]
https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java

As for this particular commit, can HLL be added as a dependency instead? Is
there any particular reason to include the source code? @Sasha Belyak
 , can you please chime in?

-Val

On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov  wrote:

> Hi Nikolai,
>
> thank you for noticing. I guess it's not about license, but about
> Intellectual Property (IP) ownership.
>
> AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed to be
> used in the codebase for an Apache project (
> https://www.apache.org/legal/resolved.html)
>
> But licenese and IP owner are 2 sligthly different things. E.g at the end
> of any website you can find:
> Copyright © 2021 The Apache Software Foundation, Licensed under the Apache
> License, Version 2.0.
>
> Incubated projects are mandated to transfer IP to the ASF. And I'm not
> aware of any exceptions.
>
> In this PR there are 5 classes which licenses with AL 2.0, but IP owner is
> 3rd party company.
>
> I'm a bit concerned about having such code in the project. I'd rather
> reverted it until we have approval from experts at mailing list:
> legal-disc...@apache.org
>
> Sincerely,
> Dmitriy Pavlov
>
> On 2021/06/22 14:56:49, Nikolay Izhikov  wrote:
> > Hello, Igniters.
> >
> > Recently huge commit was merged [1].
> >
> > Taras, Alexander, can you, please, explain what is purpose of the commit?
> > What feature it implemented?
> >
> > Looked inside the ticket and found no explanation.
> > Description is "Add statistics collection and usage.»
> >
> > Do we have plans to document this new feature?
> >
> > Also, I see very strange license in added files [2]
> >
> > ```
> >  * Copyright 2013 Aggregate Knowledge, Inc.
> >  *
> >  * Licensed under the Apache License, Version 2.0 (the "License");
> > ```
> >
> > Is it OK to have those copyright inside ASF codebase?
> > Is it some kind of external dependency we adopted as part of the code?
> > Why do we need it?
> >
> > [1]
> https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb
> > [2]
> https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java
>

Re: IGNITE-14812: Statistics

[Valentin Kulichenko]

Taras,

It is true that we try to minimize dependencies, but there are some anyway.
I think it's perfectly fine to add this library as a dependency.

Will you be able to do this asap? It is surely better than reverting the
commit :)

-Val

On Wed, Jun 23, 2021 at 12:32 AM Taras Ledkov  wrote:

> Hi,
>
>  > Taras, can you, please, describe the features that was implemented in
> this merge?
>  > How users supposed to use them?
>  > Do we have plans to document?
>
> Sure. Alexander Belyak will describe and file ticket to documentation.
>
> On 23.06.2021 9:27, Nikolay Izhikov wrote:
> > Hello, Taras.
> >
> > Thanks for feedback.
> >
> >> AFAIK and as long as I can remember Ignite project try to minimize
> external dependencies usage and adds new external dependency only when
> there is no other way out.
> > Does it mean we have to incapsulate every external library we want to
> use?
> >
> > Taras, can you, please, describe the features that was implemented in
> this merge?
> > How users supposed to use them?
> > Do we have plans to document?
> >
> >
> >> 23 июня 2021 г., в 09:21, Taras Ledkov 
> написал(а):
> >>
> >> Hi,
> >>
> >> We have discussed BCrypt include/add dependency here [1].
> >> AFAIK and as long as I can remember Ignite project try to minimize
> external dependencies usage
> >> and adds new external dependency only when there is no other way out.
> >>
> >> [1].
> http://apache-ignite-developers.2346864.n4.nabble.com/Username-password-authentication-for-thin-clients-tp26058p26954.html
> >>
> >> On 23.06.2021 3:08, Valentin Kulichenko wrote:
> >>> Dmitry,
> >>>
> >>> As the PMC chair, would you mind contacting legal regarding the matter?
> >>> This is not the only example of such code (e.g. [1]), so we should look
> >>> into this asap.
> >>>
> >>> [1]
> >>>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/mindrot/BCrypt.java
> >>>
> >>> As for this particular commit, can HLL be added as a dependency
> instead? Is
> >>> there any particular reason to include the source code? @Sasha Belyak
> >>>  , can you please chime in?
> >>>
> >>> -Val
> >>>
> >>> On Tue, Jun 22, 2021 at 8:10 AM Dmitry Pavlov 
> wrote:
> >>>
> >>>> Hi Nikolai,
> >>>>
> >>>> thank you for noticing. I guess it's not about license, but about
> >>>> Intellectual Property (IP) ownership.
> >>>>
> >>>> AFAIK, Apache License 2.0 is here and AL 2.0 is definetely allowed to
> be
> >>>> used in the codebase for an Apache project (
> >>>> https://www.apache.org/legal/resolved.html)
> >>>>
> >>>> But licenese and IP owner are 2 sligthly different things. E.g at the
> end
> >>>> of any website you can find:
> >>>> Copyright © 2021 The Apache Software Foundation, Licensed under the
> Apache
> >>>> License, Version 2.0.
> >>>>
> >>>> Incubated projects are mandated to transfer IP to the ASF. And I'm not
> >>>> aware of any exceptions.
> >>>>
> >>>> In this PR there are 5 classes which licenses with AL 2.0, but IP
> owner is
> >>>> 3rd party company.
> >>>>
> >>>> I'm a bit concerned about having such code in the project. I'd rather
> >>>> reverted it until we have approval from experts at mailing list:
> >>>> legal-disc...@apache.org
> >>>>
> >>>> Sincerely,
> >>>> Dmitriy Pavlov
> >>>>
> >>>> On 2021/06/22 14:56:49, Nikolay Izhikov  wrote:
> >>>>> Hello, Igniters.
> >>>>>
> >>>>> Recently huge commit was merged [1].
> >>>>>
> >>>>> Taras, Alexander, can you, please, explain what is purpose of the
> commit?
> >>>>> What feature it implemented?
> >>>>>
> >>>>> Looked inside the ticket and found no explanation.
> >>>>> Description is "Add statistics collection and usage.»
> >>>>>
> >>>>> Do we have plans to document this new feature?
> >>>>>
> >>>>> Also, I see very strange license in added files [2]
> >>>>>
> >>>>> ```
> >>>>>   * Copyright 2013 Aggregate Knowledge, Inc.
> >>>>>   *
> >>>>>   * Licensed under the Apache License, Version 2.0 (the "License");
> >>>>> ```
> >>>>>
> >>>>> Is it OK to have those copyright inside ASF codebase?
> >>>>> Is it some kind of external dependency we adopted as part of the
> code?
> >>>>> Why do we need it?
> >>>>>
> >>>>> [1]
> >>>>
> https://github.com/apache/ignite/commit/503a98495433e1d0cf84f8be8c1e2adc57034fbb
> >>>>> [2]
> >>>>
> https://github.com/apache/ignite/blob/master/modules/indexing/src/main/java/org/apache/ignite/internal/processors/query/stat/hll/serialization/IHLLMetadata.java
> >>>>
> >> --
> >> Taras Ledkov
> >> Mail-To: tled...@gridgain.com
> >>
> --
> Taras Ledkov
> Mail-To: tled...@gridgain.com
>
>

Re: ASF report May 2021, due 12 May, eod

[Valentin Kulichenko]

Hi Dmitry,

I've restored the section. Thanks for pointing it out.

-Val

On Fri, May 7, 2021 at 6:35 AM Dmitry Pavlov  wrote:

> Igniters,
>
> It seems that extensions sectoin was removed by Val's commit
>
>
> https://github.com/apache/ignite-website/commit/70edeaa1c24d9d60ba173bbd03bedb6a29f4cc22#diff-23dd22649c2608cb0d2665a931eec1b010bfa3b559afb398978d816790af8c25L1065
>
> Was that by mistake? or done intentionally?
>
> Sincerely,
> Dmitriy Pavlov
>
> On 2021/05/06 16:04:10, Dmitry Pavlov  wrote:
> > Hi Igniters,
> >
> > Its time for our TLP to prepare next report to the Board.
> >
> > Drafting report in reported on in a mailing thread is not so
> convinitent, so I've started to prepare it here:
> >
> https://cwiki.apache.org/confluence/display/IGNITE/ASF+Board+Report+Drafts
> >
> > It contains now very first version, you're welcome to suggest here in
> the thread any of
> > - significiant community decisions
> > - released artifacts
> > - any events/facts worth mentioning.
> >
> > There are 2 issues that stops from publish the report rigth now,
> >
> > 1) for some reason our user@iao list statistics is not collected since
> March (since 10th week of 2021). Most likely Infra and/or com.devs could
> help here.
> > 2) our downloads page does not show downloads for extensions:
> > https://ignite.apache.org/download.cgi#extensions is not visible for
> me. Is it my local issue?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
>

Re: Proposal to remove explicit "GC disable" startup suggestion

[Valentin Kulichenko]

Got it. The proposal makes sense to me, let's remove the suggestion.

-Val

On Sun, May 16, 2021 at 5:50 AM Ilya Korol  wrote:

> In some circumstances there could be a state when direct buffers
> (instantiated for IO purposes) hold too many memory and to get this
> memory back JVM calls System.gc() (as was mentioned earlier by Ivan
> Daschinsky):
>
> ByteBuffer.allocateDirect(capacity)
>  new DirectButeBuffer(capacity)
>  Bits.reserveMemory(size, cap)
> ...
>  System.gc() // if optimistic attempts to reserve required
> memory failed
>
> See also:
> [1]
>
> http://www.mastertheboss.com/other/java-stuff/troubleshooting-outofmemoryerror-direct-buffer-memory
>
> See
>
> 16.05.2021 01:21, Valentin Kulichenko пишет:
> > Hi Ilya,
> >
> > Out of curiosity - what are the potential problems with memory buffers
> that
> > you're referring to?
> >
> > -Val
> >
> > On Fri, May 14, 2021 at 5:06 AM Ilya Korol  wrote:
> >
> >> Hi, everyone.
> >>
> >> There is a proposal to remove suggestion that user should disable
> >> explicit GC calls in https://issues.apache.org/jira/browse/IGNITE-14720
> .
> >>
> >> Nowadays people usually don't use this facility directly without huge
> >> need (at least we hope so), so this suggestion doesn't bring much
> >> profit, but instead could lead to potential problems with reclaiming
> >> memory used by direct memory buffers.
> >>
> >> I'm going to fix this.
> >>
> >> Is there any objections (suggestions)?
> >>
> >>
>

Re: Proposal to remove explicit "GC disable" startup suggestion

[Valentin Kulichenko]

Hi Ilya,

Out of curiosity - what are the potential problems with memory buffers that
you're referring to?

-Val

On Fri, May 14, 2021 at 5:06 AM Ilya Korol  wrote:

> Hi, everyone.
>
> There is a proposal to remove suggestion that user should disable
> explicit GC calls in https://issues.apache.org/jira/browse/IGNITE-14720.
>
> Nowadays people usually don't use this facility directly without huge
> need (at least we hope so), so this suggestion doesn't bring much
> profit, but instead could lead to potential problems with reclaiming
> memory used by direct memory buffers.
>
> I'm going to fix this.
>
> Is there any objections (suggestions)?
>
>

Re: Extensions section on the downloads page

[Valentin Kulichenko]

Agree. I brought the section back because there was a broken link, and I
believe I removed it earlier by mistake. We definitely should add more
details though.

-Val

On Wed, May 12, 2021 at 9:20 AM Dmitry Pavlov  wrote:

> Hi Denis,
>
> +1 from my side.
>
> Now extensions in the download page looks like a bit incomplete snippet of
> code.
>
> It is better than nothing, but completed table would look better.
>
> Sincerely,
> Dmitriy Pavlov
>
> On 2021/05/12 14:02:04, Denis Magda  wrote:
> > Igniters, Val,
> >
> > I came across the Extensions section that was restored on the downloads
> > page. It looks clumsy, broken, and incomplete. We're listing spring boot
> > artifacts without any reference to other modules:
> > https://ignite.apache.org/download.cgi#extensions
> >
> > How about:
> >
> >- Removing the code snippets of the Spring Boot extensions
> >- Add a paragraph and a list with references to all the extensions we
> >have documented on our pages. See extensions and integrations section:
> >https://ignite.apache.org/docs/latest/
> >
> > @Igor Gusev , @Nikita Safonov
> >  could
> > you also join this thread?
> >
> > -
> > Denis
> >
>

Re: [DISCUSSION] Code style. Variable abbrevations

[Valentin Kulichenko]

I also support removing this requirement. It’s not the first time someone
brings this up, and so far we haven’t been able to fix it. Not worth it in
my view.

-Val

On Sat, Jun 5, 2021 at 11:54 AM Nikolay Izhikov  wrote:

> Hello, guys.
>
> Thanks for the feedback.
>
> Dmitry,
>
> > Manual rule enforcement saves a couple of symbols in code
>
> I’m talking about automatic check.
> We can implement it easily.
> So, if we decide to keep this rule all we need is to fix current
> violations (several thousand!).
> After it, checkstyle will automatically enforce this rule.
> As you may know, currently, any PR checked according to our checkstyle
> rules.
> Please, take a look at little green check sign after PR name.
>
>
> > 5 июня 2021 г., в 00:57, Dmitry Pavlov  написал(а):
> >
> > +1 for removal. Cnt and count both seem to be fine.
> >
> > Manual rule enforcement saves a couple of symbols in code, but requires
> some time from both contributor and reviewer.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > On 2021/06/04 19:18:37, Pavel Tupitsyn  wrote:
> >> In my opinion, we should remove this rule.
> >> Looks like a waste of time. freq or frequency, cnt or count, it is fine
> >> either way.
> >>
> >> On Fri, Jun 4, 2021 at 7:39 PM Nikolay Izhikov 
> wrote:
> >>
> >>> Hello, Igniters.
> >>>
> >>> Right now, we have the rule to use some predefined list of abbrevation
> for
> >>> variable names [1].
> >>> Some of the reviewers ask to follow this rule strictly.
> >>>
>  It is required to use abbreviated form for code consistency.
> >>>
> >>> I tried to implement this rule in form of checkstyle check [2] and it
> >>> seems like many of use doesn’t follow this requirement.
> >>> My check found 4124 violation in core module.
> >>>
> >>> Should we make this rule optional in documentation or should we remove
> it
> >>> completely?
> >>> Or should someone proceed and fix all the violations?
> >>>
> >>> WDYT?
> >>>
> >>>
> >>> Example of output:
> >>> ```
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:94:
> >>> Abbrevation should be used for CACHE_NAME_LOCAL_ATOMIC! Please, use
> loc,
> >>> instead of LOCAL [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:97:
> >>> Abbrevation should be used for CACHE_NAME_LOCAL_TX! Please, use loc,
> >>> instead of LOCAL [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWithTtlTest.java:264:
> >>> Abbrevation should be used for checkpointManager! Please, use mgr,
> instead
> >>> of Manager [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsPartitionPreloadTest.java:63:
> >>> Abbrevation should be used for DEFAULT_REGION! Please, use dflt,
> instead of
> >>> DEFAULT [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsWholeClusterRestartTest.java:47:
> >>> Abbrevation should be used for ENTRIES_COUNT! Please, use cnt, instead
> of
> >>> COUNT [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsRebalancingOnNotStableTopologyTest.java:49:
> >>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
> >>> instead of FREQUENCY [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:75:
> >>> Abbrevation should be used for MAX_KEY_COUNT! Please, use cnt, instead
> of
> >>> COUNT [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgnitePdsTransactionsHangTest.java:78:
> >>> Abbrevation should be used for CHECKPOINT_FREQUENCY! Please, use freq,
> >>> instead of FREQUENCY [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/SlowHistoricalRebalanceSmallHistoryTest.java:57:
> >>> Abbrevation should be used for SUPPLY_MESSAGE_LATCH! Please, use msg,
> >>> instead of MESSAGE [IgniteAbbrevationsRule]
> >>> [ERROR]
> >>>
> /Users/sbt-izhikov-nv/work/ignite/modules/core/src/test/java/org/apache/ignite/internal/processors/cache/persistence/db/IgniteLogicalRecoveryTest.java:74:
> >>> Abbrevation should be used for SHARED_GROUP_NAME! Please, use grp,
> instead
> >>> of 

Re: Node and cluster life-cycle in ignite-3

[Valentin Kulichenko]

Hi Sergey,

Sounds interesting, I do agree that it might be beneficial to improve the
lifecycle management in 3.0 - 2.x version is far from perfect.

Regarding your questions:

1. Can this be done via the metastore?
2. I think we should list the run levels that we think should be there, and
then it will be easier to identify dependencies between them. Can you give
an example of independent run levels?

-Val

On Tue, Jun 1, 2021 at 7:57 AM Sergey Chugunov 
wrote:

>  Hello Igniters,
>
> I would like to start a discussion on evolving IEP-73 [1]. Now it covers a
> narrow topic about components dependencies but it makes sense to cover in
> the IEP a broader question: how different components should be initialized
> to support different modes of an individual node or a whole cluster.
>
> There is an idea to borrow the notion of run-levels from Unix-like systems,
> and I suggest the following design to implement it.
>
>1. To start and function at a specific run-level node needs to start and
>initialize components in a proper order. During initialization
> components
>may need to notify each other about reaching a particular run-level so
>other components are able to execute their actions. Orchestrating of
> this
>process should be a responsibility of a new component.
>
>2. Orchestration component doesn't manage the initialization process
>directly but uses another abstraction called scenario. Examples of
>run-levels in the context of Ignite 2.x may be Maintenance Mode,
>INACTIVE-READONLY-ACTIVE states of a cluster, and each level is reached
>when a corresponding scenario has executed.
>
>So the responsibility of the orchestrator will be managing scenarios and
>providing them with infrastructure of spreading notification events
> between
>components. All low-level details and knowledge of existing components
> and
>their dependencies are encapsulated inside scenarios.
>
>3. Scenarios allow nesting, e.g. a scenario for INACTIVE cluster state
>can be "upgraded" to READONLY state by executing diff between INACTIVE
> and
>READONLY scenarios.
>
>
> I see several advantages of this design compared to existing model in
> Ignite 2.x (mostly implemented in IgniteKernal and based on two main
> methods: start and onKernalStart):
>
>1. More flexible model allows implementing more diverse run-levels for
>different needs (already mentioned Maintenance Mode, cluster state modes
>like ACTIVE-INACTIVE and smart strategies for cache warmup on node
> start).
>
>2. Knowledge of components and their dependencies is encapsulated inside
>scenarios which makes it easier to create new scenarios.
>
>
> Open questions:
>
>1. As I see right now it is hard to standardize initialization events
>components notify each other with.
>
>2. It is not clear if run-levels should be organized into one rigid
>hierarchy (when the first run-level should always precede the second
> and so
>on) or they should be more independent.
>
>
> What do you think?
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-73%3A+Node+startup
>

Re: Default Lucene index for String cache values

[Valentin Kulichenko]

Folks,

This is a usability issue of the current API, which I don't think we can
easily fix right now without breaking compatibility.

There are two separate cases.

First - the value class is a user's POJO. This one is straightforward - we
analyze annotations and create indexes based on that. The user has full
control.

Second - the value class is a "first-class citizen", i.e. primitive,
String, UUID, or similar. In this case, there are no annotations
(obviously), so we have to implicitly create an index for the value itself.
String type is a special case, as a user might want a full-text index for
it as well. So we *additionally* create one.

We can't just remove the aforementioned code, because there is no explicit
way to create a full-text index for the String value type. I guess such a
way can be provided by introducing additional configuration property (or
properties), but frankly, it will only complicate things even more. The use
case is not that popular, and the overhead of a single index is not that
big, so I don't think it worth it.

I would not change anything here.

-Val

On Thu, Jun 3, 2021 at 1:38 AM Maksim Timonin 
wrote:

> Hi, Ilya.
>
> Could you please provide a reproducer of wrong behavior? Looks like I'm
> missing smth in your idea. There is my code [1] with String class as value,
> and TextQuery works correctly without the annotation QueryTextField. And I
> expect this behavior.
>
> > case of plane strings as cache values (instead of pojo class from my
> example) lucene index would be created no matter if we ask for it or not
> It's not correct. If you create cache CacheConfiguration
> it will not create LuceneIndex, as there is no table with String as value
> class. You can check it with my code just replace Long and String
> classes in cache definition.
>
> [1] https://gist.github.com/timoninmaxim/e477ddfcbe56ec9892c7ba6ad44bfadb
>
> On Thu, Jun 3, 2021 at 10:56 AM Ilya Korol  wrote:
>
> > Thanks for feedback Maksim, but let me disagree with you.
> >
> > As far as i understand
> >
> > CacheConfiguration.setIndexedTypes(Long.class, String.class);
> >
> > Is just the hint for Ignite about data that should be indexed, but what
> > kind of index should be created depends on our configuration. For
> example:
> >
> > public class StringValue implements Serializable {
> >
> >  @QuerySqlField(index = true)
> >  private final long id;
> >
> >  @QueryTextField
> >  private final String name;
> >
> > }
> >
> > In this case i explicitly define that i need text/lucene index for name
> > field via marking it with @QueryTextField annotation. Doing so i'm able
> > to perform text queries:
> >
> > cache.query(new TextQuery(StringValue.class, "value"))
> >  .getAll()
> >  .forEach(e -> System.out.println(e.toString()));
> >
> > Without @QueryTextField annotation query will return nothing.
> >
> > So setIndexedTypes(..) is just not enough for proper lucene index
> > initiation. But in case of plane strings as cache values (instead pf
> > pojo class from my example) lucene index would be created no matter if
> > we ask for it or not:
> >
> > // IgniteH2Indexing#queryLocalText(..)
> >
> > if (tbl != null && tbl.luceneIndex() != null) {
> >  Long qryId = runningQueryManager().register(qry, TEXT, schemaName,
> > true, null);
> >
> >  try {
> >  // We will reach this line if we use String as cache values or
> > if we used @QueryTextField annotation
> >  return tbl.luceneIndex().query(qry.toUpperCase(), filters,
> limit);
> >
> >  } finally {
> >  runningQueryManager().unregister(qryId, null);
> >  }
> > }
> >
> >
> >
> > 02.06.2021 23:40, Maksim Timonin пишет:
> > > Hi, Ilya! AFAIK, to create LuceneIndex it's required to do this:
> > >
> > > CacheConfiguration.setIndexedTypes(Long.class, String.class);
> > >
> > > It's pretty straightforward, a user wants the value class to be
> indexed.
> > If
> > > you just create a simple cache (without entities, indexed types) with
> > > String.class as value it won't be indexed, as indexes created per
> table,
> > > not per cache.
> > >
> > > Do I miss something?
> > >
> > > On Wed, Jun 2, 2021 at 12:56 PM Ilya Korol 
> wrote:
> > >
> > >> Hi, All.
> > >>
> > >> According to https://issues.apache.org/jira/browse/IGNITE-14805 there
> > is
> > >> default index creation for caches with String values:
> > >>
> > >>
> > >> if (type().valueClass() == String.class) {
> > >>   try {
> > >>   luceneIdx = new GridLuceneIndex(idx.kernalContext(),
> > >> tbl.cacheName(), type);
> > >>   }
> > >>   catch (IgniteCheckedException e1) {
> > >>   throw new IgniteException(e1);
> > >>   }
> > >> }
> > >>
> > >>
> > >> Does this really necessary?  What about disabling this feature by
> > >> default and enabling it only by demand (to reduce unnecessary
> > >> performance hit even if its not very huge)? I guess additional option
> > >> could be introduced to do so.
> > >>
> > >> 

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Hi Pavel,

I don't think we will need the pure embedded mode, but we still need to be
able to access the API from compute and services. That said, there are two
usages of the 'Ignite' API:

   1. Remote, via the binary protocol.
   2. Local - needed for compute and services. (This is how it works now.)

I believe that the API should be the same, and there should be a unified
access point. Ignition seems to be a good candidate for this.

Ignition#start should eventually be removed from the public API. It is
currently there only because we don't have the thin client yet.

-Val

On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn  wrote:

> Igniters,
>
> I have a few questions regarding server node startup and thin clients.
>
> State of things:
> - Server nodes will be started with 'ignite run' from CLI [1]
> - ignite-api module represents our public API
> - ignite-api has Ignition interface to start server nodes
>
> Questions:
> - What's the idea behind Ignition interface in the public API? Are we going
> to have an "embedded mode" where servers can be started from code? I
> thought this was not planned.
> - How are users supposed to retrieve an instance of the Ignition interface?
> - Are there any plans to start thin clients from Ignition interface, or
> should we have a separate way of doing this?
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
>

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Hi Ivan,

Ignition IS the entry point to Ignite, so I'm not sure I got your point :)
Where is the contradiction?

Either way, please feel free to give your suggestions for an alternative
name if you have any.

-Val

On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina  wrote:

> A side note. Actually “Ignition” naming always confused me. I think about
> it as some fancy named API entry point for Ignite. Perhaps it is a good
> moment to revisit naming.
>
> > On 8 Jul 2021, at 07:09, Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
> >
> > Hi Pavel,
> >
> > I don't think we will need the pure embedded mode, but we still need to
> be
> > able to access the API from compute and services. That said, there are
> two
> > usages of the 'Ignite' API:
> >
> >   1. Remote, via the binary protocol.
> >   2. Local - needed for compute and services. (This is how it works now.)
> >
> > I believe that the API should be the same, and there should be a unified
> > access point. Ignition seems to be a good candidate for this.
> >
> > Ignition#start should eventually be removed from the public API. It is
> > currently there only because we don't have the thin client yet.
> >
> > -Val
> >
> >> On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn 
> wrote:
> >>
> >> Igniters,
> >>
> >> I have a few questions regarding server node startup and thin clients.
> >>
> >> State of things:
> >> - Server nodes will be started with 'ignite run' from CLI [1]
> >> - ignite-api module represents our public API
> >> - ignite-api has Ignition interface to start server nodes
> >>
> >> Questions:
> >> - What's the idea behind Ignition interface in the public API? Are we
> going
> >> to have an "embedded mode" where servers can be started from code? I
> >> thought this was not planned.
> >> - How are users supposed to retrieve an instance of the Ignition
> interface?
> >> - Are there any plans to start thin clients from Ignition interface, or
> >> should we have a separate way of doing this?
> >>
> >>
> >> [1]
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> >>
>

Re: [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1

[Valentin Kulichenko]

The vote is closed with 4 "+1" votes:
https://lists.apache.org/thread.html/r65e139ad16bbc074b09d2e0a28ca24f2314a25ce8b5061c0f5fb02d3%40%3Cdev.ignite.apache.org%3E

-Val

On Mon, Jun 28, 2021 at 12:37 PM Pavel Tupitsyn 
wrote:

> +1 (binding)
>
> On Mon, Jun 28, 2021 at 8:23 PM Вячеслав Коптилин <
> slava.kopti...@gmail.com>
> wrote:
>
> > +1
> >
> > Thanks,
> > S.
> >
> > пн, 28 июн. 2021 г. в 18:09, Igor Sapego :
> >
> > > +1
> > >
> > > Best Regards,
> > > Igor
> > >
> > >
> > > On Sat, Jun 26, 2021 at 1:41 AM Nikita Ivanov 
> > wrote:
> > >
> > > > +1
> > > >
> > > > --
> > > > Nikita Ivanov
> > > >
> > > >
> > > >
> > > > On Fri, Jun 25, 2021 at 3:31 PM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Dear Community,
> > > > >
> > > > > In the last several months, the development of Ignite 3 has been
> > moving
> > > > > forward significantly. On top of what we already had in the first
> > > Alpha,
> > > > we
> > > > > have the following features ready:
> > > > >
> > > > > - Replication infrastructure based on Raft
> > > > > - In-memory atomic storage with the basic insert-read functionality
> > > > > - New schema management engine and API
> > > > >
> > > > > In my view, this constitutes a significant milestone, and I propose
> > to
> > > > > release it as the Alpha 2. This way it will be available for
> > download,
> > > > and
> > > > > anyone will be able to play with the project, run examples, get a
> > feel
> > > of
> > > > > how Ignite will work in the future, and provide feedback.
> > > > >
> > > > > Please vote.
> > > > >
> > > > > ---
> > > > >
> > > > > The release candidate is uploaded here:
> > > > > https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/
> > > > > Maven staging:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheignite-1522/
> > > > > Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1
> > > > >
> > > > > Jira tickets:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC
> > > > >
> > > > > DEVNOTES:
> > > > >
> https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md
> > > > >
> > > > > The vote is formal, see voting guidelines:
> > > > > https://www.apache.org/foundation/voting.html
> > > > >
> > > > > +1 - accept Apache Ignite 3.0.0-alpha2 RC1
> > > > > 0 - don't care either way
> > > > > -1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why)
> > > > >
> > > > > See notes on how to verify release here:
> > > > > https://www.apache.org/info/verification.html
> > > > >
> > > > > This vote will be open for 72 hours and will close on June 28th at
> > 11pm
> > > > > UTC:
> > > > >
> > > > >
> > > >
> > >
> >
> https://www.timeanddate.com/countdown/to?iso=20210628T16=224=Apache+Ignite+3.0.0-alpha2+RC1=cursive=1
> > > > >
> > > > > -Val
> > > > >
> > > >
> > >
> >
>

[RESULT] [VOTE] Release Apache Ignite 3.0.0-alpha2 RC1

[Valentin Kulichenko]

Igniters,

Apache Ignite 3.0.0-alpha2 RC1 has been accepted.

4 "+1" votes received:
   - Nikita Ivanov (binding)
   - Igor Sapego (binding)
   - Vyacheslav Koptilin (non-binding)
   - Pavel Tupitsyn (binding)

No "0" or "-1" votes.

Vote thread:
https://lists.apache.org/thread.html/rc7156cdc8b7097d4b081750cf4cd1892a0791cde520815f081520ec9%40%3Cdev.ignite.apache.org%3E

-Val

[VOTE] Release Apache Ignite 3.0.0-alpha2 RC1

[Valentin Kulichenko]

Dear Community,

In the last several months, the development of Ignite 3 has been moving
forward significantly. On top of what we already had in the first Alpha, we
have the following features ready:

- Replication infrastructure based on Raft
- In-memory atomic storage with the basic insert-read functionality
- New schema management engine and API

In my view, this constitutes a significant milestone, and I propose to
release it as the Alpha 2. This way it will be available for download, and
anyone will be able to play with the project, run examples, get a feel of
how Ignite will work in the future, and provide feedback.

Please vote.

---

The release candidate is uploaded here:
https://dist.apache.org/repos/dist/dev/ignite/3.0.0-alpha2-rc1/
Maven staging:
https://repository.apache.org/content/repositories/orgapacheignite-1522/
Git tag: https://github.com/apache/ignite-3/tree/3.0.0-alpha2-rc1

Jira tickets:
https://issues.apache.org/jira/issues/?jql=project%20%3D%2012315922%20AND%20fixVersion%20%3D%2012349574%20ORDER%20BY%20priority%20DESC%2C%20key%20ASC

DEVNOTES:
https://github.com/apache/ignite-3/blob/3.0.0-alpha2-rc1/DEVNOTES.md

The vote is formal, see voting guidelines:
https://www.apache.org/foundation/voting.html

+1 - accept Apache Ignite 3.0.0-alpha2 RC1
0 - don't care either way
-1 - DO NOT accept Apache Ignite 3.0.0-alpha2 RC1 (explain why)

See notes on how to verify release here:
https://www.apache.org/info/verification.html

This vote will be open for 72 hours and will close on June 28th at 11pm
UTC:
https://www.timeanddate.com/countdown/to?iso=20210628T16=224=Apache+Ignite+3.0.0-alpha2+RC1=cursive=1

-Val

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Valentin Kulichenko]

Hi Pavel,

Please see my comments below.

-Val

On Tue, Jun 29, 2021 at 2:23 PM Pavel Tupitsyn  wrote:

> Igniters,
>
> While working on "IEP-76 Thin Client Protocol for Ignite 3.0" [1] (to be
> discussed separately), the following suggestions for the Table API came up:
>
> 1. Expose table IDs: sending table name with every operation (e.g. GET) is
> inefficient, string serialization is expensive by itself and names can be
> long.
> - Table.id()
> - IgniteTables.table(UUID)
> - IgniteTables.dropTable(UUID)
>

I don't think this should be a part of the public API. Users will identify
tables using names, they will never use IDs. As an internal optimization
though - sure, we can have that. Sounds similar to the cache ID in 2.x.


>
> 2. Expose tuple schemas: to reduce payload size when sending tuples to the
> client, we'll write only the schema version and column values, then the
> client can retrieve and cache schemas (ordered set of columns per version).
> - Tuple.schema()
> - Table.schemas()
> - Table.schema(ver)
>

Exposing the schema of a tuple makes sense. However, whether a user might
need to get a table schema for a particular version, I'm not sure. Do you
have a use case in mind for this? If not, I would keep this internal as
well.


>
> 3. Expose tuple values as a collection: to serialize tuples efficiently
> (see p2) we need an API to get all values at once. Right now the only API
> is to get values by column name, which involves a HashMap lookup on every
> call.
> - Tuple.values()
>

Sounds like the Tuple should implement Iterable.


>
> 4. Simplify createTable API: use POJO-based configuration.
> Creating a Consumer when some properties are optional seems to be
> non-trivial.
>

Yes, it's currently convoluted for sure. To my knowledge, there are plans
to improve this, but I will let other folks chime in with more details.


>
> 5. Add async methods to IgniteTables API (all methods are async inside
> anyway)
>

We already have async counterparts for all the methods where this is
applicable. E.g., for TableView#get, there is TableView#getAsync. Is there
anything else that you propose to add?


>
>
> Thoughts, objections?
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
>
> [2]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

Regarding the API, please take a look at this package:
https://github.com/apache/ignite-3/tree/main/modules/api/src/main/java/org/apache/ignite/table

'Table' is the primary API, which works with raw tuples. There are also
multiple views on top of it, including KeyValueView and KeyValueBinaryView
- these provide the key-value API.

The ultimate goal is for the thin client to implement all these APIs.

-Val

On Thu, Jul 1, 2021 at 8:42 AM Igor Sapego  wrote:

> Ivan, what are extra serde steps you are talking about?
>
> Best Regards,
> Igor
>
>
> On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky 
> wrote:
>
> > > I agree. But this was decided before in IEP-54, and is out of scope for
> > current IEP.
> > Would you like to start a separate thread to discuss this? Or I can do
> this
> > a bit later.
> >
> > Great idea, let's discuss it. I suppose this will simplify many aspects
> of
> > realization and improve performance a lot
> >
> > чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > > utf-8 reasonable field name), overhead will be negligible, but
> > realization
> > > of a client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > >
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >>
> > >> There is no code for batch operations yet.
> > >>
> > >> Here is the description of TUPLE_GET_ALL:
> > >> - UUID: table ID
> > >> - int: schema ID
> > >> - arr of arr: array of rows with values for all columns in given
> schema
> > >> (nil when value is missing for a column)
> > >>
> > >> As you can see, schema ID is written once for all rows.
> > >> A row is just a set of values according to the schema.
> > >>
> > >>
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >>
> > >> I agree. But this was decided before in IEP-54, and is out of scope
> for
> > >> current IEP.
> > >> Would you like to start a separate thread to discuss this? Or I can do
> > >> this
> > >> a bit later.
> > >>
> > >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > >> wrote:
> > >>
> > >> > > This is described in all operations that include multiple tuples.
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >> >
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >> >
> > >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > > >  that there is not neccesary to write schema versions in each
> row
> > >> > > > in collectionof tuples
> > >> > >
> > >> > > This is described in all operations that include multiple tuples.
> > >> > >
> > >> > >
> > >> > > > it is not clear from your code (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each other
> > >> > >
> > >> > > Key tuples include only key columns. Key columns come first in the
> > >> > schema.
> > >> > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > >> > > section.
> > >> > >
> > >> > >
> > >> > > > As for me, these excercises with schema's doesn't worth a lot
> > >> > >
> > >> > > I'll add a benchmark and we'll see.
> > >> > >
> > >> > >
> > >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > I suppose, that there is not neccesary to write schema versions
> in
> > >> each
> > >> > > row
> > >> > > > in collectionof tuples. Also it is not clear from your code
> > >> (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each
> other.
> > In
> > >> > > > readTuple you always read full schema and check for full length.
> > As
> > >> for
> > >> > > me,
> > >> > > > these excercises with schema's doesn't worth a lot. I.e.
> postgres
> > >> just
> > >> > > > writes field names and then simpy rows with data. Saving few
> bytes
> > >> > > doesn't
> > >> > > > make much deal. Btw, msgpack has special types for short strings
> > >> (i.e.
> > >> > > > str8). It is much easier use it and write field name as is.
> > >> > > >
> > >> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn  >:
> > >> > > >
> > >> > > 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Folks,

*Anything* that a user provides to the system can potentially be considered
sensitive information. This includes the VM arguments. We can't predict
what exactly one can put there, so let's not make assumptions.

When dealing with security, we should be as conservative as possible. That
said, I do not even agree with the pattern approach - there might be a
user's system property named IGNITE_xxx. It is also possible for our
internal properties to contain sensitive information (not all of them are
boolean flags).

The only option I see is to print out specific properties for which we
agree that they are safe. For example, we can introduce an annotation that
would mark "safe" properties in IgniteSystemProperties; we will then print
out only those that are marked with the annotation.

-Val


On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин 
wrote:

> Hello Ivan,
>
> > At least, we could just hide params that match a specific pattern
> Yes, we can filter out all vm options that do not relate to Ignite, for
> instance.
>
> > Ilya, go ahead, file ticket and prepare a PR.
> Please do not rush. Let's listen to other community members. This question
> is about security and it should not be discussed in a hurry (even though it
> looks like an obvious thing).
>
> Thanks,
> S.
>
> чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
>
> > I suppose, that all normal users should not suffer from this
> restrictions.
> > Nobody will pass password using jvm options. It is absolutely insane,
> > normal users pass passwords using environment variables.
> >
> > At least, we could just hide params that match specific pattern
> >
> > Ilya, go ahead, file ticket and prepare a PR.
> >
> > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин :
> >
> > > Hello,
> > >
> > > Unfortunately, the user can pass its own system properties via JVM
> > options
> > > as follows: -DMY_SECRET_PASSWORD=123
> > > It does not seem, this approach is the best one. However, the user
> should
> > > have a "kostyl" in order to hide these properties and values in the log
> > > file, IMHO.
> > >
> > > Thanks,
> > > S.
> > >
> > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :
> > >
> > > > Hi Igniters,
> > > >
> > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now,
> > > method
> > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > >
> > > > Is this behaviour actual now? Often, we should be able to get from
> logs
> > > the
> > > > actual VM options used at startup even if output of sensitive data is
> > > > restricted.
> > > >
> > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > 2.
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > 3.
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > >
> > >
> >
>

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Not all of them are OK to be printed out. At the very least, we should have
a mechanism to exclude some of them. I would still go with opt-in rather
than opt-out though, but I guess that is up to a discussion.

-Val

On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky  wrote:

> This is security through obscurity, an obvious and a well-known anti
> pattern. I suppose that printing jvm options, that is registered by
> @IgniteSystemProperty annotation is an ideal variant
>
> чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Folks,
> >
> > *Anything* that a user provides to the system can potentially be
> considered
> > sensitive information. This includes the VM arguments. We can't predict
> > what exactly one can put there, so let's not make assumptions.
> >
> > When dealing with security, we should be as conservative as possible.
> That
> > said, I do not even agree with the pattern approach - there might be a
> > user's system property named IGNITE_xxx. It is also possible for our
> > internal properties to contain sensitive information (not all of them are
> > boolean flags).
> >
> > The only option I see is to print out specific properties for which we
> > agree that they are safe. For example, we can introduce an annotation
> that
> > would mark "safe" properties in IgniteSystemProperties; we will then
> print
> > out only those that are marked with the annotation.
> >
> > -Val
> >
> >
> > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> slava.kopti...@gmail.com
> > >
> > wrote:
> >
> > > Hello Ivan,
> > >
> > > > At least, we could just hide params that match a specific pattern
> > > Yes, we can filter out all vm options that do not relate to Ignite, for
> > > instance.
> > >
> > > > Ilya, go ahead, file ticket and prepare a PR.
> > > Please do not rush. Let's listen to other community members. This
> > question
> > > is about security and it should not be discussed in a hurry (even
> though
> > it
> > > looks like an obvious thing).
> > >
> > > Thanks,
> > > S.
> > >
> > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > >
> > > > I suppose, that all normal users should not suffer from this
> > > restrictions.
> > > > Nobody will pass password using jvm options. It is absolutely insane,
> > > > normal users pass passwords using environment variables.
> > > >
> > > > At least, we could just hide params that match specific pattern
> > > >
> > > > Ilya, go ahead, file ticket and prepare a PR.
> > > >
> > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> slava.kopti...@gmail.com
> > >:
> > > >
> > > > > Hello,
> > > > >
> > > > > Unfortunately, the user can pass its own system properties via JVM
> > > > options
> > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > It does not seem, this approach is the best one. However, the user
> > > should
> > > > > have a "kostyl" in order to hide these properties and values in the
> > log
> > > > > file, IMHO.
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya  >:
> > > > >
> > > > > > Hi Igniters,
> > > > > >
> > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till
> > now,
> > > > > method
> > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > >
> > > > > > Is this behaviour actual now? Often, we should be able to get
> from
> > > logs
> > > > > the
> > > > > > actual VM options used at startup even if output of sensitive
> data
> > is
> > > > > > restricted.
> > > > > >
> > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > 2.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > 3.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

I was answering your question about the KV API. The API I provided has been
discussed and agreed upon. One of the goals of the protocol is to implement
this API, so it should give you a clear idea of what we're looking for.

Of course, I agree with you that the protocol should be simple and flexible
enough to allow other implementations for different languages and platforms.

-Val

On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky  wrote:

> Andrey, yep, you are right. This was just a quick idea. As for me, I just
> don't want to repeat the same problem with compactFooter in thin client api
> of ignite 2.x.
>
> чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov :
>
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> >
> >
> > Ivan,
> > I think GET and PUT operation parameters should be consistent.
> > With PUT operation this way may be tricky.
> >
> > SQL INSERT operation (which is similar PUT operation) semantic allows
> > skipping columns that have a default value.
> > Assume we have smth like this:
> >
> > CREATE TABLE t1 (
> >'id' INT;
> >'colname' VARCHAR DEFAULT "abc";
> > )
> > INSERT INTO t1 VALUES(1)
> >
> > Actually, this will add a row (1, "abc")
> >
> > Your suggestion related to missed fields will not work this way as it is
> > impossible to distinct
> > case with 'null' value from the case with a default value.
> >
> >
> > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> > wrote:
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > utf-8
> > > reasonable field name), overhead will be negligible, but realization
> of a
> > > client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > >
> > > > > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > > > > schema id in each tuple.
> > > >
> > > > There is no code for batch operations yet.
> > > >
> > > > Here is the description of TUPLE_GET_ALL:
> > > > - UUID: table ID
> > > > - int: schema ID
> > > > - arr of arr: array of rows with values for all columns in given
> schema
> > > > (nil when value is missing for a column)
> > > >
> > > > As you can see, schema ID is written once for all rows.
> > > > A row is just a set of values according to the schema.
> > > >
> > > >
> > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > > > bytearray to internal api, and use msgpack throughout all wire
> > > protocols,
> > > > > as tarantool does.
> > > >
> > > > I agree. But this was decided before in IEP-54, and is out of scope
> for
> > > > current IEP.
> > > > Would you like to start a separate thread to discuss this? Or I can
> do
> > > this
> > > > a bit later.
> > > >
> > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > This is described in all operations that include multiple tuples.
> > > > > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > > > > schema id in each tuple.
> > > > >
> > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > > > bytearray to internal api, and use msgpack throughout all wire
> > > protocols,
> > > > > as tarantool does.
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan,
> > > > > >
> > > > > > >  that there is not neccesary to write schema versions in each
> row
> > > > > > > in collectionof tuples
> > > > > >
> > > > > > This is described in all operations that include multiple tuples.
> > > > > >
> > > > > >
> > > > > > > it is not clear from your code (probably
> > > > > > > mistake?) how differ key tuples and value tuples from each
> other
> > > > > >
> > > > > > Key tuples include only key columns. Key columns come first in
> the
> > > > > schema.
> > > > > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > > > > > section.
> > > > > >
> > > > > >
> > > > > > > As for me, these excercises with schema's doesn't worth a lot
> > > > > >
> > > > > > I'll add a benchmark and we'll see.
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I suppose, that there is not neccesary to write schema versions
> > in
> > > > each
> > > > > > row
> 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Ivan,

IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
(e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to be
sensitive.

Once again - I would exclude any property that can contain user-specific
information. Only our internal settings (stuff
like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.

-Val

On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky  wrote:

> We can add add an extra param in annotation, that blocks param to be
> printed, just set it to false by default and block it wheb set to true
>
> чт, 1 июл. 2021 г., 19:45 Atri Sharma :
>
> > What if we allowed a blocklist of parameters that are never printed?
> >
> > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Not all of them are OK to be printed out. At the very least, we should
> > have
> > > a mechanism to exclude some of them. I would still go with opt-in
> rather
> > > than opt-out though, but I guess that is up to a discussion.
> > >
> > > -Val
> > >
> > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > > wrote:
> > >
> > > > This is security through obscurity, an obvious and a well-known anti
> > > > pattern. I suppose that printing jvm options, that is registered by
> > > > @IgniteSystemProperty annotation is an ideal variant
> > > >
> > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com
> > > > >:
> > > >
> > > > > Folks,
> > > > >
> > > > > *Anything* that a user provides to the system can potentially be
> > > > considered
> > > > > sensitive information. This includes the VM arguments. We can't
> > predict
> > > > > what exactly one can put there, so let's not make assumptions.
> > > > >
> > > > > When dealing with security, we should be as conservative as
> possible.
> > > > That
> > > > > said, I do not even agree with the pattern approach - there might
> be
> > a
> > > > > user's system property named IGNITE_xxx. It is also possible for
> our
> > > > > internal properties to contain sensitive information (not all of
> them
> > > are
> > > > > boolean flags).
> > > > >
> > > > > The only option I see is to print out specific properties for which
> > we
> > > > > agree that they are safe. For example, we can introduce an
> annotation
> > > > that
> > > > > would mark "safe" properties in IgniteSystemProperties; we will
> then
> > > > print
> > > > > out only those that are marked with the annotation.
> > > > >
> > > > > -Val
> > > > >
> > > > >
> > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > slava.kopti...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Ivan,
> > > > > >
> > > > > > > At least, we could just hide params that match a specific
> pattern
> > > > > > Yes, we can filter out all vm options that do not relate to
> Ignite,
> > > for
> > > > > > instance.
> > > > > >
> > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > Please do not rush. Let's listen to other community members. This
> > > > > question
> > > > > > is about security and it should not be discussed in a hurry (even
> > > > though
> > > > > it
> > > > > > looks like an obvious thing).
> > > > > >
> > > > > > Thanks,
> > > > > > S.
> > > > > >
> > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky  >:
> > > > > >
> > > > > > > I suppose, that all normal users should not suffer from this
> > > > > > restrictions.
> > > > > > > Nobody will pass password using jvm options. It is absolutely
> > > insane,
> > > > > > > normal users pass passwords using environment variables.
> > > > > > >
> > > > > > > At least, we could just hide params that match specific pattern
> > > > > > >
> &g

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Ilya,

I don't think this is the best approach because there are so many
properties that contain so many different types of information. "All or
nothing" doesn't really fit here. We want to have the ability to exclude
sensitive information, but still print out internal settings that are
useful for debugging, etc.

-Val

On Thu, Jul 1, 2021 at 9:54 AM Shishkov Ilya  wrote:

> Folks,
>
> > Maybe we should add an extra JVM option (e.g.
> IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> > but if set to 'true' then #ackVmOptions will print VM arguments even if
> sensitive data is restricted?
>
> What do you think about an extra JVM option?
>
> чт, 1 июл. 2021 г. в 19:51, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Ivan,
> >
> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to
> be
> > sensitive.
> >
> > Once again - I would exclude any property that can contain user-specific
> > information. Only our internal settings (stuff
> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> > wrote:
> >
> > > We can add add an extra param in annotation, that blocks param to be
> > > printed, just set it to false by default and block it wheb set to true
> > >
> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> > >
> > > > What if we allowed a blocklist of parameters that are never printed?
> > > >
> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Not all of them are OK to be printed out. At the very least, we
> > should
> > > > have
> > > > > a mechanism to exclude some of them. I would still go with opt-in
> > > rather
> > > > > than opt-out though, but I guess that is up to a discussion.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is security through obscurity, an obvious and a well-known
> > anti
> > > > > > pattern. I suppose that printing jvm options, that is registered
> by
> > > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > *Anything* that a user provides to the system can potentially
> be
> > > > > > considered
> > > > > > > sensitive information. This includes the VM arguments. We can't
> > > > predict
> > > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > > >
> > > > > > > When dealing with security, we should be as conservative as
> > > possible.
> > > > > > That
> > > > > > > said, I do not even agree with the pattern approach - there
> might
> > > be
> > > > a
> > > > > > > user's system property named IGNITE_xxx. It is also possible
> for
> > > our
> > > > > > > internal properties to contain sensitive information (not all
> of
> > > them
> > > > > are
> > > > > > > boolean flags).
> > > > > > >
> > > > > > > The only option I see is to print out specific properties for
> > which
> > > > we
> > > > > > > agree that they are safe. For example, we can introduce an
> > > annotation
> > > > > > that
> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > > then
> > > > > > print
> > > > > > > out only those that are marked with the annotation.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

An extra argument for IgniteSystemProperty sounds reasonable.

-Val

On Thu, Jul 1, 2021 at 10:04 AM Ivan Daschinsky  wrote:

> Ok, this can be excluded using blocklist-jvm-params.properties or just by
> providing and extra arg to annotation, as I have just suggested
>
> чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to
> be
> > sensitive.
> >
> > Once again - I would exclude any property that can contain user-specific
> > information. Only our internal settings (stuff
> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> > wrote:
> >
> > > We can add add an extra param in annotation, that blocks param to be
> > > printed, just set it to false by default and block it wheb set to true
> > >
> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> > >
> > > > What if we allowed a blocklist of parameters that are never printed?
> > > >
> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Not all of them are OK to be printed out. At the very least, we
> > should
> > > > have
> > > > > a mechanism to exclude some of them. I would still go with opt-in
> > > rather
> > > > > than opt-out though, but I guess that is up to a discussion.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is security through obscurity, an obvious and a well-known
> > anti
> > > > > > pattern. I suppose that printing jvm options, that is registered
> by
> > > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > *Anything* that a user provides to the system can potentially
> be
> > > > > > considered
> > > > > > > sensitive information. This includes the VM arguments. We can't
> > > > predict
> > > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > > >
> > > > > > > When dealing with security, we should be as conservative as
> > > possible.
> > > > > > That
> > > > > > > said, I do not even agree with the pattern approach - there
> might
> > > be
> > > > a
> > > > > > > user's system property named IGNITE_xxx. It is also possible
> for
> > > our
> > > > > > > internal properties to contain sensitive information (not all
> of
> > > them
> > > > > are
> > > > > > > boolean flags).
> > > > > > >
> > > > > > > The only option I see is to print out specific properties for
> > which
> > > > we
> > > > > > > agree that they are safe. For example, we can introduce an
> > > annotation
> > > > > > that
> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > > then
> > > > > > print
> > > > > > > out only those that are marked with the annotation.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ivan,
> > > > > > > >
> > > > > > > > > At least, we could just hide params that match a specific
> > > pattern
> > > > > >

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

KV view does work over the tuples. Nested objects and arbitrary structures
can be stored as blobs. So if you need a basic KV cache, you can always
create a table with two blob fields - one for key and one for value - and
store anything there.

-Val

On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky  wrote:

> Val, am I right, that kv view over the tuples is just simple mapping from
> POJO to tuple? No collections, no nested objects? I have discussed this in
> private with Igor and Pavel and they told me different info.
>
> чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > I was answering your question about the KV API. The API I provided has
> been
> > discussed and agreed upon. One of the goals of the protocol is to
> implement
> > this API, so it should give you a clear idea of what we're looking for.
> >
> > Of course, I agree with you that the protocol should be simple and
> flexible
> > enough to allow other implementations for different languages and
> > platforms.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> > wrote:
> >
> > > Andrey, yep, you are right. This was just a quick idea. As for me, I
> just
> > > don't want to repeat the same problem with compactFooter in thin client
> > api
> > > of ignite 2.x.
> > >
> > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov  >:
> > >
> > > > >
> > > > > I suppose that we should describe this more verbose and explicit. I
> > > > > nevertheless suggest to also consider writing values this way:
> > > > > - arr of fields names (if name is missed, corresponding field is
> nil)
> > > > > - arr of rows (row as array, length equal to fields array)
> > > >
> > > >
> > > > Ivan,
> > > > I think GET and PUT operation parameters should be consistent.
> > > > With PUT operation this way may be tricky.
> > > >
> > > > SQL INSERT operation (which is similar PUT operation) semantic allows
> > > > skipping columns that have a default value.
> > > > Assume we have smth like this:
> > > >
> > > > CREATE TABLE t1 (
> > > >'id' INT;
> > > >'colname' VARCHAR DEFAULT "abc";
> > > > )
> > > > INSERT INTO t1 VALUES(1)
> > > >
> > > > Actually, this will add a row (1, "abc")
> > > >
> > > > Your suggestion related to missed fields will not work this way as it
> > is
> > > > impossible to distinct
> > > > case with 'null' value from the case with a default value.
> > > >
> > > >
> > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > - UUID: table ID
> > > > > - int: schema ID
> > > > > - arr of arr: array of rows with values for all columns in given
> > schema
> > > > >
> > > > > I suppose that we should describe this more verbose and explicit. I
> > > > > nevertheless suggest to also consider writing values this way:
> > > > > - arr of fields names (if name is missed, corresponding field is
> nil)
> > > > > - arr of rows (row as array, length equal to fields array)
> > > > >
> > > > > It is quite simple and if we use str8 (it is more than enough for
> any
> > > > utf-8
> > > > > reasonable field name), overhead will be negligible, but
> realization
> > > of a
> > > > > client will be way simpler
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > > > >
> > > > > > > No it isn't, I have carefully read code and IEP, in your code
> you
> > > > write
> > > > > > > schema id in each tuple.
> > > > > >
> > > > > > There is no code for batch operations yet.
> > > > > >
> > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > > - UUID: table ID
> > > > > > - int: schema ID
> > > > > > - arr of arr: array of rows with values for all columns in given
> > > schema
> > > > > > (nil when value is missing for a column)
> > > > > >
> > > > > > As you can see, schema ID is wri

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Pavel,

If Ignition is the single entry point, it needs to have access to the
server node instance, so it needs to depend on ignite-runner. Therefore,
including Ignition in ignite-client means that ignite-client depends on
ignite-runner, which we cannot have.

-Val

On Sat, Jul 10, 2021 at 4:17 AM Pavel Tupitsyn  wrote:

> Val,
>
> My suggestion is to have Ignition class in ignite-client module.
>
> On Fri, Jul 9, 2021 at 11:01 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Pavel,
> >
> > Ivan actually brings a good point. While the client is in a separate
> > module, Ignition (if we make it static) will have to depend on both
> > ignite-client and ignite-runner, and we will have to ship it along with
> the
> > client. This indeed creates an uber-jar, so we can't really have a single
> > entry point, unfortunately.
> >
> > I'm not sure what is the best way to proceed here. Let's think it over
> and
> > see if there are any suggestions.
> >
> > -Val
> >
> > On Fri, Jul 9, 2021 at 6:31 AM Pavel Tupitsyn 
> > wrote:
> >
> > > > why thin client should be in core module
> > >
> > > It will be in a separate module (ignite-client).
> > > I was talking about "core library" as a primary set of modules that we
> > > ship.
> > > Integrations with 3rd party libraries and frameworks can be shipped as
> > > extensions.
> > >
> > > Anyway, let's postpone the discussion of Rx and Kotlin.
> > > The main goal right now is to implement the most basic Java thin
> client.
> > > CompletableFuture is the primary way to deliver async APIs in Ignite
> 3.0,
> > > other things can be added later.
> > >
> > > On Fri, Jul 9, 2021 at 3:37 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > > I don't think we need an explicit reactive API in the core library.
> > > >
> > > > Have you ever thought about why thin client should be in core module?
> > Why
> > > > we do the same thing as we did in ignite 2.x? In the days of cloud
> > native
> > > > we still think about large uber-jar with everything?
> > > >
> > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > Users don't want to code by theyselves, they want to use ready and
> > > complete
> > > > clients. Please, don't underestimate kotlin, kotlin coroutines and
> > > reactive
> > > > streams. They are all the first class citizens in spring 5 for 3
> years
> > > >
> > > > пт, 9 июл. 2021 г., 14:43 Pavel Tupitsyn :
> > > >
> > > > > > You forget about reactive api
> > > > >
> > > > > I don't think we need an explicit reactive API in the core library.
> > > > > Observable.fromFuture bridges async to Rx easily:
> > > > >
> > > > > Observable.fromFuture(client.putAsync(k, v)).flatMap(...)
> > > > >
> > > > > Same story with Kotlin, it works with CompletableFuture.
> > > > >
> > > > > On Fri, Jul 9, 2021 at 1:31 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > You forget about reactive api :)
> > > > > >
> > > > > > And whats a problem with discocerability?
> > > > > >
> > > > > > var syncApi = client.sync();
> > > > > > syncApi.put(k, v);
> > > > > >
> > > > > > var rxApi = client.reactive();
> > > > > > rxApi.put(k,v).flatMap(res -> );
> > > > > >
> > > > > > And sync, async and reactive is not enough, it is good idea to
> > > support
> > > > > > kotlin coroutines also :)
> > > > > >
> > > > > > пт, 9 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > > > >
> > > > > > > Ivan D.,
> > > > > > >
> > > > > > > > container of properties
> > > > > > >
> > > > > > > What is a container of properties?
> > > > > > > As a user, I want a simple way to start a client and perform
> > > > > operations.
> > > > > > >
> > > > > > > I don't want anything confusing and complicated like Netty
> > > Bootstrap.
> > > > > > There
> > > > > > > might be a reason for Nett

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

The proposal looks good to me.

-Val

On Tue, Jul 6, 2021 at 2:24 AM Ivan Daschinsky  wrote:

> I suppose, that general idea is great. Some details are missing, but I
> suppose during implementation of clients we will add more details and
> redefine some parts.
>
> вт, 6 июл. 2021 г., 09:59 Pavel Tupitsyn :
>
> > Ivan, Val, and others - are there any open objections or questions?
> > Can we accept the proposal?
> >
> > On Mon, Jul 5, 2021 at 1:28 PM Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > I've updated the IEP to support "Live Schema" [1] from IEP-54.
> > > Some operations now have schemaless variants, where tuples are
> serialized
> > > as maps (String -> val).
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-54%3A+Schema-first+Approach#IEP54:SchemafirstApproach-Dynamicschemaexpansion(Live-schema)
> > >
> > > On Thu, Jul 1, 2021 at 8:31 PM Ivan Daschinsky 
> > > wrote:
> > >
> > >> Val, my understanding about it was exactly the same as yours, but,
> > again,
> > >> I
> > >> heard a different opinion.
> > >>
> > >> But nevertheless, binary protocol should not be about objects, records
> > aka
> > >> tuples are the best varii, simple and powerful.
> > >>
> > >> As for me, I want to take part in implementing python and golang thin
> > >> clients for ignite 3, so consider my remarks using this info. I am not
> > >> just
> > >> a rude critic,
> > >> I am just interested in consice and universal binary prorocol
> > >> чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com
> > >> >:
> > >>
> > >> > Ivan,
> > >> >
> > >> > KV view does work over the tuples. Nested objects and arbitrary
> > >> structures
> > >> > can be stored as blobs. So if you need a basic KV cache, you can
> > always
> > >> > create a table with two blob fields - one for key and one for value
> -
> > >> and
> > >> > store anything there.
> > >> >
> > >> > -Val
> > >> >
> > >> > On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky  >
> > >> > wrote:
> > >> >
> > >> > > Val, am I right, that kv view over the tuples is just simple
> mapping
> > >> from
> > >> > > POJO to tuple? No collections, no nested objects? I have discussed
> > >> this
> > >> > in
> > >> > > private with Igor and Pavel and they told me different info.
> > >> > >
> > >> > > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > >> > > valentin.kuliche...@gmail.com
> > >> > > >:
> > >> > >
> > >> > > > Ivan,
> > >> > > >
> > >> > > > I was answering your question about the KV API. The API I
> provided
> > >> has
> > >> > > been
> > >> > > > discussed and agreed upon. One of the goals of the protocol is
> to
> > >> > > implement
> > >> > > > this API, so it should give you a clear idea of what we're
> looking
> > >> for.
> > >> > > >
> > >> > > > Of course, I agree with you that the protocol should be simple
> and
> > >> > > flexible
> > >> > > > enough to allow other implementations for different languages
> and
> > >> > > > platforms.
> > >> > > >
> > >> > > > -Val
> > >> > > >
> > >> > > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky <
> > ivanda...@gmail.com
> > >> >
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Andrey, yep, you are right. This was just a quick idea. As for
> > >> me, I
> > >> > > just
> > >> > > > > don't want to repeat the same problem with compactFooter in
> thin
> > >> > client
> > >> > > > api
> > >> > > > > of ignite 2.x.
> > >> > > > >
> > >> > > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
> > >> > andrey.mashen...@gmail.com
> > >> >

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Valentin Kulichenko]

Pavel,

Optionals are available in Java and we can use them. This is still boxing
though, and I don't know what the performance impact would be. In addition,
optional API is redundant for non-nullable fields. Perhaps, we can provide
both options (e.g. having intValue() and intValueOptional() methods).

Out of curiosity, what would this code do if the value is NULL? What is the
type of the 'weight' variable?

if (tuple.TryGetFloatValue("weight", out var weight))
doSomething(weight)

-Val

On Tue, Jul 6, 2021 at 2:13 AM Ivan Daschinsky  wrote:

> Ah, I see, you meant Optionals family. Yep, it is worth to think about.
>
> вт, 6 июл. 2021 г., 10:06 Pavel Tupitsyn :
>
> > Ivan,
> >
> > Nothing wrong except for performance concerns.
> > The following code looks up the column by name twice:
> >
> > if (!tuple.isNull("weight"))
> >doSomething(tuple.floatValue("weight"))
> >
> > Whereas in other languages you could do it in one shot:
> >
> > if (tuple.TryGetFloatValue("weight", out var weight))
> > doSomething(weight)
> >
> > or Option weight = tuple.floatValue("weight") and so on.
> >
> > On Tue, Jul 6, 2021 at 9:58 AM Ivan Daschinsky 
> > wrote:
> >
> > > Sorry, but what is wrong with simple method isNull()
> > >
> > > вт, 6 июл. 2021 г., 09:55 Pavel Tupitsyn :
> > >
> > > > Val,
> > > >
> > > > > I don't think there is a significantly better way
> > > > > of doing this in Java.
> > > >
> > > > Yep looks like there is no way to return two values without boxing.
> > > > No ref, no out, no value types.
> > > >
> > > > > Schema already provides this information, doesn't it?
> > > >
> > > > It does, though we don't have an agreement on how to expose this on
> > > public
> > > > API yet,
> > > > or do we?
> > > >
> > > > On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Pavel,
> > > > >
> > > > > That's a good point, but I don't think there is a significantly
> > better
> > > > way
> > > > > of doing this in Java.
> > > > >
> > > > > There should be a way to check if a field is nullable or not
> though.
> > > > Schema
> > > > > already provides this information, doesn't it?
> > > > >
> > > > > -Val
> > > > >
> > > > > On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn <
> ptupit...@apache.org
> > >
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > Looks like Tuple API has no efficient way to tell if a value for
> a
> > > > > nullable
> > > > > > column of primitive type is null.
> > > > > >
> > > > > > - Tuple#intValue() will return 0 when the actual value is null =>
> > not
> > > > > clear
> > > > > > if 0 is 0 or null.
> > > > > > - Tuple#value() works, but is more expensive due to boxing and
> type
> > > > > lookup.
> > > > > >
> > > > > > Any ideas on how to improve this?
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Valentin Kulichenko]

Pavel,

That's a good point, but I don't think there is a significantly better way
of doing this in Java.

There should be a way to check if a field is nullable or not though. Schema
already provides this information, doesn't it?

-Val

On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn  wrote:

> Igniters,
>
> Looks like Tuple API has no efficient way to tell if a value for a nullable
> column of primitive type is null.
>
> - Tuple#intValue() will return 0 when the actual value is null => not clear
> if 0 is 0 or null.
> - Tuple#value() works, but is more expensive due to boxing and type lookup.
>
> Any ideas on how to improve this?
>

Re: Ignite 3.0 Tuple API: how to check if value is null?

[Valentin Kulichenko]

So what happens if the value is NULL? Exception?

-Val

On Tue, Jul 6, 2021 at 1:52 PM Ivan Daschinsky  wrote:

> > Out of curiosity, what would this code do if the value is NULL? What is
> the
> type of the 'weight' variable?
>
> float of course.
> https://www.c-sharpcorner.com/article/out-parameter-in-c-sharp-7/
>
>
> вт, 6 июл. 2021 г., 22:30 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Pavel,
> >
> > Optionals are available in Java and we can use them. This is still boxing
> > though, and I don't know what the performance impact would be. In
> addition,
> > optional API is redundant for non-nullable fields. Perhaps, we can
> provide
> > both options (e.g. having intValue() and intValueOptional() methods).
> >
> > Out of curiosity, what would this code do if the value is NULL? What is
> the
> > type of the 'weight' variable?
> >
> > if (tuple.TryGetFloatValue("weight", out var weight))
> > doSomething(weight)
> >
> > -Val
> >
> > On Tue, Jul 6, 2021 at 2:13 AM Ivan Daschinsky 
> > wrote:
> >
> > > Ah, I see, you meant Optionals family. Yep, it is worth to think about.
> > >
> > > вт, 6 июл. 2021 г., 10:06 Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > Nothing wrong except for performance concerns.
> > > > The following code looks up the column by name twice:
> > > >
> > > > if (!tuple.isNull("weight"))
> > > >doSomething(tuple.floatValue("weight"))
> > > >
> > > > Whereas in other languages you could do it in one shot:
> > > >
> > > > if (tuple.TryGetFloatValue("weight", out var weight))
> > > > doSomething(weight)
> > > >
> > > > or Option weight = tuple.floatValue("weight") and so on.
> > > >
> > > > On Tue, Jul 6, 2021 at 9:58 AM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > Sorry, but what is wrong with simple method isNull()
> > > > >
> > > > > вт, 6 июл. 2021 г., 09:55 Pavel Tupitsyn :
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > > I don't think there is a significantly better way
> > > > > > > of doing this in Java.
> > > > > >
> > > > > > Yep looks like there is no way to return two values without
> boxing.
> > > > > > No ref, no out, no value types.
> > > > > >
> > > > > > > Schema already provides this information, doesn't it?
> > > > > >
> > > > > > It does, though we don't have an agreement on how to expose this
> on
> > > > > public
> > > > > > API yet,
> > > > > > or do we?
> > > > > >
> > > > > > On Tue, Jul 6, 2021 at 12:44 AM Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com> wrote:
> > > > > >
> > > > > > > Pavel,
> > > > > > >
> > > > > > > That's a good point, but I don't think there is a significantly
> > > > better
> > > > > > way
> > > > > > > of doing this in Java.
> > > > > > >
> > > > > > > There should be a way to check if a field is nullable or not
> > > though.
> > > > > > Schema
> > > > > > > already provides this information, doesn't it?
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > > On Mon, Jul 5, 2021 at 11:03 AM Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Igniters,
> > > > > > > >
> > > > > > > > Looks like Tuple API has no efficient way to tell if a value
> > for
> > > a
> > > > > > > nullable
> > > > > > > > column of primitive type is null.
> > > > > > > >
> > > > > > > > - Tuple#intValue() will return 0 when the actual value is
> null
> > =>
> > > > not
> > > > > > > clear
> > > > > > > > if 0 is 0 or null.
> > > > > > > > - Tuple#value() works, but is more expensive due to boxing
> and
> > > type
> > > > > > > lookup.
> > > > > > > >
> > > > > > > > Any ideas on how to improve this?
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

atter. It can be Ignition#startClient or
> > > > > > Ignition#startClientAsync, but I'd prefer lettuce approach
> > > > > >
> > > > > >
> > > > > > Also, it would be great to have separate facades for sync, async
> > and
> > > > > > reactive api. Mixing all of them in one interface is a
> > documentation
> > > > > > nightmare.
> > > > > >
> > > > > > пт, 9 июл. 2021 г., 11:55 Pavel Tupitsyn :
> > > > > >
> > > > > > > Ivan P., Ivan D.,
> > > > > > >
> > > > > > > I don't think it makes sense to separate IgniteConnection and
> > > > > > IgniteClient
> > > > > > > like Lettuce does,
> > > > > > > because IgniteClient will maintain connections to multiple
> server
> > > > nodes
> > > > > > > automatically,
> > > > > > > and the number of connections can grow and shrink dynamically.
> > > > > > >
> > > > > > > This is required to support dynamic clusters together with
> > > partition
> > > > > > > awareness.
> > > > > > >
> > > > > > >
> > > > > > > > Why not to make async variant of connection
> > > > > > >
> > > > > > > Ignite API will (eventually) have both sync and async variants
> of
> > > > every
> > > > > > > method, where applicable,
> > > > > > > including the method that connects the client to the cluster.
> > > > > > >
> > > > > > > On Fri, Jul 9, 2021 at 9:55 AM Ivan Pavlukhin <
> > vololo...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > > Ignition IS the entry point to Ignite, so I'm not sure I
> got
> > > your
> > > > > > point
> > > > > > > > :)
> > > > > > > > > Either way, please feel free to give your suggestions for
> an
> > > > > > > alternative
> > > > > > > > name if you have any.
> > > > > > > >
> > > > > > > > Well, it is not only about naming but it is also about code
> > > > > > > > organization. Ivan D. already referenced to alternative API
> > > styles
> > > > (I
> > > > > > > > suppose [1] describes the idea).
> > > > > > > >
> > > > > > > > My main points are:
> > > > > > > > 1. Ignite 3 is a great opportunity to make things better.
> > > > > > > > 2. Using (or reusing) confusing names and entities should be
> > > > avoided.
> > > > > > > >
> > > > > > > > Another rather straightforward startup/bootstrap approach is
> > used
> > > > in
> > > > > > > > Netty [2]. For Ignite I would spell it like
> > > IgniteServer.Bootstrap
> > > > > and
> > > > > > > > IgniteClient.Bootstrap.
> > > > > > > >
> > > > > > > > Also I suppose that thin client API is more important because
> > > much
> > > > > > > > more users will use it. I hope that a lot of Community
> members
> > > will
> > > > > > > > share their ideas.
> > > > > > > >
> > > > > > > > [1] https://www.baeldung.com/java-redis-lettuce
> > > > > > > > [2]
> > > > https://netty.io/4.0/api/io/netty/bootstrap/ServerBootstrap.html
> > > > > > > >
> > > > > > > > 2021-07-09 1:41 GMT+03:00, Valentin Kulichenko <
> > > > > > > > valentin.kuliche...@gmail.com>:
> > > > > > > > > Ivan,
> > > > > > > > >
> > > > > > > > > I've seen the link, but I still don't understand what
> exactly
> > > you
> > > > > > > propose
> > > > > > > > > to change in the current API, and what is your concern.
> Could
> > > you
> > > > > > > please
> > > > > > > > > clarify? How you think I

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Ivan,

I've seen the link, but I still don't understand what exactly you propose
to change in the current API, and what is your concern. Could you please
clarify? How you think Ignite API should look like?

-Val

On Thu, Jul 8, 2021 at 2:18 PM Ivan Daschinsky  wrote:

> Val, I have already gave examples -- lettuce, a very performant and modern
> redis java client
>
> I can duplicate links again
> https://lettuce.io/core/release/api/io/lettuce/core/RedisClient.html
>
> https://lettuce.io/core/release/api/io/lettuce/core/api/StatefulRedisConnection.html
> https://www.baeldung.com/java-redis-lettuce
>
> чт, 8 июл. 2021 г., 23:47 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > Can you please clarify what you mean by "separate creation of client and
> > connection"? Can you give an example?
> >
> > -Val
> >
> > On Thu, Jul 8, 2021 at 12:53 PM Ivan Daschinsky 
> > wrote:
> >
> > > I'm sorry, but why we didn't consider to separate creation of Client
> and
> > > connection? Why not to make async variant of connection? See for
> example
> > > [1]
> > > [1] --- https://lettuce.io/core/release/api/index.html
> > >
> > >
> > > чт, 8 июл. 2021 г., 09:50 Pavel Tupitsyn :
> > >
> > > > Val,
> > > >
> > > > So the plan is:
> > > >
> > > > - Remove Ignition#start from the public API
> > > > - Make Ignition a class, not an interface
> > > > - Add static Ignition#startClient
> > > >
> > > > Sounds good?
> > > >
> > > > On Thu, Jul 8, 2021 at 6:13 AM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Hi Ivan,
> > > > >
> > > > > Ignition IS the entry point to Ignite, so I'm not sure I got your
> > point
> > > > :)
> > > > > Where is the contradiction?
> > > > >
> > > > > Either way, please feel free to give your suggestions for an
> > > alternative
> > > > > name if you have any.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina <
> vololo...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > A side note. Actually “Ignition” naming always confused me. I
> think
> > > > about
> > > > > > it as some fancy named API entry point for Ignite. Perhaps it is
> a
> > > good
> > > > > > moment to revisit naming.
> > > > > >
> > > > > > > On 8 Jul 2021, at 07:09, Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi Pavel,
> > > > > > >
> > > > > > > I don't think we will need the pure embedded mode, but we still
> > > need
> > > > to
> > > > > > be
> > > > > > > able to access the API from compute and services. That said,
> > there
> > > > are
> > > > > > two
> > > > > > > usages of the 'Ignite' API:
> > > > > > >
> > > > > > >   1. Remote, via the binary protocol.
> > > > > > >   2. Local - needed for compute and services. (This is how it
> > works
> > > > > now.)
> > > > > > >
> > > > > > > I believe that the API should be the same, and there should be
> a
> > > > > unified
> > > > > > > access point. Ignition seems to be a good candidate for this.
> > > > > > >
> > > > > > > Ignition#start should eventually be removed from the public
> API.
> > It
> > > > is
> > > > > > > currently there only because we don't have the thin client yet.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >> On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn <
> > > ptupit...@apache.org
> > > > >
> > > > > > wrote:
> > > > > > >>
> > > > > > >> Igniters,
> > > > > > >>
> > > > > > >> I have a few questions regarding server node startup and thin
> > > > clients.
> > > > > > >>
> > > > > > >> State of things:
> > > > > > >> - Server nodes will be started with 'ignite run' from CLI [1]
> > > > > > >> - ignite-api module represents our public API
> > > > > > >> - ignite-api has Ignition interface to start server nodes
> > > > > > >>
> > > > > > >> Questions:
> > > > > > >> - What's the idea behind Ignition interface in the public API?
> > Are
> > > > we
> > > > > > going
> > > > > > >> to have an "embedded mode" where servers can be started from
> > > code? I
> > > > > > >> thought this was not planned.
> > > > > > >> - How are users supposed to retrieve an instance of the
> Ignition
> > > > > > interface?
> > > > > > >> - Are there any plans to start thin clients from Ignition
> > > interface,
> > > > > or
> > > > > > >> should we have a separate way of doing this?
> > > > > > >>
> > > > > > >>
> > > > > > >> [1]
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Pavel,

Yes, makes sense.

-Val

On Wed, Jul 7, 2021 at 11:50 PM Pavel Tupitsyn  wrote:

> Val,
>
> So the plan is:
>
> - Remove Ignition#start from the public API
> - Make Ignition a class, not an interface
> - Add static Ignition#startClient
>
> Sounds good?
>
> On Thu, Jul 8, 2021 at 6:13 AM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Ivan,
> >
> > Ignition IS the entry point to Ignite, so I'm not sure I got your point
> :)
> > Where is the contradiction?
> >
> > Either way, please feel free to give your suggestions for an alternative
> > name if you have any.
> >
> > -Val
> >
> > On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina 
> > wrote:
> >
> > > A side note. Actually “Ignition” naming always confused me. I think
> about
> > > it as some fancy named API entry point for Ignite. Perhaps it is a good
> > > moment to revisit naming.
> > >
> > > > On 8 Jul 2021, at 07:09, Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > Hi Pavel,
> > > >
> > > > I don't think we will need the pure embedded mode, but we still need
> to
> > > be
> > > > able to access the API from compute and services. That said, there
> are
> > > two
> > > > usages of the 'Ignite' API:
> > > >
> > > >   1. Remote, via the binary protocol.
> > > >   2. Local - needed for compute and services. (This is how it works
> > now.)
> > > >
> > > > I believe that the API should be the same, and there should be a
> > unified
> > > > access point. Ignition seems to be a good candidate for this.
> > > >
> > > > Ignition#start should eventually be removed from the public API. It
> is
> > > > currently there only because we don't have the thin client yet.
> > > >
> > > > -Val
> > > >
> > > >> On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn  >
> > > wrote:
> > > >>
> > > >> Igniters,
> > > >>
> > > >> I have a few questions regarding server node startup and thin
> clients.
> > > >>
> > > >> State of things:
> > > >> - Server nodes will be started with 'ignite run' from CLI [1]
> > > >> - ignite-api module represents our public API
> > > >> - ignite-api has Ignition interface to start server nodes
> > > >>
> > > >> Questions:
> > > >> - What's the idea behind Ignition interface in the public API? Are
> we
> > > going
> > > >> to have an "embedded mode" where servers can be started from code? I
> > > >> thought this was not planned.
> > > >> - How are users supposed to retrieve an instance of the Ignition
> > > interface?
> > > >> - Are there any plans to start thin clients from Ignition interface,
> > or
> > > >> should we have a separate way of doing this?
> > > >>
> > > >>
> > > >> [1]
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > >>
> > >
> >
>

Re: Ignite 3.0 Ignition API, node startup, and thin client startup

[Valentin Kulichenko]

Ivan,

Can you please clarify what you mean by "separate creation of client and
connection"? Can you give an example?

-Val

On Thu, Jul 8, 2021 at 12:53 PM Ivan Daschinsky  wrote:

> I'm sorry, but why we didn't consider to separate creation of Client and
> connection? Why not to make async variant of connection? See for example
> [1]
> [1] --- https://lettuce.io/core/release/api/index.html
>
>
> чт, 8 июл. 2021 г., 09:50 Pavel Tupitsyn :
>
> > Val,
> >
> > So the plan is:
> >
> > - Remove Ignition#start from the public API
> > - Make Ignition a class, not an interface
> > - Add static Ignition#startClient
> >
> > Sounds good?
> >
> > On Thu, Jul 8, 2021 at 6:13 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Hi Ivan,
> > >
> > > Ignition IS the entry point to Ignite, so I'm not sure I got your point
> > :)
> > > Where is the contradiction?
> > >
> > > Either way, please feel free to give your suggestions for an
> alternative
> > > name if you have any.
> > >
> > > -Val
> > >
> > > On Wed, Jul 7, 2021 at 7:56 PM Ivan Pavlukhina 
> > > wrote:
> > >
> > > > A side note. Actually “Ignition” naming always confused me. I think
> > about
> > > > it as some fancy named API entry point for Ignite. Perhaps it is a
> good
> > > > moment to revisit naming.
> > > >
> > > > > On 8 Jul 2021, at 07:09, Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > > >
> > > > > Hi Pavel,
> > > > >
> > > > > I don't think we will need the pure embedded mode, but we still
> need
> > to
> > > > be
> > > > > able to access the API from compute and services. That said, there
> > are
> > > > two
> > > > > usages of the 'Ignite' API:
> > > > >
> > > > >   1. Remote, via the binary protocol.
> > > > >   2. Local - needed for compute and services. (This is how it works
> > > now.)
> > > > >
> > > > > I believe that the API should be the same, and there should be a
> > > unified
> > > > > access point. Ignition seems to be a good candidate for this.
> > > > >
> > > > > Ignition#start should eventually be removed from the public API. It
> > is
> > > > > currently there only because we don't have the thin client yet.
> > > > >
> > > > > -Val
> > > > >
> > > > >> On Wed, Jul 7, 2021 at 5:47 AM Pavel Tupitsyn <
> ptupit...@apache.org
> > >
> > > > wrote:
> > > > >>
> > > > >> Igniters,
> > > > >>
> > > > >> I have a few questions regarding server node startup and thin
> > clients.
> > > > >>
> > > > >> State of things:
> > > > >> - Server nodes will be started with 'ignite run' from CLI [1]
> > > > >> - ignite-api module represents our public API
> > > > >> - ignite-api has Ignition interface to start server nodes
> > > > >>
> > > > >> Questions:
> > > > >> - What's the idea behind Ignition interface in the public API? Are
> > we
> > > > going
> > > > >> to have an "embedded mode" where servers can be started from
> code? I
> > > > >> thought this was not planned.
> > > > >> - How are users supposed to retrieve an instance of the Ignition
> > > > interface?
> > > > >> - Are there any plans to start thin clients from Ignition
> interface,
> > > or
> > > > >> should we have a separate way of doing this?
> > > > >>
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158873958
> > > > >>
> > > >
> > >
> >
>

[ANNOUNCE] Apache Ignite 3.0.0-alpha2 is released!

[Valentin Kulichenko]

Igniters,

I'm happy to announce that Ignite 3 project reached a significant
milestone, as we release the 2nd alpha version of the product.

On top of the functionality that was previously released, Alpha 2 adds
the following major features:
- Replication infrastructure based on Raft.
- New in-memory atomic storage with the basic insert-read functionality.
- New schema management engine and API.

There is an ability to create a cluster, and use the new Table to API to
write and read the data. Basic code examples are available here:
https://github.com/apache/ignite-3/tree/3.0.0-alpha2/examples

The best way to try the release out is to go through the Getting Started
Guide: https://ignite.apache.org/docs/3.0.0-alpha

If there are any questions, issues, or thoughts, please do not hesitate to
reply to this email. Ignite Community is welcoming any feedback and will
consider it in future development.

-Val

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Valentin Kulichenko]

Pavel,

Thanks for your response, makes sense.

Regarding the values() method: I would instead add the required methods to
the Tuple itself. E.g., it can implement Iterable, and additionally have
the value(int index) method, plus anything else that we might need.
I don't like returning a collection, because in Java it's a much wider API.
We will end up returning an implementation that throws
UnsupportedOperationException for most of the methods. I know this approach
is not uncommon in the Java world, but this doesn't make it good.
In .NET and other languages, we can use other abstractions, of course.
Every platform has its own specifics and practices, so APIs don't have to
be identical.

-Val

On Wed, Jun 30, 2021 at 7:44 AM Pavel Tupitsyn  wrote:

> Hi Andrey,
>
> > This will force us to bother about interfaces/contracts and compatibility
> > aspects in the future
>
> Schemas and versions are part of thin client wire protocol.
> This protocol is a public API - we'll have to care about compatibility
> anyway.
>
> Schema evolution is an important feature that users should understand.
> I see no reason to hide it from the API.
>
>
> > We already have a ticket [1]...
> > Will 'idx -> column' mapping only be enough for your purposes?
>
> The ticket sounds reasonable, but there are no API details.
> At the very least we need public access to column names, types, and
> indexes.
> I propose something like this:
>
> interface Tuple {
> TupleSchema getSchema();
> }
>
> class TupleSchema {
> int getVersion();
> List getColumns();
> }
>
> class TupleSchemaColumn {
> int index;
> String name;
> String typeName;
> bool isKey;
> }
>
> On Wed, Jun 30, 2021 at 11:05 AM Andrey Mashenkov <
> andrey.mashen...@gmail.com> wrote:
>
> > Hi Pavel,
> >
> > 2. Schema and its version are internal things and shouldn't be exposed,
> > otherwise, eventually, it will lead to the user will manage schemas
> > manually on his side for some purposes.
> > This will force us to bother about interfaces/contracts and compatibility
> > aspects in the future with uncertain benefits for a user.
> >
> > As SchemaDescriptor is an internal API and the user expects a public API.
> > I don't think we want to convert the descriptor back into public API
> terms
> > and it may be impossible for some data.
> >
> > We already have a ticket [1] to support accessing a column by an index
> and
> > exposing some metadata.
> > Will 'idx -> column' mapping only be enough for your purposes?
> >
> > > int Tuple.columnIndex(String)
> >
> >
> >
> >  [1] https://issues.apache.org/jira/browse/IGNITE-14342
> >
> > On Wed, Jun 30, 2021 at 9:34 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Hi Val,
> > >
> > > Thanks for the comments, please see below:
> > >
> > >
> > > > Users will identify tables using names, they will never use IDs
> > >
> > > Ok, let's keep it this way.
> > >
> > >
> > > > Sounds like the Tuple should implement Iterable.
> > >
> > > I don't think Iterable is enough.
> > > We should have a way to get values by column index: Tuple.value(int
> > index),
> > > where index is according to column order in schema.
> > >
> > > Combined with Tuple.schema(), it will provide an efficient way to
> consume
> > > data -
> > > users will be able to read columns in any order, skip some of them,
> etc.
> > >
> > > This is a common pattern in APIs like ODBC or System.Data [1],
> > > which we'll need to implement on top of our thin clients later.
> > >
> > >
> > > > However, whether a user might
> > > > need to get a table schema for a particular version, I'm not sure. Do
> > you
> > > > have a use case in mind for this? If not, I would keep this internal
> > >
> > > Ok, we can keep the Table.schema(ver) method internal, as long as
> > > Table.schemas() is public and includes schema versions.
> > >
> > >
> > > > We already have async counterparts for all the methods where this is
> > > applicable
> > >
> > > IgniteTables.createTable(), dropTable(), tables(), table() do not have
> > > async counterparts,
> > > while internally they perform blocking wait on some futures.
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.data.idatareader?view=net-5.0
> > >

Re: IP Filtering in IPFinders

[Valentin Kulichenko]

Hi Atri,

I've noticed that you added the property to the IgniteConfiguration, but
it's applied only within the discovery. I feel like something is wrong here.

If this feature only relates to the discovery, then we should have the
configuration property on the TcpDiscoverySpi instead. Otherwise, the
filtering should be applied to all network components (although I'm not
necessarily sure what that would imply).

What do you think?

-Val

On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma  wrote:

> Hi Val and Ilya,
>
> Thank you for taking the time to pursue this issue.
>
> I have raised a new PR for the discussed approach. Please see and let
> me know what you think:
>
> https://github.com/apache/ignite/pull/9048
>
> Regards,
>
> Atri
>
> On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
>  wrote:
> >
> > Hello!
> >
> > I'm still not fully convinced, but Val's approach sounds rational to me.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 22 апр. 2021 г. в 12:45, Atri Sharma :
> >
> > > Hello!
> > >
> > > I actually saw the shared container scenario being tried by somebody
> > > who wanted an external script to monitor all IPs being used by his
> > > clusters and hence thought of this idea. Another thing that came in
> > > was the Firewall blocking a few IP addresses, hence the idea.
> > >
> > > I feel that the footprint of this change is small, and can be useful
> > > for esoteric use cases too without really interfering in any existing
> > > code path. Val's suggestion seems the right way to go since it gives
> > > the functionality without much change.
> > >
> > > Thoughts?
> > >
> > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > >  wrote:
> > > >
> > > > Hello!
> > > >
> > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > > lightweight entity, similar to a table in an SQL database. Why would
> > > > different clusters need to share the same discovery storage
> container?
> > > > When I tested Azure IP finder, it created several blob containers
> for me
> > > on
> > > > demand, based on the parameter passed to IP finder. If I wanted to
> have
> > > > more than one cluster it should have been seamless already.
> > > >
> > > > I can theoretically see how address filtering may be useful to remove
> > > > public / private addresses or Docker gateway address, but it is
> usually
> > > > handled by setting localHost setting, although requiring tuning it
> for
> > > each
> > > > instance individually. Overall benefit seems to small.
> > > >
> > > > This is why I am asking, do you have any specific scenario in mind
> where
> > > > this feature is an enabler? How did you arrive at the conclusion to
> go
> > > > forward with it?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma :
> > > >
> > > > > Hi Val,
> > > > >
> > > > > Consider a scenario where multiple Ignite clusters are running and
> for
> > > > > operational ease (and also compliance, in some cases, e.g. to make
> > > > > auditing easier), people can configure cloud based IP finders to
> share
> > > > > the same container (blob container in Azure, S3 container in AWS
> etc).
> > > > >
> > > > > In such a case, IPs for all clusters will be in the same container.
> > > > > IPFinders of both the clusters will read the entire list. In this
> > > > > case, address filtering will help ignore the irrelevant IP
> addresses.
> > > > >
> > > > > Thank you for pointing me to the alternate direction. Let me
> research
> > > > > that and revert.
> > > > >
> > > > > Atri
> > > > >
> > > > > On Wed, Apr 21, 2021 at 10:46 PM Valentin Kulichenko
> > > > >  wrote:
> > > > > >
> > > > > > Hi Atri,
> > > > > >
> > > > > > Can you describe the scenario in a little more detail? What
> exactly
> > > do
> > > > > you
> > > > > > mean by a container shared by multiple clusters? What are the
> > > > > consequences
> > > > > > of this? How do

Re: [DISCUSSION] Array to BinaryObject serialization

[Valentin Kulichenko]

Hi Nikolay,

Is there a specific motivation behind your proposal? I do acknowledge that
the semantics of the toBinary method is a little weird, but my concern is
that the way it works with arrays is just an example. Are you suggesting
changing collections, primitives, and other "first citizen" data types as
well? To me, that looks like a huge risky change without a clear value.

I actually believe that binary format *should not* be used as a universal
serde mechanism. It was designed as a data storage format, and the fact
that Ignite uses it elsewhere is a fundamental design flaw. BTW, this is
something we're trying to fix in 3.0.

That said, I'm not necessarily against the change (especially if we do not
change the default behavior). But I would like to better understand its
scope (e.g., arrays only or beyond?), as well as get some examples of use
cases that would benefit from the change.

-Val


On Fri, Apr 30, 2021 at 7:35 AM Nikolay Izhikov  wrote:

> Hello, Ilya.
>
> Thanks for the feedback!
>
> > For me it sounds like something we would like to do in 3.0
>
> Ignite 3 is a very long way to go, so I prefer to target this fix in
> Ignite 2.x.
>
> > I think making it default "true" is a breaking change and is not
> possible in a minor release
>
> Yes, you are correct it is a breaking change.
> It seems for me, we all agreed that breaking changes are possible in minor
> releases.
>
> Anyway, if we will decide do not to enable this feature by default it’s OK
> for me.
> We still can implement it and improve the binary SerDe mechanism.
>
>
> > 30 апр. 2021 г., в 17:23, Ilya Kasnacheev 
> написал(а):
> >
> > Hello!
> >
> > For me it sounds like something we would like to do in 3.0 (if indeed it
> > will have arrays as possible value (or key) type), but doing it in 2.x
> > raises concerns whether it has enough time left to stabilize.
> >
> > Also, I think making it default "true" is a breaking change and is not
> > possible in a minor release, case in point,
> > IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is less
> > destructive.
> >
> > Of course I would also like to hear what other community members think.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
> >
> >> Igniters,
> >>
> >> Want to clarify my proposal about new array store format.
> >> I think we should store array in special binary wrapper that will keep
> >> original component type
> >>
> >> ```
> >> public class BinaryArrayWrapper implements BinaryObjectEx,
> Externalizable {
> >>/** Type ID. */
> >>private int compTypeId;
> >>
> >>/** Raw data. */
> >>private String compClsName;
> >>
> >>/** Value. */
> >>private Object[] arr;
> >>
> >>// Further implementation.
> >> }
> >> ```
> >>
> >>
> >>> 30 апр. 2021 г., в 16:31, Nikolay Izhikov 
> >> написал(а):
> >>>
> >>> Hello, Igniters.
> >>>
> >>> Currently, binary marshaller works as follows(Say, we have a class
> >> `User` then):
> >>>
> >>> IgniteBinary#toBinary(User)` -> BinaryObject
> >>> IgniteBinary#toBinary(User[])` -> Object[]
> >>> IgniteBinary#toBinary(Object[])` -> Object[]
> >>>
> >>> This means, that we lose array component type information during binary
> >> serialization.
> >>> AFAIK, it’s a design choice made during binary infrastructure
> >> development.
> >>>
> >>> This lead to the following disadvantages:
> >>>
> >>> 1. `IgniteBinary` can’t be used as a universal SerDe mechanism.
> >>> 2. Ignite internals(service grid, .Net calls) contains many tweaks and
> >> hacks to deal with custom user array and still has many issues [1]
> >>>
> >>> I propose to make breaking changes and fix the custom user array SeDe
> as
> >> follows:
> >>>
> >>>  1. Implement binary serialization that correctly Ser and Deser
> >> array using some kind of the wrapper (BinaryArrayWrapper).
> >>>
> >>>  IgniteBinary#toBinary(User)` -> BinaryObject
> >>>  IgniteBinary#toBinary(User[])` -> BinaryObject
> >>>  IgniteBinary#toBinary(Object[])` -> BinaryObject
> >>>
> >>>  2. Introduce system flag `IGNITE_USE_BINARY_ARRAY` that enables
> >> correct SerDe of arrays. The default value is false to keep backward
> >> compatibility in the next Ignite release(2.11).
> >>>
> >>>  3. Set  `IGNITE_USE_BINARY_ARRAY` to `true` in the ongoing Ignite
> >> releases (2.12).
> >>>
> >>> WDYT?
> >>>
> >>> [1] https://issues.apache.org/jira/browse/IGNITE-14299
> >>
> >>
>
>

Re: [DISCUSSION] Array to BinaryObject serialization

[Valentin Kulichenko]

Hi Ivan,

Thanks for chiming in. My comments are below.

-Val

On Sat, May 1, 2021 at 12:20 AM Ivan Daschinsky  wrote:

> Hi!
> First of all, when array is serialized, marshaller actually DO
> PRESERVE type of element (seel
> org.apache.ignite.internal.binary.BinaryUtils#doReadObjectArray and
> org.apache.ignite.internal.binary.BinaryWriterExImpl#doWriteObjectArray).
> AFAIK, the motivation of Nickolay proposal, is the fact, that during
> call of PlatformService we do additional marshal/unmarshall step and
> during this step we loose array type info
> See org.apache.ignite.internal.binary.BinaryReaderExImpl#readObjectDetached
> and org.apache.ignite.internal.binary.BinaryUtils#doReadObjectArray)
>
> In order to handle this problem, we can just add some wrapper that
> contains element type info and use it in our INTERNAL API.
> I just don't understand why we should change IgniteBinary API.
>

Makes sense to me. I would also avoid changing the public API.


>
> >>  It was designed as a data storage format, and the fact
> that Ignite uses it elsewhere is a fundamental design flaw. BTW, this is
> something we're trying to fix in 3.0.
>
> Please, do not. There are many cases when this can really improve
> performance. Reflection is always slower than low level api and many
> users are happy with low level API.
>

Low-level APIs are not being removed. If anything, they are likely to
become more low-level :) Either way, that's off-topic. I would recommend
reviewing the related IEPs and starting a separate discussion if you have
any questions or concerns.


>
> сб, 1 мая 2021 г. в 00:51, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> >
> > Hi Nikolay,
> >
> > Is there a specific motivation behind your proposal? I do acknowledge
> that
> > the semantics of the toBinary method is a little weird, but my concern is
> > that the way it works with arrays is just an example. Are you suggesting
> > changing collections, primitives, and other "first citizen" data types as
> > well? To me, that looks like a huge risky change without a clear value.
> >
> > I actually believe that binary format *should not* be used as a universal
> > serde mechanism. It was designed as a data storage format, and the fact
> > that Ignite uses it elsewhere is a fundamental design flaw. BTW, this is
> > something we're trying to fix in 3.0.
> >
> > That said, I'm not necessarily against the change (especially if we do
> not
> > change the default behavior). But I would like to better understand its
> > scope (e.g., arrays only or beyond?), as well as get some examples of use
> > cases that would benefit from the change.
> >
> > -Val
> >
> >
> > On Fri, Apr 30, 2021 at 7:35 AM Nikolay Izhikov 
> wrote:
> >
> > > Hello, Ilya.
> > >
> > > Thanks for the feedback!
> > >
> > > > For me it sounds like something we would like to do in 3.0
> > >
> > > Ignite 3 is a very long way to go, so I prefer to target this fix in
> > > Ignite 2.x.
> > >
> > > > I think making it default "true" is a breaking change and is not
> > > possible in a minor release
> > >
> > > Yes, you are correct it is a breaking change.
> > > It seems for me, we all agreed that breaking changes are possible in
> minor
> > > releases.
> > >
> > > Anyway, if we will decide do not to enable this feature by default
> it’s OK
> > > for me.
> > > We still can implement it and improve the binary SerDe mechanism.
> > >
> > >
> > > > 30 апр. 2021 г., в 17:23, Ilya Kasnacheev  >
> > > написал(а):
> > > >
> > > > Hello!
> > > >
> > > > For me it sounds like something we would like to do in 3.0 (if
> indeed it
> > > > will have arrays as possible value (or key) type), but doing it in
> 2.x
> > > > raises concerns whether it has enough time left to stabilize.
> > > >
> > > > Also, I think making it default "true" is a breaking change and is
> not
> > > > possible in a minor release, case in point,
> > > > IGNITE_BINARY_SORT_OBJECT_FIELDS is still waiting for 3.0 and it is
> less
> > > > destructive.
> > > >
> > > > Of course I would also like to hear what other community members
> think.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 30 апр. 2021 г. в 17:16, Nikolay Izhikov :
> > > >
>

Re: IP Filtering in IPFinders

[Valentin Kulichenko]

I'm OK with the design.

Ilya, please feel free to merge if the implementation and tests look good
to you.

-Val

On Wed, Apr 28, 2021 at 1:07 AM Atri Sharma  wrote:

> Hi Ilya and Val,
>
> Thank you for taking a look and providing insights. I have updated the
> PR and raised another iteration.
>
> Val, I have moved the configuration to TcpDiscoverySpi.
>
> Please see and let me know your thoughts and comments.
>
> Regards,
>
> Atri
>
> On Wed, Apr 28, 2021 at 2:11 AM Valentin Kulichenko
>  wrote:
> >
> > Hi Atri,
> >
> > I've noticed that you added the property to the IgniteConfiguration, but
> > it's applied only within the discovery. I feel like something is wrong
> here.
> >
> > If this feature only relates to the discovery, then we should have the
> > configuration property on the TcpDiscoverySpi instead. Otherwise, the
> > filtering should be applied to all network components (although I'm not
> > necessarily sure what that would imply).
> >
> > What do you think?
> >
> > -Val
> >
> > On Tue, Apr 27, 2021 at 2:00 AM Atri Sharma  wrote:
> >
> > > Hi Val and Ilya,
> > >
> > > Thank you for taking the time to pursue this issue.
> > >
> > > I have raised a new PR for the discussed approach. Please see and let
> > > me know what you think:
> > >
> > > https://github.com/apache/ignite/pull/9048
> > >
> > > Regards,
> > >
> > > Atri
> > >
> > > On Thu, Apr 22, 2021 at 3:34 PM Ilya Kasnacheev
> > >  wrote:
> > > >
> > > > Hello!
> > > >
> > > > I'm still not fully convinced, but Val's approach sounds rational to
> me.
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > чт, 22 апр. 2021 г. в 12:45, Atri Sharma :
> > > >
> > > > > Hello!
> > > > >
> > > > > I actually saw the shared container scenario being tried by
> somebody
> > > > > who wanted an external script to monitor all IPs being used by his
> > > > > clusters and hence thought of this idea. Another thing that came in
> > > > > was the Firewall blocking a few IP addresses, hence the idea.
> > > > >
> > > > > I feel that the footprint of this change is small, and can be
> useful
> > > > > for esoteric use cases too without really interfering in any
> existing
> > > > > code path. Val's suggestion seems the right way to go since it
> gives
> > > > > the functionality without much change.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > On Thu, Apr 22, 2021 at 2:47 PM Ilya Kasnacheev
> > > > >  wrote:
> > > > > >
> > > > > > Hello!
> > > > > >
> > > > > > AFAIK, a S3 container, Azure blob container, etc, is a relatively
> > > > > > lightweight entity, similar to a table in an SQL database. Why
> would
> > > > > > different clusters need to share the same discovery storage
> > > container?
> > > > > > When I tested Azure IP finder, it created several blob containers
> > > for me
> > > > > on
> > > > > > demand, based on the parameter passed to IP finder. If I wanted
> to
> > > have
> > > > > > more than one cluster it should have been seamless already.
> > > > > >
> > > > > > I can theoretically see how address filtering may be useful to
> remove
> > > > > > public / private addresses or Docker gateway address, but it is
> > > usually
> > > > > > handled by setting localHost setting, although requiring tuning
> it
> > > for
> > > > > each
> > > > > > instance individually. Overall benefit seems to small.
> > > > > >
> > > > > > This is why I am asking, do you have any specific scenario in
> mind
> > > where
> > > > > > this feature is an enabler? How did you arrive at the conclusion
> to
> > > go
> > > > > > forward with it?
> > > > > >
> > > > > > Regards,
> > > > > > --
> > > > > > Ilya Kasnacheev
> > > > > >
> > > > > >
> > > > > > чт, 22 апр. 2021 г. в 07:51, Atri Sharma :
> > > > &g

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Valentin Kulichenko]

Andrey,

I see. So in a nutshell, you're saying that we want to return a future that
the user's code is not allowed to complete. In this case, I think it's
clear that CompletableFuture is not what we need. We actually need a
NonCompletableFuture :)

My vote is for the custom interface.

-Val

On Fri, Mar 26, 2021 at 2:25 AM Andrey Mashenkov 
wrote:

> Val,
>
> The methods below shouldn't be accessible for user:
> complete()
> completeExceptionaly()
>
> Returning CompletableFuture we must always make a copy to prevent the
> original future from being completed by mistake.
> I think it will NOT be enough to do that returing the future to the
> end-user, but from every critical module to the outside of the module,
> e.g. to plugins. The impact of disclosing ExchangeFuture,
> PartitionReleaseFuture to plugins may be serious.
>
> IgniteFuture extends Future, CompletionStage which implementation
> will just wrap CompletableFuture these issues will be resolved in natural
> way.
> In addition we can force toCompletableFuture() method to return a defensive
> copy(), that resolves the last concern.
>
>
> On Fri, Mar 26, 2021 at 11:38 AM Konstantin Orlov 
> wrote:
>
> > CompletableFuture seems a better option to me.
> >
> > --
> > Regards,
> > Konstantin Orlov
> >
> >
> >
> >
> > > On 26 Mar 2021, at 11:07, Pavel Tupitsyn  wrote:
> > >
> > > On the one hand, I agree with Alexey.
> > > CompletableFuture has complete* methods which should not be available
> to
> > > the user code.
> > > This can be solved with a simple interface like we do in Thin Client:
> > > IgniteClientFuture extends Future, CompletionStage
> > >
> > >
> > > On the other hand:
> > > - CompletionStage has toCompletableFuture anyway (rather weird)
> > > - Other libraries use CompletableFuture and it seems to be fine
> > > - Using CompletableFuture is the simplest approach
> > >
> > >
> > > So I lean towards CompletableFuture too.
> > >
> > > On Fri, Mar 26, 2021 at 10:46 AM Alexey Kukushkin <
> > kukushkinale...@gmail.com>
> > > wrote:
> > >
> > >> I do not like Java's CompletableFuture and prefer our own Future
> > (revised
> > >> IgniteFuture).
> > >>
> > >> My understanding of the Future (or Promise) pattern in general is
> having
> > >> two separate APIs:
> > >>
> > >>   1. Server-side: create, set result, raise error, cancel from server.
> > >>   2. Client-side: get result, handle error, cancel from client
> > >>
> > >> Java's CompletableFuture looks like both the client-side and
> > >> server-side API. The "Completeable" prefix in the name is already
> > confusing
> > >> for a client since it cannot "complete" an operation, only a server
> can.
> > >>
> > >> I would create our own IgniteFuture adding client-side functionality
> we
> > >> currently miss (like client-side cancellation).
> > >>
> > >>
> > >> пт, 26 мар. 2021 г. в 01:08, Valentin Kulichenko <
> > >> valentin.kuliche...@gmail.com>:
> > >>
> > >>> Andrey,
> > >>>
> > >>> Can you compile a full list of these risky methods, and elaborate on
> > what
> > >>> the risks are?
> > >>>
> > >>> Generally, CompletableFuture is a much better option, because it's
> > >>> standard. But we need to make sure it actually fits our needs and
> > doesn't
> > >>> do more harm than good.
> > >>>
> > >>> -Val
> > >>>
> > >>> On Thu, Mar 25, 2021 at 12:23 PM Alexei Scherbakov <
> > >>> alexey.scherbak...@gmail.com> wrote:
> > >>>
> > >>>> I think both options are fine, but personally lean toward
> > >>>> CompletableFuture.
> > >>>>
> > >>>> чт, 25 мар. 2021 г. в 17:56, Atri Sharma :
> > >>>>
> > >>>>> I would suggest using CompletableFuture -- I don't see a need for a
> > >>>> custom
> > >>>>> interface that is unique to us.
> > >>>>>
> > >>>>> It also allows a lower barrier for new contributors for
> understanding
> > >>>>> existing code
> > >>>>>
> > >>>>> On Thu, 25 Mar 2021, 20:18 Andrey Mashenkov, <
> > >>> an

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Valentin Kulichenko]

Andrey,

Can you compile a full list of these risky methods, and elaborate on what
the risks are?

Generally, CompletableFuture is a much better option, because it's
standard. But we need to make sure it actually fits our needs and doesn't
do more harm than good.

-Val

On Thu, Mar 25, 2021 at 12:23 PM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> I think both options are fine, but personally lean toward
> CompletableFuture.
>
> чт, 25 мар. 2021 г. в 17:56, Atri Sharma :
>
> > I would suggest using CompletableFuture -- I don't see a need for a
> custom
> > interface that is unique to us.
> >
> > It also allows a lower barrier for new contributors for understanding
> > existing code
> >
> > On Thu, 25 Mar 2021, 20:18 Andrey Mashenkov,  >
> > wrote:
> >
> > > Hi Igniters,
> > >
> > > I'd like to start a discussion about replacing our custom IgniteFuture
> > > class with CompletableFuture - existed JDK class
> > > or rework it's implementation (like some other products done) to a
> > > composition of CompletionStage and Future interfaces.
> > > or maybe other option if you have any ideas. Do you?
> > >
> > > 1. The first approach pros and cons are
> > > + Well-known JDK class
> > > + Already implemented
> > > - It is a class, not an interface.
> > > - Expose some potentially harmful methods like "complete()".
> > >
> > > On the other side, it has copy() method to create defensive copy and
> > > minimalCompletionStage() to restrict harmful method usage.
> > > Thus, this look like an applicable solution, but we should be careful
> > > exposing internal future to the outside.
> > >
> > > 2. The second approach is to implement our own interface like the next
> > one:
> > >
> > > interface IgniteFuture extends CompletableStage, Future {
> > > }
> > >
> > > Pros and cons are
> > > + Our interfaces/classes contracts will expose an interface rather than
> > > concrete implementation.
> > > + All methods are safe.
> > > - Some implementation is required.
> > > - CompletableStage has a method toCompletableFuture() and can be
> > converted
> > > to CompletableFuture. This should be supported.
> > >
> > > However, we still could wrap CompletableFuture and don't bother about
> > > creating a defensive copy.
> > >
> > >
> > > Other project experience:
> > > * Spotify uses CompletableFuture directly [1].
> > > * Redis goes the second approach [2]
> > > * Vertx explicitly extends CompletableFuture [3]. However, they have
> > custom
> > > future classes and a number of helpers that could be replaced with
> > > CompletableStage. Maybe it is just a legacy.'
> > >
> > > Any thoughts?
> > >
> > > [1]
> > >
> > >
> >
> https://spotify.github.io/completable-futures/apidocs/com/spotify/futures/ConcurrencyReducer.html
> > > [2]
> > >
> > >
> >
> https://lettuce.io/lettuce-4/release/api/com/lambdaworks/redis/RedisFuture.html
> > > [3]
> > >
> > >
> >
> https://javadoc.io/static/org.jspare.vertx/vertx-jspare/1.1.0-M03/org/jspare/vertx/concurrent/VertxCompletableFuture.html
> > > --
> > > Best regards,
> > > Andrey V. Mashenkov
> > >
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Valentin Kulichenko]

Ivan,

It's not really about the "harm", but more about "what should we do if this
method is called?". Imagine the following code:

CompletableFuture fut = cache.getAsync(key);
fut.complete("something");

What should happen in this case?

The point is that currently a future returned from any of Ignite's async
operations is supposed to be completed with a value only by Ignite itself,
not by the user. If we follow the same approach in Ignite 3, returning
CompletableFuture is surely wrong in my view.

At the same time, if we take a fundamentally different route with the async
APIs, this whole discussion might become irrelevant. For example, can you
elaborate on your thinking around the reactive API? Do you have any
specifics in mind?

-Val

On Sat, Mar 27, 2021 at 9:18 PM Ivan Pavlukhin  wrote:

> > The methods below shouldn't be accessible for user:
> > complete()
> > completeExceptionaly()
>
> Folks, in case of user-facing API, do you think there is a real harm
> in allowing a user to manually "complete" a future? I suppose a user
> employs some post-processing for future results and potentially wants
> to have control of these results as well. E.g. premature completion in
> case when a result is no longer needed is possible usage.
>
> Also I thinkg it might be a good time to ponder about Future/Promise
> APIs in general. Why such API is our choice? Can we choose e.g.
> Reactive API style instead?
>
> 2021-03-27 0:33 GMT+03:00, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
> > Andrey,
> >
> > I see. So in a nutshell, you're saying that we want to return a future
> that
> > the user's code is not allowed to complete. In this case, I think it's
> > clear that CompletableFuture is not what we need. We actually need a
> > NonCompletableFuture :)
> >
> > My vote is for the custom interface.
> >
> > -Val
> >
> > On Fri, Mar 26, 2021 at 2:25 AM Andrey Mashenkov
> > 
> > wrote:
> >
> >> Val,
> >>
> >> The methods below shouldn't be accessible for user:
> >> complete()
> >> completeExceptionaly()
> >>
> >> Returning CompletableFuture we must always make a copy to prevent the
> >> original future from being completed by mistake.
> >> I think it will NOT be enough to do that returing the future to the
> >> end-user, but from every critical module to the outside of the module,
> >> e.g. to plugins. The impact of disclosing ExchangeFuture,
> >> PartitionReleaseFuture to plugins may be serious.
> >>
> >> IgniteFuture extends Future, CompletionStage which
> >> implementation
> >> will just wrap CompletableFuture these issues will be resolved in
> natural
> >> way.
> >> In addition we can force toCompletableFuture() method to return a
> >> defensive
> >> copy(), that resolves the last concern.
> >>
> >>
> >> On Fri, Mar 26, 2021 at 11:38 AM Konstantin Orlov 
> >> wrote:
> >>
> >> > CompletableFuture seems a better option to me.
> >> >
> >> > --
> >> > Regards,
> >> > Konstantin Orlov
> >> >
> >> >
> >> >
> >> >
> >> > > On 26 Mar 2021, at 11:07, Pavel Tupitsyn 
> >> > > wrote:
> >> > >
> >> > > On the one hand, I agree with Alexey.
> >> > > CompletableFuture has complete* methods which should not be
> available
> >> to
> >> > > the user code.
> >> > > This can be solved with a simple interface like we do in Thin
> Client:
> >> > > IgniteClientFuture extends Future, CompletionStage
> >> > >
> >> > >
> >> > > On the other hand:
> >> > > - CompletionStage has toCompletableFuture anyway (rather weird)
> >> > > - Other libraries use CompletableFuture and it seems to be fine
> >> > > - Using CompletableFuture is the simplest approach
> >> > >
> >> > >
> >> > > So I lean towards CompletableFuture too.
> >> > >
> >> > > On Fri, Mar 26, 2021 at 10:46 AM Alexey Kukushkin <
> >> > kukushkinale...@gmail.com>
> >> > > wrote:
> >> > >
> >> > >> I do not like Java's CompletableFuture and prefer our own Future
> >> > (revised
> >> > >> IgniteFuture).
> >> > >>
> >> > >> My understanding of the Future (or Promise) pattern in general is
> >> having
> >> > >&

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Valentin Kulichenko]

Hi Andrey,

Please see below.

-Val

On Wed, Mar 31, 2021 at 7:55 AM Andrey Mashenkov 
wrote:

> CompletableFuture cancellation will not work as many users expected.
> Javadoc says:
> /* Since (unlike {@link FutureTask}) this class has no direct
> * control over the computation that causes it to be completed,
> * cancellation is treated as just another form of exceptional
> * completion.
> */
>

Let's not make assumptions about the expectations of the users. That's
exactly why I initially leaned towards a custom interface as well, but
that's a mistake.
Indeed, this contract might look weird to us, because the current version
of Ignite behaves differently. However, there are much more developers
using CompletableFuture and other standard async frameworks, than
developers using the async functionality of Ignite. Therefore, our
intuitions can easily be wrong.


> Completion of a child future doesn't trigger the completion of a parent.
> So, we will need to extend the future anyway.
>

First of all, as Pavel mentioned, you can attach your own listener before
returning a CompletableFuture to the user. You don't need to extend.
Second of all, there is still a discussion to be had on whether the parent
needs to be completed. I don't actually think it's obvious, and most likely
it's case by case. With CompletableFuture you have enough flexibility to
control the behavior.


>
> Also, cancel() method contract differs from IgniteFuture in 2.0,
> Completable method return true if the future was cancelled,
> but IgniteFuture return true only if it wasn't cancel prior the call.
> Thus, if cancel() was called twice we will have different results for
> CompletableFuture and IgniteFuture,
> that makes CompletableFuture barely usable for our internal purposes.
>

It doesn't really matter how IgniteFuture in 2.0 behaves. It was created
long before continuations and other async concepts became mainstream (in
Java world at least).
Also, we don't have to use it for internal purposes, of course. I'm only
talking about public APIs.


>
> BTW, CompletableFuture.anyOf() still can be used, see
> CompletionStage.toCompletableFuture() contract.
>

In my view, this actually kills the idea of a custom future. Basically,
the proposal is to introduce a custom entity to restrict access to some of
the CompletableFuture methods, but then allow to convert this custom entity
to a CompletableFuture that has all the methods. This makes zero sense to
me.


>
> The more I learn about CompletableFuture the stronger my opinion about
> overriding it.
>
>
> On Wed, Mar 31, 2021 at 9:31 AM Denis Garus  wrote:
>
> > > Stripping them from such functionality, which they are used to, is most
> > likely a bad idea.
> >
> > Completely agree with this point of view.
> > Moreover, a user can pass CompletableFuture to another library to do any
> > manipulations.
> > So if we want to introduce our class instead of the java class, we should
> > have solid arguments;
> > otherwise, it can be a reason for irritation.
> >
> > ср, 31 мар. 2021 г. в 09:06, Pavel Tupitsyn :
> >
> > > Val,
> > >
> > > > we can have an IgniteFuture that extends CompletableFuture.
> > > > This might be useful if want the cancel() operation to cancel the
> > > > underlying operation
> > >
> > > I think we can subscribe to the cancellation of the CompletableFuture
> > > and cancel the underlying operation without an extra class,
> > > something like
> > >
> > > fut.exceptionally(t -> {
> > > if (t instanceof CancellationException) {
> > > // Cancel Ignite operation
> > > }
> > > });
> > >
> > > On Wed, Mar 31, 2021 at 7:45 AM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > These are actually some interesting points. As I'm thinking more
> about
> > > > this, I'm leaning towards changing my opinion and voting for the
> > > > CompletableFuture. Here is my reasoning.
> > > >
> > > > First, it's important to keep in mind that CompletableFuture is not
> an
> > > > interface that we will implement, it's an implemented class.
> Therefore,
> > > > some of the concerns around complete() and cancel() method are not
> > really
> > > > relevant -- it's not up to us how these methods behave, they're
> already
> > > > implemented.
> > > >
> > > > Second, CompletableFuture does provide some useful functionality
> (anyOf
> > > is
> > > > one of the examples). I can even envision users wanting 

Re: [DISCUSSION] IgniteFuture class future in Ignite-3.0.

[Valentin Kulichenko]

; > backs Val's message).
> > > > > Let's say a user started a compute with fut =
> compute.runAsync(task);
> > > and
> > > > > now calls fut.complete(someVal); Does this mean that Ignite no
> longer
> > > > needs
> > > > > to execute the task? If the task is currently running, does it need
> > to
> > > be
> > > > > canceled?
> > > > >
> > > > > Using CompletableFuture.anyOf() is a good instrument in this case
> > > because
> > > > > it makes the 'first future wins' contract explicit in the code.
> > Besides
> > > > > that, the point regarding the cancel() method is valid, and we will
> > > need
> > > > > some custom mechanics to cancel a computation, so a custom
> interface
> > > that
> > > > > simply extends both Future and CompletableStage seems reasonable to
> > me.
> > > > >
> > > > > --AG
> > > > >
> > > > > пн, 29 мар. 2021 г. в 09:12, Ivan Pavlukhin :
> > > > >
> > > > > > Val,
> > > > > >
> > > > > > There were enough hype around Reactive programming past years. I
> > > > > > remind a lot of talks about RxJava. And I suppose it worth to
> > > consider
> > > > > > it. But it requires some time to study modern trends to make a
> > > choice.
> > > > > > So far I am not ready to facilitate Reactive API for Ignite 3.
> > > > > >
> > > > > > Regarding CompletableFuture.
> > > > > >
> > > > > > > The point is that currently a future returned from any of
> > Ignite's
> > > > > async
> > > > > > > operations is supposed to be completed with a value only by
> > Ignite
> > > > > > itself,
> > > > > > > not by the user. If we follow the same approach in Ignite 3,
> > > > returning
> > > > > > > CompletableFuture is surely wrong in my view.
> > > > > >
> > > > > > My first thoughts was similar. But later I thought what a user
> > would
> > > > > > like do with returned future. And one of cases I imagined was a
> > case
> > > > > > of alternative result. E.g. a user uses Ignite and another data
> > > source
> > > > > > in his application. He wants to use a value arrived faster. He
> > > > > > combines 2 futures like CompletableFuture.anyOf(...).
> Consequently
> > > > > > even if we prohibit CompletableFuture.complete(...) explicitly
> then
> > > it
> > > > > > will be possible to create a combination that will allow
> premature
> > > > > > future completion. After all generally CompletableFuture is a
> > > > > > placeholder for async computaion result and if a user wants to
> > > > > > substitute result returned from Ignite why should we disallow him
> > to
> > > > > > do it?
> > > > > >
> > > > > > Also I found one more suspicious thing with CompletableFuture. As
> > it
> > > > > > is a concrete class it implements a cancel() method. And as I see
> > the
> > > > > > implementation does not try to cancel underlying computations. Is
> > not
> > > > > > it a problem?
> > > > > >
> > > > > > 2021-03-29 7:30 GMT+03:00, Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com>:
> > > > > > > Ivan,
> > > > > > >
> > > > > > > It's not really about the "harm", but more about "what should
> we
> > do
> > > > if
> > > > > > this
> > > > > > > method is called?". Imagine the following code:
> > > > > > >
> > > > > > > CompletableFuture fut = cache.getAsync(key);
> > > > > > > fut.complete("something");
> > > > > > >
> > > > > > > What should happen in this case?
> > > > > > >
> > > > > > > The point is that currently a future returned from any of
> > Ignite's
> > > > > async
> > > > > > > operations is supposed to be completed with a value only by
> > Ignite
> > > > > > itself,
> > > > > > >

Re: [DISCUSSION] IEP-69 The evolutionary release process

[Valentin Kulichenko]

; >
> > чт, 11 мар. 2021 г. в 23:11, Maxim Muzafarov :
> >
> > > Val,
> > >
> > >
> > > I'm sorry if anything from what I've said sounded disrespectful. All
> > > of you are examples for me to follow :-)
> > >
> > > Have you checked the `motivation` [1] topic on the IEP-69 page? Should
> > > I add more details to it prior to the call? I want to make Ignite
> > > better and also think that the current 2.x version with all the
> > > advantages and disadvantages is far from exhausted its capabilities.
> > > I'm pretty sure the same motivation page exists for 3.0 version
> > > describing the advantages and disadvantages of developing mentioned
> > > IEPs. It will be good to share it prior to the cal also.
> > >
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-69%3A+The+evolutionary+release+process#IEP69:Theevolutionaryreleaseprocess-Motivation
> > >
> > > On Thu, 11 Mar 2021 at 01:21, Valentin Kulichenko
> > >  wrote:
> > > >
> > > > Ksenya, thanks for scheduling this so quickly!
> > > >
> > > > Guys, I hope we can make this discussion constructive. Please keep in
> > > mind
> > > > that Ignite 3 is an ongoing project supported by multiple
> contributors,
> > > > committers, and PMC members. Neglecting 6+ months of effort and
> > > suggesting
> > > > that it's just "prototyping some cool features and nothing more" is
> > > really
> > > > bizarre, and, quite frankly, sounds disrespectful to fellow
> developers
> > > > (although I'm 100% sure it was not intended this way).
> > > >
> > > > Maxim, one of the biggest issues I have with your IEP is that I don't
> > > > understand the motivation behind it. If you don't mind, I would like
> to
> > > > suggest that you kick off the meeting with a detailed explanation
> > > > of exactly that. The first step is to achieve a mutual understanding
> of
> > > > each other's goals. Once we do that, I'm sure we will easily find a
> > > > solution.
> > > >
> > > > -Val
> > > >
> > > > On Wed, Mar 10, 2021 at 8:55 AM Kseniya Romanova <
> > > romanova.ks@gmail.com>
> > > > wrote:
> > > >
> > > > > Let's make a quick call next week and try to find a compromise
> which
> > > can
> > > > > get the process moving:
> > > > >
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/276851588/
> > > > >
> > > > > ср, 10 мар. 2021 г. в 16:27, Maxim Muzafarov :
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > >
> > > > > > Agree, the discussion may be endless without compromises on all
> > > sides.
> > > > > > I always think that if there is no consensus (and I see from the
> > > > > > thread [1] that it's was no found) for such important decisions
> like
> > > > > > product future development and releases AFS provides the voting
> > > > > > procedure. Without fixing the results of the discussion [1] it
> sounds
> > > > > > like prototyping some cool features and nothing more.
> > > > > >
> > > > > > So, back to Denis suggestion can you share - what would be the
> best
> > > > > > time for all of us (considering different time zones) to have a
> call?
> > > > > >
> > > > > > I also think that we should start a vote about the future
> releases on
> > > > > > our Apache Ignite web-site and user-list, thus all who are using
> the
> > > > > > Apache Ignite may choose the best option they like.
> > > > > >
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html
> > > > > >
> > > > > > On Wed, 10 Mar 2021 at 03:57, Valentin Kulichenko
> > > > > >  wrote:
> > > > > > >
> > > > > > > Hi Maxim,
> > > > > > >
> > > > > > > I disagree with the suggestions. Several community members have
> > > already
> > > > > > > pointed out the discussion about Ignite 3.0 [1]. During

Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore

[Valentin Kulichenko]

Atri,

I've added my comments to the PR.

-Val

On Wed, Mar 10, 2021 at 2:44 AM Atri Sharma  wrote:

> I have just updated the PR:
>
> https://github.com/apache/ignite/pull/8820
>
> Please review.
>
> On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma  wrote:
> >
> > Hi Val,
> >
> > Thank you for taking the time on this one.
> >
> > The main reason as to why I chose that signature was because I felt it
> > gave a clear idea of the interface that the user should expect when
> > using the method. So essentially, the user is providing a callback
> > listener himself/herself and the API is just using that to tie the
> > lifecycle of holding the semaphore.
> >
> > However, I do see your point and feel that the current signature that
> > the PR has will limit the usability of the method and make users jump
> > through more hoops. I have changed the signature to accept Callable
> > returning T.
> >
> > Regards,
> >
> > Atri
> >
> > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko
> >  wrote:
> > >
> > > Hi Atri,
> > >
> > > First and foremost, we need to clarify the API for this functionality.
> > > There are two options presented in the ticket, but no clear decision
> about
> > > which one should be used. I personally think that the original
> signature
> > > (the one in the ticket description) makes more sense, but it looks like
> > > you've implemented an alternative suggested in the comments. What was
> your
> > > motivation behind that?
> > >
> > > As for where the method can be located, I'm OK if we add it directly
> to the
> > > IgniteSemaphore interface.
> > >
> > > -Val
> > >
> > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma  wrote:
> > >
> > > > Gentle ping
> > > >
> > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma,  wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I would like to start a discussion around IGNITE-2399.
> > > > >
> > > > > Background: The typical use of IgniteSemaphore consists of
> acquiring
> > > > > the semaphore, performing the task and releasing the semaphore.
> This
> > > > > JIRA proposes a new method which will accept a callable, acquire
> the
> > > > > semaphore, and return a future. Upon completion of the future, the
> > > > > semaphore is released.
> > > > >
> > > > > This API seems useful for an easy encapsulation of the described
> use
> > > > > case and does not cause an internal flux since all the changes are
> > > > > focused at the public API.
> > > > >
> > > > > WIP PR for the same:
> > > > >
> > > > > https://github.com/apache/ignite/pull/8820
> > > > >
> > > > > Please share your thoughts and comments.
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > > >
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
>
>
>
> --
> Regards,
>
> Atri
> Apache Concerted
>

Re: [DISCUSSION] IEP-69 The evolutionary release process

[Valentin Kulichenko]

Ksenya, thanks for scheduling this so quickly!

Guys, I hope we can make this discussion constructive. Please keep in mind
that Ignite 3 is an ongoing project supported by multiple contributors,
committers, and PMC members. Neglecting 6+ months of effort and suggesting
that it's just "prototyping some cool features and nothing more" is really
bizarre, and, quite frankly, sounds disrespectful to fellow developers
(although I'm 100% sure it was not intended this way).

Maxim, one of the biggest issues I have with your IEP is that I don't
understand the motivation behind it. If you don't mind, I would like to
suggest that you kick off the meeting with a detailed explanation
of exactly that. The first step is to achieve a mutual understanding of
each other's goals. Once we do that, I'm sure we will easily find a
solution.

-Val

On Wed, Mar 10, 2021 at 8:55 AM Kseniya Romanova 
wrote:

> Let's make a quick call next week and try to find a compromise which can
> get the process moving:
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/276851588/
>
> ср, 10 мар. 2021 г. в 16:27, Maxim Muzafarov :
>
> > Folks,
> >
> >
> > Agree, the discussion may be endless without compromises on all sides.
> > I always think that if there is no consensus (and I see from the
> > thread [1] that it's was no found) for such important decisions like
> > product future development and releases AFS provides the voting
> > procedure. Without fixing the results of the discussion [1] it sounds
> > like prototyping some cool features and nothing more.
> >
> > So, back to Denis suggestion can you share - what would be the best
> > time for all of us (considering different time zones) to have a call?
> >
> > I also think that we should start a vote about the future releases on
> > our Apache Ignite web-site and user-list, thus all who are using the
> > Apache Ignite may choose the best option they like.
> >
> >
> > [1]
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html
> >
> > On Wed, 10 Mar 2021 at 03:57, Valentin Kulichenko
> >  wrote:
> > >
> > > Hi Maxim,
> > >
> > > I disagree with the suggestions. Several community members have already
> > > pointed out the discussion about Ignite 3.0 [1]. During that
> discussion,
> > we
> > > did agree on the scope of the changes for 3.0, as well as the general
> > > direction for the product. The new repo was created not to "develop
> from
> > > scratch", but to provide an opportunity for the community members to
> > > actively work on Ignite 3 without killing the Ignite 2.x. No
> alternative
> > > solution for this was presented, so we went ahead with the process -- I
> > > consider that to be an example of the silent consensus.
> > >
> > > I also want to emphasize that Ignite 3 is active and is moving forward.
> > If
> > > you look at the ignite-3 repo, commits and PRs are coming in on regular
> > > basis. We also had the first alpha release early in the year. I do
> agree
> > > with you, however, that there is not too much activity on the dev list.
> > As
> > > far as I can tell, the main reason for this is that communication moved
> > to
> > > IEPs and GitHub PRs, for better or worse. This is something we all can
> > talk
> > > about -- I personally would like to see more discussions on the dev
> list.
> > >
> > > And finally, I agree with Denis. This whole situation is
> > > counter-productive. I'm happy to jump on a Discord or any other voice
> > chat
> > > to discuss in more detail.
> > >
> > > [1]
> > >
> >
> http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html
> > >
> > > -Val
> > >
> > > On Fri, Mar 5, 2021 at 11:09 AM Maxim Muzafarov 
> > wrote:
> > >
> > > > Ignites,
> > > >
> > > >
> > > > I've created the IEP-69 [1] which describes the evolutionary release
> > > > process for the Apache Ignite 2.x version. You can find all the
> > > > details of my suggestion there, but here you can find the crucial
> > > > points:
> > > >
> > > > 0. Versioning - grand.major.bug-fix[-rc_number]
> > > >
> > > > 1. Prepare the next 3.0 release based on 2.x with some breaking
> > > > compatibility changes. The same things happen from time to time with
> > > > other Apache projects like Hadoop, Spark.
> &g

Re: IEP-70: Async Continuation Executor

[Valentin Kulichenko]

Hi Denis,

I think Pavel's main point is that behavior is unpredictable. For example,
AFAIK, putAsync can be executed in the main thread instead of the striped
pool thread if the operation is local. The listener can also be executed in
the main thread - this happens if the future is completed prior to listener
invocation (this is actually quite possible in the unit test environment
causing the test to pass). Finally, I'm pretty sure there are many cases
when a deadlock does not occur right away, but instead it will reveal
itself under high load due to thread starvation. The latter type of issues
is the most dangerous because they are often reproduced only in production.
Finally, there are performance considerations as well - cache operations
and listeners share the same fixed-sized pool which can have negative
effects.

I'm OK with the change. Although, it might be better to introduce a new
fixed-sized pool instead of ForkJoinPool for listeners, simply because this
is the approach taken throughout the project. But this is up to a debate.

-Val

On Wed, Mar 17, 2021 at 11:31 AM Denis Garus  wrote:

> Pavel,
> I tried this:
>
> @Test
> public void test() throws Exception {
> IgniteCache cache =
> startGrid().getOrCreateCache("test_cache");
>
> cache.putAsync(1, "one").listen(f -> cache.replace(1, "two"));
>
> assertEquals("two", cache.get(1));
> }
>
> and this test is green.
> I believe that an user can make listener that leads to deadlock, but
> the example in the IEP does not reflect this.
>
>
>
> ср, 17 мар. 2021 г. в 17:36, Вячеслав Коптилин :
>
> > Hi Pavel,
> >
> > > Not a good excuse really. We have a usability problem, you have to
> admit
> > it.
> > Fair enough. I agree that this is a usability issue, but I have doubts
> that
> > the proposed approach to overcome it is the best one.
> >
> > > Documentation won't help - no one is going to read the Javadoc for a
> > trivial method like putAsync
> > That is sad... However, I don't think that this is a strong argument
> here.
> >
> > This is just my opinion. Let's see what other community members have to
> > say.
> >
> > Thanks,
> > S.
> >
> >
> > ср, 17 мар. 2021 г. в 17:01, Pavel Tupitsyn :
> >
> > > > the user should use the right API
> > >
> > > Not a good excuse really. We have a usability problem, you have to
> admit
> > > it.
> > > "The brakes did not work on your car - too bad, you should have known
> > that
> > > on Sundays only your left foot is allowed on the pedal"
> > >
> > > This particular use case is too intricate.
> > > Even when you know about that, it is difficult to decide what can run
> on
> > > the striped pool,
> > > and what can't. It is too easy to forget.
> > > And most people don't know, even among Ignite developers.
> > >
> > > Documentation won't help - no one is going to read the Javadoc for a
> > > trivial method like putAsync.
> > >
> > >
> > > So I propose to have a safe default.
> > > Then document the performance tuning opportunity on [1].
> > >
> > > Think about how many users abandon a product because it mysteriously
> > > crashes and hangs.
> > >
> > > [1]
> > >
> > >
> >
> https://ignite.apache.org/docs/latest/perf-and-troubleshooting/general-perf-tips
> > >
> > >
> > > On Wed, Mar 17, 2021 at 4:21 PM Вячеслав Коптилин <
> > > slava.kopti...@gmail.com>
> > > wrote:
> > >
> > > > Hi Pavel,
> > > >
> > > > Well, I think that the user should use the right API instead of
> > > introducing
> > > > uncontested overhead for everyone.
> > > > For instance, the code that is provided by IEP can changed as
> follows:
> > > >
> > > > IgniteFuture fut = cache.putAsync(1, 1);
> > > > fut.listenAync(f -> {
> > > > // Executes on Striped pool and deadlocks.
> > > > cache.replace(1, 2);
> > > > }, ForkJoinPool.commonPool());
> > > >
> > > > Of course, it does not mean that this fact should not be properly
> > > > documented.
> > > > Perhaps, I am missing something.
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > ср, 17 мар. 2021 г. в 16:01, Pavel Tupitsyn :
> > > >
> > > > > Slava,
> > > > >
> > > > > Your suggestion is to keep things as is and discard the IEP, right?
> > > > >
> > > > > >  this can lead to significant overhead
> > > > > Yes, there is some overhead, but the cost of accidentally starving
> > the
> > > > > striped pool is worse,
> > > > > not to mention the deadlocks.
> > > > >
> > > > > I believe that we should favor correctness over performance in any
> > > case.
> > > > >
> > > > >
> > > > > On Wed, Mar 17, 2021 at 3:34 PM Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Well, the specified method already exists :)
> > > > > >
> > > > > > /**
> > > > > >  * Registers listener closure to be asynchronously notified
> > > > whenever
> > > > > > future completes.
> > > > > >  * Closure will be processed in specified executor.
> > > > > >  *
> > > > > >  * @param lsnr Listener closure to register. Cannot be {@code
> > > > null}.
> 

Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore

[Valentin Kulichenko]

Hi Atri,

Looks good to me, I've merged the changes. Please resolve the ticket.

-Val

On Wed, Mar 17, 2021 at 5:07 AM Atri Sharma  wrote:

> Hi Valentine,
>
> Hoping you are well.
>
> Please let me know if the PR looks ok.
>
> Regards,
>
> Atri
>
>
> On Thu, 11 Mar 2021, 12:09 Atri Sharma,  wrote:
>
> > Hi Val,
> >
> > Thanks for taking a look. I have updated the PR, please see and let me
> > know your thoughts and feedback.
> >
> > Regards,
> >
> > Atri
> >
> > On Thu, Mar 11, 2021 at 6:16 AM Valentin Kulichenko
> >  wrote:
> > >
> > > Atri,
> > >
> > > I've added my comments to the PR.
> > >
> > > -Val
> > >
> > > On Wed, Mar 10, 2021 at 2:44 AM Atri Sharma  wrote:
> > >
> > > > I have just updated the PR:
> > > >
> > > > https://github.com/apache/ignite/pull/8820
> > > >
> > > > Please review.
> > > >
> > > > On Wed, Mar 10, 2021 at 1:51 PM Atri Sharma  wrote:
> > > > >
> > > > > Hi Val,
> > > > >
> > > > > Thank you for taking the time on this one.
> > > > >
> > > > > The main reason as to why I chose that signature was because I felt
> > it
> > > > > gave a clear idea of the interface that the user should expect when
> > > > > using the method. So essentially, the user is providing a callback
> > > > > listener himself/herself and the API is just using that to tie the
> > > > > lifecycle of holding the semaphore.
> > > > >
> > > > > However, I do see your point and feel that the current signature
> that
> > > > > the PR has will limit the usability of the method and make users
> jump
> > > > > through more hoops. I have changed the signature to accept Callable
> > > > > returning T.
> > > > >
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > >
> > > > > On Wed, Mar 10, 2021 at 5:29 AM Valentin Kulichenko
> > > > >  wrote:
> > > > > >
> > > > > > Hi Atri,
> > > > > >
> > > > > > First and foremost, we need to clarify the API for this
> > functionality.
> > > > > > There are two options presented in the ticket, but no clear
> > decision
> > > > about
> > > > > > which one should be used. I personally think that the original
> > > > signature
> > > > > > (the one in the ticket description) makes more sense, but it
> looks
> > like
> > > > > > you've implemented an alternative suggested in the comments. What
> > was
> > > > your
> > > > > > motivation behind that?
> > > > > >
> > > > > > As for where the method can be located, I'm OK if we add it
> > directly
> > > > to the
> > > > > > IgniteSemaphore interface.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > > On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma 
> > wrote:
> > > > > >
> > > > > > > Gentle ping
> > > > > > >
> > > > > > > On Mon, 8 Mar 2021, 13:51 Atri Sharma, 
> wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I would like to start a discussion around IGNITE-2399.
> > > > > > > >
> > > > > > > > Background: The typical use of IgniteSemaphore consists of
> > > > acquiring
> > > > > > > > the semaphore, performing the task and releasing the
> semaphore.
> > > > This
> > > > > > > > JIRA proposes a new method which will accept a callable,
> > acquire
> > > > the
> > > > > > > > semaphore, and return a future. Upon completion of the
> future,
> > the
> > > > > > > > semaphore is released.
> > > > > > > >
> > > > > > > > This API seems useful for an easy encapsulation of the
> > described
> > > > use
> > > > > > > > case and does not cause an internal flux since all the
> changes
> > are
> > > > > > > > focused at the public API.
> > > > > > > >
> > > > > > > > WIP PR for the same:
> > > > > > > >
> > > > > > > > https://github.com/apache/ignite/pull/8820
> > > > > > > >
> > > > > > > > Please share your thoughts and comments.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Atri
> > > > > > > > Apache Concerted
> > > > > > > >
> > > > > > >
> > > > >
> > > > > --
> > > > > Regards,
> > > > >
> > > > > Atri
> > > > > Apache Concerted
> > > >
> > > >
> > > >
> > > > --
> > > > Regards,
> > > >
> > > > Atri
> > > > Apache Concerted
> > > >
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >
>

Re: IGNITE-2399 : Asynchronous Acquire method in IgniteSemaphore

[Valentin Kulichenko]

Hi Atri,

First and foremost, we need to clarify the API for this functionality.
There are two options presented in the ticket, but no clear decision about
which one should be used. I personally think that the original signature
(the one in the ticket description) makes more sense, but it looks like
you've implemented an alternative suggested in the comments. What was your
motivation behind that?

As for where the method can be located, I'm OK if we add it directly to the
IgniteSemaphore interface.

-Val

On Tue, Mar 9, 2021 at 8:22 AM Atri Sharma  wrote:

> Gentle ping
>
> On Mon, 8 Mar 2021, 13:51 Atri Sharma,  wrote:
>
> > Hi All,
> >
> > I would like to start a discussion around IGNITE-2399.
> >
> > Background: The typical use of IgniteSemaphore consists of acquiring
> > the semaphore, performing the task and releasing the semaphore. This
> > JIRA proposes a new method which will accept a callable, acquire the
> > semaphore, and return a future. Upon completion of the future, the
> > semaphore is released.
> >
> > This API seems useful for an easy encapsulation of the described use
> > case and does not cause an internal flux since all the changes are
> > focused at the public API.
> >
> > WIP PR for the same:
> >
> > https://github.com/apache/ignite/pull/8820
> >
> > Please share your thoughts and comments.
> >
> > --
> > Regards,
> >
> > Atri
> > Apache Concerted
> >
>

Re: [DISCUSSION] IEP-69 The evolutionary release process

[Valentin Kulichenko]

Hi Maxim,

I disagree with the suggestions. Several community members have already
pointed out the discussion about Ignite 3.0 [1]. During that discussion, we
did agree on the scope of the changes for 3.0, as well as the general
direction for the product. The new repo was created not to "develop from
scratch", but to provide an opportunity for the community members to
actively work on Ignite 3 without killing the Ignite 2.x. No alternative
solution for this was presented, so we went ahead with the process -- I
consider that to be an example of the silent consensus.

I also want to emphasize that Ignite 3 is active and is moving forward. If
you look at the ignite-3 repo, commits and PRs are coming in on regular
basis. We also had the first alpha release early in the year. I do agree
with you, however, that there is not too much activity on the dev list. As
far as I can tell, the main reason for this is that communication moved to
IEPs and GitHub PRs, for better or worse. This is something we all can talk
about -- I personally would like to see more discussions on the dev list.

And finally, I agree with Denis. This whole situation is
counter-productive. I'm happy to jump on a Discord or any other voice chat
to discuss in more detail.

[1]
http://apache-ignite-developers.2346864.n4.nabble.com/DISCUSS-Ignite-3-0-development-approach-td49922.html

-Val

On Fri, Mar 5, 2021 at 11:09 AM Maxim Muzafarov  wrote:

> Ignites,
>
>
> I've created the IEP-69 [1] which describes the evolutionary release
> process for the Apache Ignite 2.x version. You can find all the
> details of my suggestion there, but here you can find the crucial
> points:
>
> 0. Versioning - grand.major.bug-fix[-rc_number]
>
> 1. Prepare the next 3.0 release based on 2.x with some breaking
> compatibility changes. The same things happen from time to time with
> other Apache projects like Hadoop, Spark.
>
> 2. Discuss with the whole Community and assign the right release
> version to the activities related to the development of the new Ignite
> architecture (currently all the changes you can find in the ignite-3
> branch).
> I see no 3.0 discussions on the dev-list and I see no-activity with
> the 3.0 version currently. So,  it's better to remove the `lock` from
> the 3.0 version and allow the removal of obsolete features.
>
> 3. Guarantee the PDS compatibility between the `grand` versions of the
> Apache Ignite for the next year.
>
> 4. Guarantee the bug-fix release for the last 2.x Apache Ignite
> version for the next year.
>
> 5. Perform some improvements which break the backward compatibility,
> for instance: removing @deprecated API (except metrics), removing
> obsolete modules, changing the cluster defaults. You can find
> additional details on the IEP-69 page [1].
>
>
> Please, share your thoughts.
>
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-69%3A+The+evolutionary+release+process
>

Re: Review Request

[Valentin Kulichenko]

Atri,

I've added my comments in the PR.

-Val

On Wed, Mar 3, 2021 at 7:29 AM Denis Magda  wrote:

> @Valentin Kulichenko , @Nikolay Izhikov
> , @samvi...@yandex.ru ,
>
> I saw you reviewing the ticket. Could you please double-check the changes?
> "IGNITE-2399: Implement acquireAndExecute In IgniteSemaphore"
>
>
> Atri, please put a ticket number and name in the title of an email, so that
> community member can see right away what change is being contributed.
>
> -
> Denis
>
>
> On Wed, Mar 3, 2021 at 7:27 AM Atri Sharma  wrote:
>
> > Hi,
> >
> > Please help in reviewing:
> >
> > https://github.com/apache/ignite/pull/8820
> >
> > Atri
> >
>

Re: Apache Ignite 3.x: order in repository

[Valentin Kulichenko]

Petr,

Any suggestions on how to fix this? One of the ideas that come to my mind
is to unify the process. For example, by enforcing that all changes are
merged only through PRs. That, however, means that all contributors are
forced to work with GitHub, which is not necessarily a great thing.

Thoughts?

-Val

On Wed, Feb 24, 2021 at 12:45 AM Petr Ivanov  wrote:

> I'd guess that one of the main problem with inactive PRs is in creation of
> PR for reviewing but merging it from command line (not via GitHub
> interface).
> Also, of course, there are lots of efforts which are abandoned after first
> review, or even do not have a chance to be reviewed at all.
>
>
> > On 22 Feb 2021, at 11:51, Stephen Darlington <
> stephen.darling...@gridgain.com> wrote:
> >
> > I think we need to be able answer the question “Why are there so many
> inactive PRs?" before we automate their removal. If perfectly good changes
> are being ignored, we have a problem.
> >
> > Removing branches of merged PRs and protecting the main branch make
> sense.
> >
> >> On 20 Feb 2021, at 18:30, Pavel Tupitsyn  wrote:
> >>
> >> +1
> >>
> >> - Close inactive PRs (1 month or so?)
> >> - Enable main branch protection (no force pushes, require linear
> history,
> >> require status checks)
> >>
> >> On Sat, Feb 20, 2021 at 2:31 PM Petr Ivanov 
> wrote:
> >>
> >>> Hi, Igniters!
> >>>
> >>>
> >>> When we started Ignite 3.x in new repository, not only we have
> received a
> >>> chance to cleanup codebase, but to maintain some order in development
> >>> tools, like GitHub.
> >>> Currently in 2.x repository we have lots of stalled PRs and branches,
> >>> which not only clog the repository, but also indirectly influence TC
> >>> performance (due to necessity to check for updates every ref: branches
> and
> >>> PRs).
> >>>
> >>> Could I suggest we devise some recommendations for using PR's and
> branches
> >>> in new repo and add some rules about stalled PRs at least, like closing
> >>> them if inactive for some time.
> >>> Also we can activate some settings in repo's configuration, like auto
> >>> delete branch after PR is merged.
> >>>
> >>>
> >>> WDYT?
> >
> >
>
>

Re: Model of permissions for Ignite 3

[Valentin Kulichenko]

Hi folks,

Did you have a discussion? How did it go? Can someone summarize the results?

-Val

On Mon, Apr 12, 2021 at 2:00 AM Kseniya Romanova 
wrote:

> Hi! Scheduled open call for Friday
> https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/277518672/
> Please join to see the zoom link. Consulted with Denis Garus and put the
> topic as "Security", cause it's definitely wider than just permissions.
>
> Cheers!
>
> пн, 12 апр. 2021 г. в 09:25, Alexei Scherbakov <
> alexey.scherbak...@gmail.com
> >:
>
> > +1
> >
> > We should rethink the security model in Ignite 3 and have a default RBAC
> > based implementation, from my point of view.
> > By default, no code should be written to enable and use it.
> >
> > Let's schedule a meeting and discuss the details.
> >
> > вс, 11 апр. 2021 г. в 19:43, Denis Garus :
> >
> > > Andrey, Alexey thank you for the feedback!
> > >
> > > > I suggest decoupling authentication from authorization
> > >
> > > Yes, it would be useful. Existing SecuritySubject and SecurityContext
> > > require reworking.
> > >
> > > > I think it would be great to have a default implementation of
> > > user-role-permission model
> > >
> > > Completely agree it is a cool idea. Ignite should have more default
> > > implementation referred to security.
> > >
> > > > Should we have a community meeting to discuss this?
> > >
> > > Yes, it would be great.
> > > The wish list for Ignite 3 does not contain security improvement that,
> > > IMHO, is wrong.
> > > We should fix that oversight on early-stage Ignite 3 development.
> > >
> > > пт, 9 апр. 2021 г. в 18:47, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> > >:
> > >
> > > > Hello Denis, Andrey, Igniters,
> > > >
> > > > Why don't we take a step further in improving the security model in
> > > Ignite
> > > > 3? I think it would be great to have a default implementation of
> > > > user-role-permission model in Ignite to be on par with security
> models
> > of
> > > > widely-used databases. This will complement community efforts in
> making
> > > > most of the Ignite 3 behavior to be changeable in runtime.
> > > >
> > > > WDYT? Should we have a community meeting to discuss this?
> > > >
> > > >
> > > > чт, 8 апр. 2021 г. в 23:54, Andrey Kuznetsov :
> > > >
> > > > > Hi Denis!
> > > > >
> > > > > The idea and prototype look great.
> > > > >
> > > > > I'd like to highlight one arguable point. Default authorization
> > > > > implementation still assumes there are permissions provided in
> > > > > SecuritySubject. In turn, authentication is still responsible for
> > > filling
> > > > > these permissions. I suggest decoupling authentication from
> > > > authorization,
> > > > > so that GridSecurityProcessor implementation is fully responsible
> for
> > > > > obtaining permissions for SecuritySubject given on authorization.
> In
> > > > > particular, implementation can choose an existing behavior of
> > bundling
> > > > > permissions with SecuritySubject.
> > > > >
> > > > > Makes sense?
> > > > >
> > > > > чт, 8 апр. 2021 г. в 17:52, Denis Garus :
> > > > >
> > > > > > Sorry, I forgot to point the link
> > > > > >
> > > > > > 1. https://github.com/apache/ignite/pull/8989
> > > > > >
> > > > > > чт, 8 апр. 2021 г. в 17:50, Denis Garus :
> > > > > >
> > > > > > > Hello, Igniters!
> > > > > > >
> > > > > > > I want to propose to improve the way which we use
> > > > > > > to present permissions in Ignite 3.
> > > > > > >
> > > > > > > The model of permission in Ignite has a set of drawbacks.
> > > > > > > The main drawback, IMHO: if you need to add a new permission,
> > > > > > > you should change the core module by extended the
> > > > 'SecurityPermission'
> > > > > > > enum.
> > > > > > > An approach like this becomes more challenged if new permission
> > is
> > > > > > created
> > > > > > > for an extension.
> > > > > > >
> > > > > > > The existing permission model is overcomplicated.
> > > > > > > The SecurityPermission enum is divided into four groups,
> > > > > > > and to determine whether a security subject has been given
> > > > permission,
> > > > > > > a plugin developer has to know what the permission group is.
> > > > > > > But 'CACHE_CREATE' and 'CACHE_DESTROY' are included in two
> groups
> > > > > (system
> > > > > > > operations and cache operations).
> > > > > > > When 'CACHE_CREATE' ('CACHE_DESTROY') is treated as system
> > > > permission,
> > > > > > > it applies to all caches. In other cases, when 'CACHE_CREATE'
> > > > > > > ('CACHE_DESTROY') is treated as cache permission,
> > > > > > > permission checking is executed with the account of the cache
> > name.
> > > > > > > IMHO, this logic is hard to understand.
> > > > > > > There is no ability to represent compound operation as single
> > > > > permission
> > > > > > > and so on.
> > > > > > >
> > > > > > >
> > > > > > > So I would like to suggest using a permission model that is
> based
> > > on
> > > > > > > 'java.security.Permission'.
> > > > > > > I prepared the concept [1] of 

Re: Model of permissions for Ignite 3

[Valentin Kulichenko]

Danis,

Got it, thanks. Please provide the link to the IEP when you have one, I’d
be happy to review!

-Val

On Tue, Apr 20, 2021 at 12:21 AM Denis Garus  wrote:

> Hello!
>
> We arranged that I prepare an IEP.
> If you have some ideas about what should be reflected in this IEP, do not
> hesitate to let me know.
>
> вт, 20 апр. 2021 г. в 02:06, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Hi folks,
> >
> > Did you have a discussion? How did it go? Can someone summarize the
> > results?
> >
> > -Val
> >
> > On Mon, Apr 12, 2021 at 2:00 AM Kseniya Romanova <
> > romanova.ks@gmail.com>
> > wrote:
> >
> > > Hi! Scheduled open call for Friday
> > > https://www.meetup.com/Moscow-Apache-Ignite-Meetup/events/277518672/
> > > Please join to see the zoom link. Consulted with Denis Garus and put
> the
> > > topic as "Security", cause it's definitely wider than just permissions.
> > >
> > > Cheers!
> > >
> > > пн, 12 апр. 2021 г. в 09:25, Alexei Scherbakov <
> > > alexey.scherbak...@gmail.com
> > > >:
> > >
> > > > +1
> > > >
> > > > We should rethink the security model in Ignite 3 and have a default
> > RBAC
> > > > based implementation, from my point of view.
> > > > By default, no code should be written to enable and use it.
> > > >
> > > > Let's schedule a meeting and discuss the details.
> > > >
> > > > вс, 11 апр. 2021 г. в 19:43, Denis Garus :
> > > >
> > > > > Andrey, Alexey thank you for the feedback!
> > > > >
> > > > > > I suggest decoupling authentication from authorization
> > > > >
> > > > > Yes, it would be useful. Existing SecuritySubject and
> SecurityContext
> > > > > require reworking.
> > > > >
> > > > > > I think it would be great to have a default implementation of
> > > > > user-role-permission model
> > > > >
> > > > > Completely agree it is a cool idea. Ignite should have more default
> > > > > implementation referred to security.
> > > > >
> > > > > > Should we have a community meeting to discuss this?
> > > > >
> > > > > Yes, it would be great.
> > > > > The wish list for Ignite 3 does not contain security improvement
> > that,
> > > > > IMHO, is wrong.
> > > > > We should fix that oversight on early-stage Ignite 3 development.
> > > > >
> > > > > пт, 9 апр. 2021 г. в 18:47, Alexey Goncharuk <
> > > alexey.goncha...@gmail.com
> > > > >:
> > > > >
> > > > > > Hello Denis, Andrey, Igniters,
> > > > > >
> > > > > > Why don't we take a step further in improving the security model
> in
> > > > > Ignite
> > > > > > 3? I think it would be great to have a default implementation of
> > > > > > user-role-permission model in Ignite to be on par with security
> > > models
> > > > of
> > > > > > widely-used databases. This will complement community efforts in
> > > making
> > > > > > most of the Ignite 3 behavior to be changeable in runtime.
> > > > > >
> > > > > > WDYT? Should we have a community meeting to discuss this?
> > > > > >
> > > > > >
> > > > > > чт, 8 апр. 2021 г. в 23:54, Andrey Kuznetsov  >:
> > > > > >
> > > > > > > Hi Denis!
> > > > > > >
> > > > > > > The idea and prototype look great.
> > > > > > >
> > > > > > > I'd like to highlight one arguable point. Default authorization
> > > > > > > implementation still assumes there are permissions provided in
> > > > > > > SecuritySubject. In turn, authentication is still responsible
> for
> > > > > filling
> > > > > > > these permissions. I suggest decoupling authentication from
> > > > > > authorization,
> > > > > > > so that GridSecurityProcessor implementation is fully
> responsible
> > > for
> > > > > > > obtaining permissions for SecuritySubject given on
> authorization.
> > > In
> > > > > > > particular, implementation can choose an e

Re: Re[2]: [DISCUSSION] Request for thread unsafe Compute functionality deprecation.

[Valentin Kulichenko]

Zhenya,

Could you clarify what you mean by "one instance is shared between numerous
of fabric"? What is the exact scenario and what are the implications of
running multiple compute tasks in parallel? A code sample demonstrating the
scenario might be helpful as well.

-Val

On Fri, Feb 5, 2021 at 12:58 PM Vladislav Pyatkov 
wrote:

> Hi Zhenya,
>
> I don't understand your proposal without a patch.
> I see that you want to mark as @Depricate three methods in IgniteCompute
> interface, but what will you give instead of?
>
> It seems to me, this interface can invoke some job without contains a class
> locally.
> How will this be supported in your mind?
>
> On Thu, Jan 28, 2021 at 6:58 PM Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > Please publish it. I don't see why not.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > чт, 28 янв. 2021 г. в 14:28, Zhenya Stanilovsky
>  > >:
> >
> > >
> > >
> > > Hi Ilya , of course it contains in my PR (i don`t know if it shout be
> > > published before this discussion will be finished).
> > > Little changes from single thread into multiple, for example here [1]
> > will
> > > highlight a problem, or i can just publish my PR.
> > >
> > > [1]
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/test/java/org/apache/ignite/internal/IgniteExplicitImplicitDeploymentSelfTest.java#L221
> > >
> > > >
> > > >>
> > > >>>Hello!
> > > >>>
> > > >>>Do you have some kind of reproducer which demonstrates the issue?
> > > >>>
> > > >>>Regards,
> > > >>>--
> > > >>>Ilya Kasnacheev
> > > >>>
> > > >>>
> > > >>>чт, 28 янв. 2021 г. в 10:32, Zhenya Stanilovsky <
> > > arzamas...@mail.ru.invalid
> > > :
> > > >>>
> > > 
> > >  Hello Igniters !
> > >  In the process of Ignite usage i found that some part of Compute
> > >  functionality are thread unsafe and seems was designed with such
> > >  limitations initially.
> > >  Example : one (client, but it doesn`t matter at all) instance is
> > >  shared between numerous of fabric, all of them calls something
> like
> > :
> > >  IgniteCompute#execute(ComputeTask, T)
> > >  or
> > >  IgniteCompute#execute(java.lang.Class>,
> > T)
> > >  and appropriate «async» methods — what kind of instance will be
> > > called is
> > >  nondeterministic for now and as a confirmation of my words — i
> found
> > > no
> > >  tests covered multi thread usage of Computing i also found nothing
> > on
> > >  documentation page [1].
> > >  We have all necessary info for correct processing of such cases:
> > >  from initiator (ignite.compute(...) starter) side we have Class or
> > it
> > >  instance and appropriate class loader which will be wired by class
> > > loader
> > >  id from execution side.
> > >  I create a fix and seems all work perfectly well besides one
> place,
> > > this
> > >  functionality :
> > >  /**
> > >  * Executes given task within the cluster group. For step-by-step
> > >  explanation of task execution process
> > >  * refer to {@link ComputeTask} documentation.
> > >  * 
> > >  * If task for given name has not been deployed yet, then {@code
> > > taskName}
> > >  will be
> > >  * used as task class name to auto-deploy the task (see {@link
> > >  #localDeployTask(Class, ClassLoader)} method).
> > >  */
> > >  public  R execute(String taskName, T arg) throws
> > > IgniteException;
> > >  and attendant
> > >  /**
> > >  * Finds class loader for the given class.
> > >  *
> > >  * @param rsrcName Class name or class alias to find class loader
> > for.
> > >  * @return Deployed class loader, or {@code null} if not deployed.
> > >  */
> > >  public DeploymentResource findResource(String rsrcName);
> > >  is thread unsafe by default, no guarantee that concurrent call of
> > >  localDeployTask and execute will bring expected result.
> > >  My proposal is to deprecate (or probably annotate [2], as a
> minimal
> > >  — additionally document it) this methods and to append additional
> :
> > >  public DeploymentResource findResource(String rsrcName,
> ClassLoader
> > >  clsLdr);
> > >  Only one problem i can observe here, if someone creates new class
> > > loaders
> > >  and appropriate class instances in loop (i don`t know the purpose)
> > and
> > >  doesn`t undeploy them then he will get possibly OOM here.
> > > 
> > >  Such approach will give a possibility to use compute in concurrent
> > >  scenario. If there is no objections here i will mark this methods
> > and
> > >  publish my PR, of course with additional tests.
> > > 
> > >  What do you think ?
> > > 
> > > 
> > >  [1]
> > > 
> > >
> https://ignite.apache.org/docs/latest/code-deployment/peer-class-loading
> > >  [2]
> > > 
> > >
> https://jcip.net/annotations/doc/net/jcip/annotations/NotThreadSafe.html
> > > 
> > > 
> > > >>

[DISCUSSION] Moving tracing feature to GA

[Valentin Kulichenko]

Igniters,

As far as I know, the tracing feature [1] is still marked as experimental.
However, It seems that we've had it for a while, it's being used, the API
is stabilized, and there are no known performance issues associated with it.

That said, should we remove the "experimental" mark and move the feature to
the GA status? Please share your thoughts.

[1] https://ignite.apache.org/docs/latest/monitoring-metrics/tracing

-Val

Re: IP Filtering in IPFinders

[Valentin Kulichenko]

Hi Atri,

Can you describe the scenario in a little more detail? What exactly do you
mean by a container shared by multiple clusters? What are the consequences
of this? How does the proposed solution solve the problem?

Also, I would suggest revisiting the design - I'm not sure such filtering
should be done on the IP finder level. Why not do this on the SPI level
instead? I would simply add something like "addressFilter" to the
TcpDiscoverySpi. The filter can be a generic IgnitePredicate, so you will
be able to provide any implementations, including regex or anything else.

-Val

On Wed, Apr 21, 2021 at 9:04 AM Atri Sharma  wrote:

> Hi,
>
> When a container is shared by multiple clusters, then this can be useful
> for filtering IPs.
>
> Also, things like VPC based barriers can be circumvented using this
> technique.
>
> On Wed, 21 Apr 2021, 15:49 Ilya Kasnacheev, 
> wrote:
>
> > Hello!
> >
> > What are the expected use cases for this feature? Can you please
> elaborate?
> >
> > Thanks,
> > --
> > Ilya Kasnacheev
> >
> >
> > ср, 21 апр. 2021 г. в 08:23, Atri Sharma :
> >
> > > Hi All,
> > >
> > > I have opened the following JIRA for the said topic:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-14606
> > >
> > > The concept is to filter IPs based on a pattern or a blocklist in
> > > IPFinders while consuming IPs. This is more pertinent for cloud based
> > > IPFinders since they can have shared containers.
> > >
> > > For the moment, I have implemented regex based filtering:
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-14607
> > >
> > > for Azure Blob Storage IP Finder. Over time, we can extend the same to
> > > other IP finders.
> > >
> > > Please see the PR:
> > >
> > > https://github.com/apache/ignite/pull/9024
> > >
> > > Regards,
> > >
> > > Atri
> > >
> > > --
> > > Regards,
> > >
> > > Atri
> > > Apache Concerted
> > >
> >
>

Re: Ignite 3 Java Thin Client configuration: HOCON and ignite-configuration module

[Valentin Kulichenko]

Pavel,

The configuration framework is designed to support dynamic configuration
changes - I doubt this is needed for the client side. I think we should
start with simple POJOs or builders.

-Val

On Sat, Aug 21, 2021 at 7:03 AM Ivan Daschinsky  wrote:

> As for me, it is very strange purpose and not very practical. I can hardly
> imagine why someone will use it. For example, if I use micronaut or spring,
> this dependency will make me angry
>
> сб, 21 авг. 2021 г., 16:13 Pavel Tupitsyn :
>
> > Ivan, the purpose is to be able to configure thin client with HOCON.
> >
> > On Sat, Aug 21, 2021 at 3:37 PM Ivan Daschinsky 
> > wrote:
> >
> > > Hi. As for me, it is quite strange to make thin client dependent on
> third
> > > party libraries like hocon parser and so on. What's the purpose of
> this?
> > >
> > > сб, 21 авг. 2021 г., 13:32 Pavel Tupitsyn :
> > >
> > > > Igniters,
> > > >
> > > > I'd like to discuss Java thin client configuration API in Ignite 3.0.
> > > >
> > > > On one hand, it would be nice to use codegen approach from
> > > > ignite-configuration module,
> > > > and have consistent config APIs across servers and thin clients.
> > > >
> > > > On the other hand, that API may seem a bit confusing, because for one
> > > > ClientConfigurationSchema we get ClientConfiguration (mutable),
> > > ClientView
> > > > (immutable, name should probably be ClientConfigurationView), and
> > > > ClientChange for mutations.
> > > >
> > > > I've drafted some changes in [1], see [2] for a usage example.
> > > >
> > > > Should we follow ignite-configuration approach or create something
> else
> > > for
> > > > the thin client?
> > > >
> > > >
> > > > [1] https://github.com/apache/ignite-3/pull/298
> > > > [2]
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite-3/blob/a26921666f7bff7c45ae35a2244a2bbb2396b241/modules/client/src/test/java/org/apache/ignite/client/ConfigurationTest.java#L42
> > > >
> > >
> >
>

[DISCUSSION] Code style for Ignite 3

[Valentin Kulichenko]

Igniters,

I would like to discuss a potential change to the coding guidelines for
Ignite 3. Currently, we're using the existing guidelines inherited from
Ignite 2, which are described here:
https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines

Current guidelines, however, exist for many years and have several issues.
They are cumbersome, carry a lot of legacy stuff, and can't be automated.
Every now and then, they seem to cause questions and confusion.

While it's hard to make drastic changes in Ignite 2, we still have a great
opportunity to update the guidelines in Ignite 3. I would identify two
major goals here:

   1. Simplification. Having too many rules and restrictions tend to
   complicate development rather than providing any value. We should come up
   with a minimum set of rules and then make amends one by one if needed.
   2. The ability for automation. I hold a strong belief that code style
   checking has to become a part of the build. Therefore, we need to make sure
   that any rule that ends up in the guideline can be automatically verified.

I propose the following process to define the new guideline:

   1. Use Google code style as the starting point:
   https://google.github.io/styleguide/javaguide.html
   2. Replace the 100 column limit with 140. The latter is the value we
   already use in Ignite 2, and it seems to be more reasonable, in my opinion.
   3. Use 4 spaces block indentation and 8 spaces for continuation (as
   opposed to 2 and 4). Nothing wrong with 2 spaces, in my view, but 4 spaces
   should provide a smoother transition, as we're really used to this style.
   4. For any other changes, initiate separate discussions going forward.

Several reasons why I specifically propose Google style:

   1. This is essentially the standard for many projects. I don't think
   there is a need for us to reinvent the wheel.
   2. It's minimalistic and developer-friendly. No overcomplication.
   3. (probably the most important) It comes with all the required tools
   and configurations for automation (e.g., here is the config for IDEA:
   https://github.com/google/error-prone/blob/master/.idea/GoogleStyle.xml)

Please let me know what you think. If there are no objections, I will start
the process.

-Val

Re: [DISCUSSION] Code style for Ignite 3

[Valentin Kulichenko]

Created tickets for the migration:
- https://issues.apache.org/jira/browse/IGNITE-15370
- https://issues.apache.org/jira/browse/IGNITE-15371

-Val

On Fri, Aug 20, 2021 at 7:23 AM Pavel Tupitsyn  wrote:

> +1, as long as 100% of the rules are checked automatically.
>
> On Fri, Aug 20, 2021 at 4:00 PM Andrey Gura  wrote:
>
> > Looks good to me. But Idea configuration for style check is not
> > enough. It helps developers but does not automate style checking.
> >
> > Checkstyle project provides ready to use config based on Google Code
> > Style [1]. I hope it matches well with Idea config and we'll avoid any
> > confusing incidents.
> >
> > Let's start with this convention and adopt it to our needs if needed.
> >
> > [1]
> >
> https://github.com/checkstyle/checkstyle/blob/master/src/main/resources/google_checks.xml
> >
> > On Fri, Aug 20, 2021 at 12:02 PM Alexei Scherbakov
> >  wrote:
> > >
> > > +1
> > >
> > > пт, 20 авг. 2021 г. в 10:54, Alexander Polovtcev <
> > alexpolovt...@gmail.com>:
> > >
> > > > Hi, Val. This is an extremely welcome change, thank you!
> > > >
> > > > On Fri, Aug 20, 2021 at 12:17 AM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > I would like to discuss a potential change to the coding guidelines
> > for
> > > > > Ignite 3. Currently, we're using the existing guidelines inherited
> > from
> > > > > Ignite 2, which are described here:
> > > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines
> > > > >
> > > > > Current guidelines, however, exist for many years and have several
> > > > issues.
> > > > > They are cumbersome, carry a lot of legacy stuff, and can't be
> > automated.
> > > > > Every now and then, they seem to cause questions and confusion.
> > > > >
> > > > > While it's hard to make drastic changes in Ignite 2, we still have
> a
> > > > great
> > > > > opportunity to update the guidelines in Ignite 3. I would identify
> > two
> > > > > major goals here:
> > > > >
> > > > >1. Simplification. Having too many rules and restrictions tend
> to
> > > > >complicate development rather than providing any value. We
> should
> > come
> > > > > up
> > > > >with a minimum set of rules and then make amends one by one if
> > needed.
> > > > >2. The ability for automation. I hold a strong belief that code
> > style
> > > > >checking has to become a part of the build. Therefore, we need
> to
> > make
> > > > > sure
> > > > >that any rule that ends up in the guideline can be automatically
> > > > > verified.
> > > > >
> > > > > I propose the following process to define the new guideline:
> > > > >
> > > > >1. Use Google code style as the starting point:
> > > > >https://google.github.io/styleguide/javaguide.html
> > > > >2. Replace the 100 column limit with 140. The latter is the
> value
> > we
> > > > >already use in Ignite 2, and it seems to be more reasonable, in
> my
> > > > > opinion.
> > > > >3. Use 4 spaces block indentation and 8 spaces for continuation
> > (as
> > > > >opposed to 2 and 4). Nothing wrong with 2 spaces, in my view,
> but
> > 4
> > > > > spaces
> > > > >should provide a smoother transition, as we're really used to
> this
> > > > > style.
> > > > >4. For any other changes, initiate separate discussions going
> > forward.
> > > > >
> > > > > Several reasons why I specifically propose Google style:
> > > > >
> > > > >1. This is essentially the standard for many projects. I don't
> > think
> > > > >there is a need for us to reinvent the wheel.
> > > > >2. It's minimalistic and developer-friendly. No
> overcomplication.
> > > > >3. (probably the most important) It comes with all the required
> > tools
> > > > >and configurations for automation (e.g., here is the config for
> > IDEA:
> > > > >
> > > >
> > https://github.com/google/error-prone/blob/master/.idea/GoogleStyle.xml
> > > > > )
> > > > >
> > > > > Please let me know what you think. If there are no objections, I
> will
> > > > start
> > > > > the process.
> > > > >
> > > > > -Val
> > > > >
> > > >
> > > >
> > > > --
> > > > With regards,
> > > > Aleksandr Polovtcev
> > > >
> > >
> > >
> > > --
> > >
> > > Best regards,
> > > Alexei Scherbakov
> >
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Valentin Kulichenko]

Makes sense, thanks!

-Val

On Fri, Sep 3, 2021 at 2:00 AM Pavel Tupitsyn  wrote:

> Hi Val,
>
> This is a very good point.
>
> I've looked around blogs, docs, and system APIs, and updated the IEP
> accordingly:
> For Ignite.NET I propose NOT to add sync methods when the actual
> implementation is async:
> - It is easy to consume async APIs in C# with async/await keywords (added
> in 2012 and widely adopted)
> - Most codebases are fully async anyway
> - System APIs and popular libraries follow this direction
> - Sync-over-async is misleading and can affect performance
>
>
> However, I'm not so sure about Java, where async/await are not present,
> overall async usage seems to be rarer, and removing sync methods may become
> an obstacle for the users in some cases.
> Let's create a separate discussion and see what others think.
>
> Pavel
>
> On Thu, Sep 2, 2021 at 10:32 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Hi Pavel,
> >
> > I've looked at the IEP and the public API - looks good to me.
> >
> > Quick question - do you plan to add sync methods to the interfaces, or
> > you're thinking to only leave async? If the latter, what are the
> arguments
> > for this? The reason I'm asking is that I'm actually thinking about
> > suggesting the same for Java as well (or at least having a discussion
> about
> > this).
> >
> > -Val
> >
> > On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Igniters,
> > >
> > > Please review the IEP [1] and the PoC [2] for .NET Thin Client in
> Ignite
> > > 3.0.
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
> > > [2] https://github.com/apache/ignite-3/pull/306
> > >
> >
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Valentin Kulichenko]

Hi Pavel,

I've looked at the IEP and the public API - looks good to me.

Quick question - do you plan to add sync methods to the interfaces, or
you're thinking to only leave async? If the latter, what are the arguments
for this? The reason I'm asking is that I'm actually thinking about
suggesting the same for Java as well (or at least having a discussion about
this).

-Val

On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn  wrote:

> Igniters,
>
> Please review the IEP [1] and the PoC [2] for .NET Thin Client in Ignite
> 3.0.
>
> [1]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
> [2] https://github.com/apache/ignite-3/pull/306
>

Re: Ban Java Streams usage in Ignite 3 code

[Valentin Kulichenko]

I don't think we should ban anything. Streams API is just a tool in the
toolbox - it should be used appropriately. It's up to the contributor and
reviewer(s) to identify whether a particular usage might cause performance
issues.

-Val

On Wed, Sep 8, 2021 at 8:01 AM Alexander Polovtcev 
wrote:

> -1
> I think that it is not very productive to assume that 100% of your code is
> on the hot path, it would be impossible to write and maintain. Humans are
> not very good at guessing where the performance bottlenecks are, so the
> performance of the possible hot paths should be measured first and only
> then optimized and documented.
>
> On Wed, Sep 8, 2021 at 5:53 PM Ivan Pavlukhin  wrote:
>
> > Does not this trivial strategy work for us?
> > https://wiki.c2.com/?OptimizeLater
> >
> > 2021-09-08 13:52 GMT+03:00, Andrey Gura :
> > > Agree that any additional object creation on a hot path could be a
> > > problem. So hot path should not contain stream API and any other
> > > potentially problem code (e.g. iterator instead of for by index).
> > >
> > > On Wed, Sep 8, 2021 at 1:45 PM Pavel Tupitsyn 
> > wrote:
> > >>
> > >> Ok, maybe a total ban is overkill, but right now streams are used even
> > on
> > >> some hot paths like getAllAsync [1].
> > >>
> > >> Another issue is that Collectors.toList() and other variants don't
> > accept
> > >> capacity, and we end up with unnecessary reallocations of underlying
> > >> arrays.
> > >>
> > >> [1]
> > >>
> >
> https://github.com/apache/ignite-3/blob/1d7d703ff2b18234b15a9a7b03104fbb73388edf/modules/table/src/main/java/org/apache/ignite/internal/table/KVBinaryViewImpl.java#L83
> > >>
> > >> On Wed, Sep 8, 2021 at 1:06 PM Konstantin Orlov 
> > >> wrote:
> > >>
> > >> > Hi!
> > >> >
> > >> > Agree with Ivan that it’s an overkill. Code readability and
> > >> > maintainability should have
> > >> > the same priority as the performance (with some exceptions of
> course).
> > >> >
> > >> > BTW the result of your benchmark looks quite strange. The
> performance
> > >> > penalty on
> > >> > my laptop (Core i7 9750H, 6 cores up to 4.50 GHz) is 25%, not 8
> times:
> > >> >
> > >> > Benchmark Mode  Cnt  Score Error
> > >> > Units
> > >> > JmhIncrementBenchmark.loopSumthrpt   10  32347.819 ± 676.548
> > >> > ops/ms
> > >> > JmhIncrementBenchmark.streamSum  thrpt   10  24459.196 ± 610.152
> > >> > ops/ms
> > >> >
> > >> >
> > >> > --
> > >> > Regards,
> > >> > Konstantin Orlov
> > >> >
> > >> >
> > >> > > On 8 Sep 2021, at 12:23, Ivan Bessonov 
> > wrote:
> > >> > >
> > >> > > Hello Igniters,
> > >> > >
> > >> > > I object, banning streams is an overkill. I would argue that most
> of
> > >> > > the
> > >> > > code
> > >> > > is not on hot paths and that allocations in TLAB don't create much
> > >> > pressure
> > >> > > on GC.
> > >> > >
> > >> > > Streams must be used cautiously, developers should know whether
> they
> > >> > > write hot methods or not. And if methods are not hot, code
> > simplicity
> > >> > must
> > >> > > be
> > >> > > the first priority. I don't want Ignite 3 code to look like
> Ignite 2
> > >> > code,
> > >> > > where
> > >> > > people would iterate over Lists using explicit access by indexes,
> > >> > because it
> > >> > > saves them a single Iterator allocation. That's absurd.
> > >> > >
> > >> > > ср, 8 сент. 2021 г. в 11:43, Pavel Tupitsyn  >:
> > >> > >
> > >> > >> Igniters,
> > >> > >>
> > >> > >> Java streams are known to be slower and cause more GC pressure
> than
> > >> > >> an
> > >> > >> equivalent loop.
> > >> > >> Below is a simple filter/map/reduce scenario (code [1]):
> > >> > >>
> > >> > >> * Benchmark
> >  Mode
> > >> > Cnt
> > >> > >>Score Error   Units
> > >> > >>
> > >> > >> * StreamVsLoopBenchmark.loopSum
> > >> > >> thrpt
> > >> >   3
> > >> > >> 7987.016 ± 293.013  ops/ms
> > >> > >> * StreamVsLoopBenchmark.loopSum:·gc.alloc.rate
> > >> > >> thrpt
> > >> >   3
> > >> > >>   ≈ 10⁻⁴MB/sec
> > >> > >> * StreamVsLoopBenchmark.loopSum:·gc.count
> > >> > >> thrpt
> > >> >   3
> > >> > >>  ≈ 0counts
> > >> > >>
> > >> > >> * StreamVsLoopBenchmark.streamSum
> > >> > >> thrpt
> > >> >   3
> > >> > >> 1060.244 ±  36.485  ops/ms
> > >> > >> * StreamVsLoopBenchmark.streamSum:·gc.alloc.rate
> > >> > >> thrpt
> > >> >   3
> > >> > >>  315.819 ±  10.844  MB/sec
> > >> > >> * StreamVsLoopBenchmark.streamSum:·gc.count
> > >> > >> thrpt
> > >> >   3
> > >> > >>   55.000counts
> > >> > >>
> > >> > >> Loop is several times faster and does not allocate at all.
> > >> > >>
> > >> > >> 1. Performance is one of the most important features of our
> > product.
> > >> > >> 2. Most of our APIs will be on the hot path.
> > >> > >>
> > >> > >> One can argue about performance differences in real-world
> > scenarios,
> > >> > >> but increasing GC pressure just to make the code a little bit
> nicer
> > >> > >> is
> > >> > >> unacceptable.
> > >> > >>
> > >> > >> I propose to ban streams usage 

Re: Ban Java Streams usage in Ignite 3 code

[Valentin Kulichenko]

Pavel,

Quite frankly, I think we used to lean into performance too much. We
generally preferred it over data consistency, project modularity and code
readability. Performance, of course, plays a very important rule in Ignite,
but it's possible to overdo anything.

There are certainly parts of the project that can benefit from features
like Stream API, without significant concern over performance. CLI is an
obvious example, but I'm sure it's not the only one.

That said, I don't think that banning something is productive. At the same
time, we should make sure we pay more attention to performance during
reviews. Maybe we should have a checklist of major things to look for? Not
as a set of strict rules, but more as a guideline for contributors and
committers.

We might also want to start benchmarking the code and tracking the progress.

-Val

On Wed, Sep 8, 2021 at 12:09 PM Pavel Tupitsyn  wrote:

> Alexander, Ivan,
>
> > not very productive to assume that 100% of your code is
> > on the hot path
>
> That is exactly what we should be doing.
> When I joined Ignite community 6 years ago, this was the prevalent mindset.
>
> I'm not sure which part of Ignite can be considered "not on a hot path".
> Create/alter table (mentioned above) should perform well too.
>
> > measured first and only then optimized
> > https://wiki.c2.com/?OptimizeLater
>
> Extra allocations are a "death by a thousand cuts".
> They add up here and there, and then there are GC pauses.
> This would be hard to "optimize later".
>
> It is common for perf-oriented projects to avoid some techniques.
> For example, LINQ (streams analog from C# with similar perf issues) is
> avoided in libraries and compilers [1].
>
> [1] https://github.com/dotnet/roslyn/blob/main/CONTRIBUTING.md
>
> On Wed, Sep 8, 2021 at 9:49 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > I don't think we should ban anything. Streams API is just a tool in the
> > toolbox - it should be used appropriately. It's up to the contributor and
> > reviewer(s) to identify whether a particular usage might cause
> performance
> > issues.
> >
> > -Val
> >
> > On Wed, Sep 8, 2021 at 8:01 AM Alexander Polovtcev <
> > alexpolovt...@gmail.com>
> > wrote:
> >
> > > -1
> > > I think that it is not very productive to assume that 100% of your code
> > is
> > > on the hot path, it would be impossible to write and maintain. Humans
> are
> > > not very good at guessing where the performance bottlenecks are, so the
> > > performance of the possible hot paths should be measured first and only
> > > then optimized and documented.
> > >
> > > On Wed, Sep 8, 2021 at 5:53 PM Ivan Pavlukhin 
> > wrote:
> > >
> > > > Does not this trivial strategy work for us?
> > > > https://wiki.c2.com/?OptimizeLater
> > > >
> > > > 2021-09-08 13:52 GMT+03:00, Andrey Gura :
> > > > > Agree that any additional object creation on a hot path could be a
> > > > > problem. So hot path should not contain stream API and any other
> > > > > potentially problem code (e.g. iterator instead of for by index).
> > > > >
> > > > > On Wed, Sep 8, 2021 at 1:45 PM Pavel Tupitsyn <
> ptupit...@apache.org>
> > > > wrote:
> > > > >>
> > > > >> Ok, maybe a total ban is overkill, but right now streams are used
> > even
> > > > on
> > > > >> some hot paths like getAllAsync [1].
> > > > >>
> > > > >> Another issue is that Collectors.toList() and other variants don't
> > > > accept
> > > > >> capacity, and we end up with unnecessary reallocations of
> underlying
> > > > >> arrays.
> > > > >>
> > > > >> [1]
> > > > >>
> > > >
> > >
> >
> https://github.com/apache/ignite-3/blob/1d7d703ff2b18234b15a9a7b03104fbb73388edf/modules/table/src/main/java/org/apache/ignite/internal/table/KVBinaryViewImpl.java#L83
> > > > >>
> > > > >> On Wed, Sep 8, 2021 at 1:06 PM Konstantin Orlov <
> > kor...@gridgain.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi!
> > > > >> >
> > > > >> > Agree with Ivan that it’s an overkill. Code readability and
> > > > >> > maintainability should have
> > > > >> > the same priority as the performance (with some exceptions o

Re: Sync vs async APIs in Ignite 3

[Valentin Kulichenko]

Pavel,

I might be missing something - could you please elaborate a little more?

When I say "sync on top of async", I basically mean that (for example)
'insert(..)' is equivalent to 'insertAsync(..).join()'. In my
understanding, it only blocks the user's thread.

Am I wrong? Or you have a different implementation in mind?

-Val

On Wed, Sep 8, 2021 at 12:50 AM Pavel Tupitsyn  wrote:

> Val,
>
> Agree with your points.
>
>
> > async API should be primary
>
> It should be noted that all our APIs are inherently async,
> because thin client is implemented asynchronously.
>
>
> > with the sync version build on top
>
> We should document somehow that sync APIs are based on async ones,
> because this may be dangerous in some use cases.
>
> For example, as a user, I may have a thread pool of 4 threads for
> Ignite-related usage, that is also set as asyncContinuationExecutor [1].
> Now if I run a lot of concurrent Ignite requests using sync API, all 4
> threads will end up blocked on CompletableFutures.
> When one of the operations completes, we enqueue the completion to that
> same thread pool, but all threads are blocked on sync APIs, resulting in a
> deadlock.
>
> This can be prevented by using a different asyncContinuationExecutor, but
> sync API users won't be usually aware of this.
>
>
> [1] https://issues.apache.org/jira/browse/IGNITE-15359
>
> On Wed, Sep 8, 2021 at 10:04 AM Courtney Robinson <
> courtney.robin...@hypi.io>
> wrote:
>
> > Hi Val,
> >
> > I'd highly support an async first API based on CompletionStage
> > <
> >
> https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/CompletionStage.html
> > >
> > or
> > its subtypes like CompletableFuture.
> > In Ignite 2 we've written a wrapper library around IgniteFuture to
> provide
> > CompletionStage instead because many of the newer libs we use support
> this.
> > If Ignite 3 went this way it'd remove a lot of boiler plate/wrapper that
> we
> > wrote to get what you're suggesting here.
> >
> > Regards,
> > Courtney Robinson
> > Founder and CEO, Hypi
> > Tel: ++44 208 123 2413 (GMT+0) <https://hypi.io>
> >
> > <https://hypi.io>
> > https://hypi.io
> >
> >
> > On Wed, Sep 8, 2021 at 12:44 AM Valentin Kulichenko <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Igniters,
> > >
> > > I would like to gather some opinions on whether we want to focus on
> sync
> > vs
> > > async APIs in Ignite 3.
> > >
> > > Here are some initial considerations that I have:
> > > 1. Ignite 2.x is essentially "sync first". Async APIs exist, but they
> use
> > > non-standard IgniteFuture and provide counterintuitive guarantees. In
> my
> > > experience, they significantly lack usability, and because of that are
> > > rarely used.
> > > 2. In general, however, async execution becomes more and more
> prominent.
> > > Something we can't ignore if we want to create a modern framework.
> > > 3. Still, async support in Java is very limited (especially if compared
> > to
> > > other languages, like C# for example).
> > >
> > > My current position is the following (happy to discuss):
> > > 1. We should pay more attention to async APIs. As a general rule, async
> > API
> > > should be primary, with the sync version build on top.
> > > 2. In languages with proper async support (async-await, etc.), we can
> > skip
> > > sync API altogether. As an example of this, you can look at the first
> > > version of the .NET client [1]. It exposes only async methods, and it
> > > doesn't look like sync counterparts are really needed.
> > > 3. In Java (as well as other languages where applicable), we will add
> > sync
> > > APIs that simply delegate to async APIs. This will help users to avoid
> > > CompletableFuture if they don't want to use it.
> > >
> > > [1] https://github.com/apache/ignite-3/pull/306
> > >
> > > Please share your thoughts.
> > >
> > > -Val
> > >
> >
>

Re: Replace Map with List and Iterable in KeyValueView Ignite 3 APIs

[Valentin Kulichenko]

How do we handle the "equality" part in 2.x? The same problem exists there
as well, but we still somehow return a Map.

Generally, I like Pavel's ideas, but there are a couple of issues with
them. Firstly, Java developers are really used to maps in this context.
Introducing something else might be confusing - it's a significant risk.
Secondly, there is no standard Pair class, so we'll have to introduce a
custom one. That said, I would not change this API in Java. In other
languages, however, we can consider this.

-Val

On Thu, Sep 9, 2021 at 8:01 AM Alexei Scherbakov <
alexey.scherbak...@gmail.com> wrote:

> Pavel,
>
> I think the current API looks more natural compared to your proposal.
>
> -1  from my side, see comments below.
>
> чт, 9 сент. 2021 г. в 15:38, Pavel Tupitsyn :
>
> > Igniters,
> >
> > I propose to replace Map with List in getAll and invokeAll, and
> > Iterable in putAll APIs of Ignite 3.x KeyValueView.
> >
> > 1. Performance
> > putAll simply iterates over the map, we can easily accept Iterable
> instead.
> > Iterable can be implemented over anything, it can lazily read data from a
> > file or some other place, instead of allocating a huge collection and
> > performing unnecessary hashing.
> >
> > getAll returns a Map, but we don't know if the user code needs a map or
> > just wants to iterate over the results, in which case Map is just
> overhead.
> >
>
> "allocating a huge collection" -
> This method is not intended for loading a huge number of entries.
> The allowed map size for the putAll should be limited to some reasonable
> value.
>
> Streaming loading should be addressed in a separate API similar to
> DataStream from Ignite 2.
>
>
> >
> > 2. Equality
> > getAll returns Map, but in many cases, the map will be useless
> > because K does not have proper equals()/hashCode() implementation, so
> > map.get(key) does not work.
> >
>
> We shouldn't rely on user equals/hashCode while working with key-value API.
> Internally it uses binary representation of a user object for comparison
> purposes.
> The returned map implementation should work in the same way.
>
>
> >
> > Notes:
> > - It is not clear which Pair class to use yet - IgniteBiTuple or
> something
> > else.
> > - Ignite 3 won't deadlock due to putAll entry order, so we don't have to
> > worry about sorting.
> >
> > Thoughts, objections?
> >
>
>
> --
>
> Best regards,
> Alexei Scherbakov
>

Re: IEP-78 .NET Thin Client for Ignite 3.0

[Valentin Kulichenko]

Hi Pavel,

I've created a thread.

-Val

On Mon, Sep 6, 2021 at 12:02 PM Pavel Tupitsyn  wrote:

> Val,
>
> Would you like me to start the discussion about sync-over-async in Ignite 3
> Java APIs, or do you plan to do it yourself?
>
> On Fri, Sep 3, 2021 at 10:10 PM Valentin Kulichenko <
> valentin.kuliche...@gmail.com> wrote:
>
> > Makes sense, thanks!
> >
> > -Val
> >
> > On Fri, Sep 3, 2021 at 2:00 AM Pavel Tupitsyn 
> > wrote:
> >
> > > Hi Val,
> > >
> > > This is a very good point.
> > >
> > > I've looked around blogs, docs, and system APIs, and updated the IEP
> > > accordingly:
> > > For Ignite.NET I propose NOT to add sync methods when the actual
> > > implementation is async:
> > > - It is easy to consume async APIs in C# with async/await keywords
> (added
> > > in 2012 and widely adopted)
> > > - Most codebases are fully async anyway
> > > - System APIs and popular libraries follow this direction
> > > - Sync-over-async is misleading and can affect performance
> > >
> > >
> > > However, I'm not so sure about Java, where async/await are not present,
> > > overall async usage seems to be rarer, and removing sync methods may
> > become
> > > an obstacle for the users in some cases.
> > > Let's create a separate discussion and see what others think.
> > >
> > > Pavel
> > >
> > > On Thu, Sep 2, 2021 at 10:32 PM Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Hi Pavel,
> > > >
> > > > I've looked at the IEP and the public API - looks good to me.
> > > >
> > > > Quick question - do you plan to add sync methods to the interfaces,
> or
> > > > you're thinking to only leave async? If the latter, what are the
> > > arguments
> > > > for this? The reason I'm asking is that I'm actually thinking about
> > > > suggesting the same for Java as well (or at least having a discussion
> > > about
> > > > this).
> > > >
> > > > -Val
> > > >
> > > > On Thu, Sep 2, 2021 at 10:08 AM Pavel Tupitsyn  >
> > > > wrote:
> > > >
> > > > > Igniters,
> > > > >
> > > > > Please review the IEP [1] and the PoC [2] for .NET Thin Client in
> > > Ignite
> > > > > 3.0.
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-78+.NET+Thin+Client
> > > > > [2] https://github.com/apache/ignite-3/pull/306
> > > > >
> > > >
> > >
> >
>

Sync vs async APIs in Ignite 3

[Valentin Kulichenko]

Igniters,

I would like to gather some opinions on whether we want to focus on sync vs
async APIs in Ignite 3.

Here are some initial considerations that I have:
1. Ignite 2.x is essentially "sync first". Async APIs exist, but they use
non-standard IgniteFuture and provide counterintuitive guarantees. In my
experience, they significantly lack usability, and because of that are
rarely used.
2. In general, however, async execution becomes more and more prominent.
Something we can't ignore if we want to create a modern framework.
3. Still, async support in Java is very limited (especially if compared to
other languages, like C# for example).

My current position is the following (happy to discuss):
1. We should pay more attention to async APIs. As a general rule, async API
should be primary, with the sync version build on top.
2. In languages with proper async support (async-await, etc.), we can skip
sync API altogether. As an example of this, you can look at the first
version of the .NET client [1]. It exposes only async methods, and it
doesn't look like sync counterparts are really needed.
3. In Java (as well as other languages where applicable), we will add sync
APIs that simply delegate to async APIs. This will help users to avoid
CompletableFuture if they don't want to use it.

[1] https://github.com/apache/ignite-3/pull/306

Please share your thoughts.

-Val

[DISCUSSION] Separate Jira project and Confluence space for Ignite 3

[Valentin Kulichenko]

Igniters,

I think it's clear to all of us that Ignite 2.x and 3.x will coexist for a
while. They are developed in separate Git repos, but we still accumulate
the tickets for both versions in the same Jira project, which seems to
complicate the ticket management.

For example, we use the "ignite-3" label for 3.x tickets, but this approach
is fragile. If someone forgets to add the label to a new ticket, it's
likely to be lost. We need a better separation.

All the above is true for Wiki as well - we use a single Confluence space.

I suggest creating a new Jira project and a new Confluence space for Ignite
3 and moving all the relevant tickets and pages there.

Any thoughts or objections?

-Val

Re: Support for Apache Spark 3.1.2

[Valentin Kulichenko]

Hello Abhinav,

Spark 3 support definitely becomes higher priority for Ignite as more
people transition to it from Spark 2. I'm sure someone in the community
will pick this up soon.

In the meantime, could you please give a little more detail on how you use
Ignite with Spark? Are you using the dataframes integration [1] or
something else?

[1]
https://ignite.apache.org/docs/latest/extensions-and-integrations/ignite-for-spark/ignite-dataframe

-Val

On Tue, Sep 21, 2021 at 11:09 PM 
wrote:

> Hello team,
>
> We have recently upgraded to use Spark 3.1.2 to utilize Kubernetes support
> and ability to use spot instances. However, we are not able to integrate
> Ignite with this version. I understand that a lot of people of raise
> interest in the same. In general, loosely couple these two projects so
> benefits of both can be achieved? Considering that Spark releases do not
> necessary align to Ignite and vice versa, would it make sense?
>
> Please suggest.
>
> Regards,
> Abhinav
>
>
> _
> “This message is for information purposes only, it is not a
> recommendation, advice, offer or solicitation to buy or sell a product or
> service nor an official confirmation of any transaction. It is directed at
> persons who are professionals and is not intended for retail customer use.
> Intended for recipient only. This message is subject to the terms at:
> www.barclays.com/emaildisclaimer.
>
> For important disclosures, please see:
> www.barclays.com/salesandtradingdisclaimer regarding market commentary
> from Barclays Sales and/or Trading, who are active market participants;
> https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html
> regarding our standard terms for the Investment Bank of Barclays where we
> trade with you in principal-to-principal wholesale markets transactions;
> and in respect of Barclays Research, including disclosures relating to
> specific issuers, please see http://publicresearch.barclays.com.”
>
> _
> If you are incorporated or operating in Australia, please see
> https://www.home.barclays/disclosures/importantapacdisclosures.html for
> important disclosure.
>
> _
> How we use personal information  see our privacy notice
> https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
>
> _
>