Re: [DISCUSSION] New Ignite settings for IGNITE-12438 and IGNITE-13013

[Dmitriy Pavlov]

Sorry for me being obnoxious here, but still. Mentioning ticket numbers
usually does not make a lot of sense for someone, who is outside of context
of popular/trendy tickets.

At the same time, usually contributors scan emails in the list using
subject. They decide what should go to archive and what should they read.
So, in future it may worth to replace all numbers with tickets' actual
topic/summary. People are still humans and they may be too lazy to open
tickets to decide if they need to read this topic or not.

пт, 22 мая 2020 г. в 02:48, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Ivan,
>
> Have you considered eliminating server to client connections altogether?
> Or, at the very least making the "client to server only" mode the default
> one?
>
> All the suggested names are confusing and not intuitive, and I doubt we
> will be able to find a good one. A server initiating a TCP connection with
> a client is confusing in the first place and creates a usability issue. We
> now want to solve it by introducing an additional configuration
> parameter, and therefore additional complexity. I don't think this is the
> right approach.
>
> What are the drawbacks of permanently switching to client-to-server
> connections? Is there any value provided by the server-to-client option?
>
> As for pair connections, I'm not sure I understand why there is a
> limitation. As far as I know, the idea behind this feature is that we
> maintain two connections between two nodes instead of one, so that every
> connection is used for communication in a single direction only. Why does
> it matter which node initiates the connection? Why can't one of the nodes
> (e.g., a client) initiate both connections, and then use them accordingly?
> Correct me if I'm wrong, but I don't see why we can't do this.
>
> -Val
>
> On Thu, May 21, 2020 at 1:58 PM Denis Magda  wrote:
>
> > Ivan,
> >
> > Considering that the setting controls the way a communication SPI
> > connection is open I would add the new parameter to CommunicationSpi
> > interface naming it as follows:
> >
> > >
> > > CommunicationSpi.connectionInitiationMode
> > > {
> > > BIDIRECTIONAL, //both clients and servers initiate a connection
> > > initiation procedure
> > > CLIENTS_TO_SERVERS //servers cannot open a connection to clients,
> > only
> > > clients can do that
> > > }
> >
> >
> > The problem with the environment type approach is that private networks
> of
> > bare-metal environments usually impose restrictions similar to virtual
> > environments powered by Kubernetes. Thus, environmentType.VIRTUALIZED
> > doesn't cover all the cases and I'm struggling to come up with a
> universal
> > alternative.
> >
> > -
> > Denis
> >
> >
> > On Thu, May 21, 2020 at 5:38 AM Ivan Bessonov 
> > wrote:
> >
> > > Hello Igniters,
> > >
> > > I'd like to discuss with you changes related to [1] and [2]. Both
> issues
> > > are mostly the same so
> > > let's discuss the core idea.
> > >
> > > *Motivation.*
> > >
> > > There are certain environments that don't allow Ignite server nodes to
> > open
> > > TCP connections to
> > > thick clients, e.g. K8S, AWS Lambda or Azure Functions. To operate in
> > such
> > > environments, the
> > > server needs a way to request a client to open an "inverse"
> communication
> > > connection to it.
> > >
> > > I've prepared a PR (still in progress) that introduces new mechanism of
> > > opening connection and
> > > related configuration.
> > >
> > > *Main idea*
> > >
> > > This mechanism is called "communication via discovery" or "inverse
> > > connection", it works as
> > > follows:
> > >  - server that needs to connect to "unreachable" thick client sends a
> > > specific Discovery message
> > >(inverse communication request) to that client;
> > >  - client node upon receiving the request opens communication
> connection
> > to
> > > that server;
> > >  - server sees connection opened by client and proceeds with its task
> > (that
> > > required opening
> > >connection to the client).
> > >
> > > Working name for new configuration parameter for this feature is
> > > environmentType, it is an
> > > enum with two values (again, working names): STANDALONE (default) and
> > > VIRTUALIZED.
> > > It is used as a hint to server to speed-up establishing of connections:
> > > when server sees a client
> > > with VIRTUALIZED environmentType it doesn't try to open connection to
> it
> > > and sends inverse
> > > communication request right away.
> > > If environmentType is STANDALONE then server tries to open a connection
> > in
> > > a regular way
> > > (iterating over all client addresses) and sends request only if all
> > > attempts failed.
> > >
> > > There is a concern about naming of the configuration as it catches only
> > one
> > > use-case: when we
> > > deal with some kind of virtualized environment like K8S.
> > > There are other options I've encountered in private discussion:
> > > - connectionMode - ALWAYS_INITIATE / 

Re: Re[2]: [DISCUSS] Ignite process exit code on node stop by failure handler

[Dmitriy Pavlov]

It seems reasonable to me. Also I would like to propose adding value of
Ignition.KILL_EXIT_CODE into javadoc using @value javadoc tag.

Dev ops/admins/anyone who admins Ignite may want to know it's value without
going to Java code.

чт, 21 мая 2020 г. в 09:39, Zhenya Stanilovsky :

>
> Thank you Sergey, as for me — very useful proposal huge +1 here.
>
>
> >Четверг, 21 мая 2020, 0:51 +03:00 от Sergey Antonov <
> antonovserge...@gmail.com>:
> >
> >I've created the Ignite ticket for this improvement [1].
> >
> >[1]  https://issues.apache.org/jira/browse/IGNITE-13047
> >
> >чт, 21 мая 2020 г. в 00:46, Sergey Antonov < antonovserge...@gmail.com >:
> >
> >> Hello, Igniters!
> >>
> >> I'd like to discuss behaviour of Ignite process exit code if the node
> was
> >> stopped by failure handler [1][2]. At the moment ignite process returns
> >> exit code 0 after the stop in all scenarios, except runtime halt by
> >> StopNodeOrHaltFH [1]. In this case, the exit code will be 130 [3]
> >>
> >> My proposal: always finish Ignite process with code [3], if a node was
> >> stopped by FH. It could be helpful for administration purposes, you can
> >> distinguish normal node stop from node stop by FH on OS level.
> >>
> >> WDYT?
> >>
> >> [1]
> >>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeOrHaltFailureHandler.html
> >>
> >> [2]
> >>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/failure/StopNodeFailureHandler.html
> >>
> >> [3]
> >>
> https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/Ignition.html#KILL_EXIT_CODE
> >> --
> >> BR, Sergey Antonov
> >>
> >
> >--
> >BR, Sergey Antonov
> >
>
>
>
>

Re: IGNITE-12359 Migrate RocketMQ module to ignite-extensions

[Saikat Maitra]

Hi Ilya,

Thank you for reviewing the changes.

I have updated the PR.

Please review and share your feedback.

Regards,
Saikat

On Sat, May 16, 2020 at 7:30 PM Saikat Maitra 
wrote:

> Hi,
>
> I have raised PRs for the following issue:
>
> IGNITE-12359 Migrate RocketMQ module to ignite-extensions
>
> https://issues.apache.org/jira/browse/IGNITE-12359
>
> PR
> https://github.com/apache/ignite-extensions/pull/14
> https://github.com/apache/ignite/pull/7809
>
> Please take a look and share feedback.
>
> Regards,
> Saikat
>

Re: [DISCUSS] Apache Ignite mascot

[Saikat Maitra]

Thank you Denis, that would be great.

Regards,
Saikat

On Thu, May 21, 2020 at 11:02 AM Denis Magda  wrote:

> Folks, thanks for sharing ideas.
>
> Let me find a designer who can turn the ideas into pictures. Then we can
> compare and adjust.
>
> -
> Denis
>
>
> On Sun, May 17, 2020 at 5:50 AM Manuel Núñez  wrote:
>
>> What about a hellboy’s style animal?, with Ignite’s flame between the
>> horns
>>
>> El 16 may 2020, a las 23:22, Saikat Maitra 
>> escribió:
>>
>> 
>> Hi Denis,
>>
>> Awesome ideas !!! yes, I am aligned on a friendly dragon character as a
>> mascot, like Spyro. I also like the idea of peacock as its feathers colors
>> shows a nice clusters.
>>
>> Here are few proposals I wanted to share:
>>
>> 1. Dragon (fire, spark, ignite)
>> 2. Peacock (feathers, node clusters)
>> 3. Cheetah (speed, fast)
>> 4. Beluga Whales (echolocation, multicasting)
>>
>> Let me know if you have more suggestions.
>>
>> Regards,
>> Saikat
>>
>>
>>
>>
>> On Sun, May 10, 2020 at 3:16 AM Denis Magda  wrote:
>>
>>> Hi Saikat,
>>>
>>> I love this idea and, frankly, have been contemplating for a while about
>>> the same. Such characters (aka. mascots) can convey the spirit of our
>>> project and community as well as make life more entertaining.
>>>
>>> I bet that every Java developer instantly associates Java (as a project
>>> and
>>> community) once he/she comes accross Java Duke [1]. Those of us who live
>>> in
>>> the U.S. know who Geigo Gecko [1] is. What to say about Ronald McDonald
>>> [3]
>>> is famous worldwide.
>>>
>>> How do you see the Ignite's mascot? I imagine it as a dragon who is kind
>>> and friendly rather than furious and hostile. Our logo is a spark of fire
>>> and dragons are those whimsical creatures who are usually associated with
>>> fire and flames.
>>>
>>> [1] 568px-Duke_(Java_mascot)_waving.svg.png
>>> <
>>> https://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/Duke_%28Java_mascot%29_waving.svg/568px-Duke_%28Java_mascot%29_waving.svg.png
>>> >
>>> [2] maxresdefault.jpg <
>>> https://i.ytimg.com/vi/8lZeTTXyIk4/maxresdefault.jpg>
>>> [3] ronald-mcdonald-052011.jpg
>>> <
>>> https://s3-prod.adage.com/s3fs-public/styles/800x600/public/ronald-mcdonald-052011.jpg
>>> >
>>>
>>> -
>>> Denis
>>>
>>>
>>> On Sat, May 9, 2020 at 1:07 PM Saikat Maitra 
>>> wrote:
>>>
>>> > Hi,
>>> >
>>> > I wanted to discuss the idea of creating Apache Ignite mascot and if
>>> it is
>>> > something we would be interested. I was recently following the process
>>> of
>>> > creating a mascot  Firefly[1] in Apache Beam project and absolutely
>>> loved
>>> > the concept.
>>> >
>>> >
>>> https://blogs.apache.org/foundation/entry/success-at-apache-bringing-the
>>> >
>>> > Regards,
>>> > Saikat
>>> >
>>>
>>

Re: Apache Ignite and Mono

[Ilya Kasnacheev]

Hello!

We still say we support Mono:
https://apacheignite-net.readme.io/docs/cross-platform-support

Do we deprecate building with Mono or running also? I think we need to do
something formal here since we are removing functionality.

Regards,
-- 
Ilya Kasnacheev


пн, 25 мая 2020 г. в 18:21, Pavel Tupitsyn :

> Ilya, good point.
> I'll remove build-mono.sh, it does not make sense anyway.
>
>
> On Mon, May 25, 2020 at 11:03 AM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com>
> wrote:
>
> > Hello!
> >
> > Well, while we ship build-mono.sh I'll try to run it every time :) I'm
> not
> > running tests with it, just smoke testing to see if anything happens at
> > all.
> >
> > I admit it looks like Mono no longer has an ambition to be the
> stand-alone
> > dotnet runtime for Linux.
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 22 мая 2020 г. в 19:47, Pavel Tupitsyn :
> >
> > > Ilya,
> > >
> > > Mono 6.x is broken [1]. The fix is merged, but only included in nightly
> > > builds [2].
> > > Mono 5.x works.
> > >
> > > Ignite.NET does not have official Mono support.
> > > Please use .NET Core 2+ to test Ignite.NET (on any OS) [3].
> > >
> > > [1] https://github.com/mono/mono/issues/18418
> > > [2] https://github.com/mono/mono/pull/18430
> > > [3] https://dotnet.microsoft.com/download
> > >
> > > (sorry for hijacking the release vote thread)
> > >
> > > On Fri, May 22, 2020 at 7:21 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > +1 (binding)
> > > >
> > > > Checked that .deb package installs and runs.
> > > > Checked node start and C++ build in -bin.
> > > > Checked source compilation and mono compilation in -src (note that
> > > running
> > > > with Mono is still broken, but problem is not new and seems to be on
> > Mono
> > > > side).
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 22 мая 2020 г. в 15:15, Pavel Tupitsyn :
> > > >
> > > > > +1 (binding)
> > > > > Checked .NET packages and examples.
> > > > >
> > > > > On Fri, May 22, 2020 at 2:38 PM Alexey Zinoviev <
> > > zaleslaw@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > пт, 22 мая 2020 г. в 13:57, Nikolay Izhikov  >:
> > > > > >
> > > > > > > Dear Community,
> > > > > > >
> > > > > > > I have uploaded release candidate to
> > > > > > > https://dist.apache.org/repos/dist/dev/ignite/2.8.1-rc1/
> > > > > > >
> > https://dist.apache.org/repos/dist/dev/ignite/packages_2.8.1-rc1/
> > > > > > >
> > > > > > > The following staging can be used for testing:
> > > > > > >
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheignite-1481
> > > > > > >
> > > > > > > This is the first maintenance release for 2.8.x with a number
> of
> > > > fixes.
> > > > > > >
> > > > > > > Tag name is 2.8.1-rc1:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.8.1-rc1
> > > > > > >
> > > > > > > RELEASE NOTES:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=RELEASE_NOTES.txt;h=531b2ff7d72bdec9e0d9130efb27bf64f2b4f3fd;hb=9e214ee8ce7b87780d904111d359c74e90613e3f#l5
> > > > > > >
> > > > > > > Complete list of resolved issues:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.8.1
> > > > > > >
> > > > > > > DEVNOTES
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=DEVNOTES.txt;h=307189059ae70834ead4c127cc18295ce4d0735a;hb=9e214ee8ce7b87780d904111d359c74e90613e3
> > > > > > >
> > > > > > > The vote is formal, see voting guidelines
> > > > > > > https://www.apache.org/foundation/voting.html
> > > > > > >
> > > > > > > +1 - to accept Apache Ignite 2.8.1-rc1
> > > > > > > 0 - don't care either way
> > > > > > > -1 - DO NOT accept Apache Ignite Ignite 2.8.1-rc1 (explain why)
> > > > > > >
> > > > > > > See notes on how to verify release here
> > > > > > > https://www.apache.org/info/verification.html
> > > > > > > and
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> > > > > > >
> > > > > > > Additional links:
> > > > > > >
> > > > > > > NuGet staging -
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
> > > > > > > Comparsion of 2.8.0 and 2.8.1 -
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.ignite.apache.org/repository/download/ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/5329458:id/results/result.log
> > > > > > >
> > > > > > >
> > > > > > > This vote will be open for at least 3 days till Sun May 22,
> > > 23:59:59
> > > > > UTC.
> > > > > > >
> > > > > >
> > > > >
> > > >

[jira] [Created] (IGNITE-13070) SQL regressions detection framework

[Roman Kondakov (Jira)]

Roman Kondakov created IGNITE-13070:
---

 Summary: SQL regressions detection framework
 Key: IGNITE-13070
 URL: https://issues.apache.org/jira/browse/IGNITE-13070
 Project: Ignite
  Issue Type: Test
  Components: sql
Reporter: Roman Kondakov


We need to detect SQL regressions early. We can do it by comparing the SQL 
query performance for different Ignite versions. This test framework should 
work in the following way:
 # It starts two Ignite clusters with different versions (current version and 
the previous release version).
 # Framework then runs randomly generated queries in both clusters and checks 
the execution time for each cluster. We need to port SQLSmith library from C++ 
to java for this step. But initially we can start with some set of hardcoded 
queries and postpone the SQLSmith port. Randomized queries can be added later.
 # All problematic queries are then reported as performance issues. In this way 
we can manually examine the problems.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSSION] Framework for SQL performance regressions detection.

[Roman Kondakov]

I've found out that we have `ignite-compatibility` module. It already
has nice tools for launching different Ignite version simultaneously. We
can reuse these tools for SQL regressions detection.

I'm going to place the SQL regressions detection framework to this
module and add a separate TeamCity suite for it. We can run this suite
nightly to detect regressions problems early.

What do you think?


-- 
Kind Regards
Roman Kondakov


On 23.05.2020 16:12, Roman Kondakov wrote:
> Hi Denis,
> 
> I'm not sure we need a separate repository for it. What would be the
> benefit of using a separate repo?
> 
> BTW I noticed that Ignite has `ignite-benchmarks` module. It contains
> JMH/JOL benchmarks for now. We can also put the SQL benchmark to this
> module. What do you think?
> 
> 

Re: Apache Ignite and Mono

[Pavel Tupitsyn]

Ilya, good point.
I'll remove build-mono.sh, it does not make sense anyway.


On Mon, May 25, 2020 at 11:03 AM Ilya Kasnacheev 
wrote:

> Hello!
>
> Well, while we ship build-mono.sh I'll try to run it every time :) I'm not
> running tests with it, just smoke testing to see if anything happens at
> all.
>
> I admit it looks like Mono no longer has an ambition to be the stand-alone
> dotnet runtime for Linux.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пт, 22 мая 2020 г. в 19:47, Pavel Tupitsyn :
>
> > Ilya,
> >
> > Mono 6.x is broken [1]. The fix is merged, but only included in nightly
> > builds [2].
> > Mono 5.x works.
> >
> > Ignite.NET does not have official Mono support.
> > Please use .NET Core 2+ to test Ignite.NET (on any OS) [3].
> >
> > [1] https://github.com/mono/mono/issues/18418
> > [2] https://github.com/mono/mono/pull/18430
> > [3] https://dotnet.microsoft.com/download
> >
> > (sorry for hijacking the release vote thread)
> >
> > On Fri, May 22, 2020 at 7:21 PM Ilya Kasnacheev <
> ilya.kasnach...@gmail.com
> > >
> > wrote:
> >
> > > Hello!
> > >
> > > +1 (binding)
> > >
> > > Checked that .deb package installs and runs.
> > > Checked node start and C++ build in -bin.
> > > Checked source compilation and mono compilation in -src (note that
> > running
> > > with Mono is still broken, but problem is not new and seems to be on
> Mono
> > > side).
> > >
> > > Regards,
> > > --
> > > Ilya Kasnacheev
> > >
> > >
> > > пт, 22 мая 2020 г. в 15:15, Pavel Tupitsyn :
> > >
> > > > +1 (binding)
> > > > Checked .NET packages and examples.
> > > >
> > > > On Fri, May 22, 2020 at 2:38 PM Alexey Zinoviev <
> > zaleslaw@gmail.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > пт, 22 мая 2020 г. в 13:57, Nikolay Izhikov :
> > > > >
> > > > > > Dear Community,
> > > > > >
> > > > > > I have uploaded release candidate to
> > > > > > https://dist.apache.org/repos/dist/dev/ignite/2.8.1-rc1/
> > > > > >
> https://dist.apache.org/repos/dist/dev/ignite/packages_2.8.1-rc1/
> > > > > >
> > > > > > The following staging can be used for testing:
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapacheignite-1481
> > > > > >
> > > > > > This is the first maintenance release for 2.8.x with a number of
> > > fixes.
> > > > > >
> > > > > > Tag name is 2.8.1-rc1:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.8.1-rc1
> > > > > >
> > > > > > RELEASE NOTES:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=RELEASE_NOTES.txt;h=531b2ff7d72bdec9e0d9130efb27bf64f2b4f3fd;hb=9e214ee8ce7b87780d904111d359c74e90613e3f#l5
> > > > > >
> > > > > > Complete list of resolved issues:
> > > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.8.1
> > > > > >
> > > > > > DEVNOTES
> > > > > >
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=DEVNOTES.txt;h=307189059ae70834ead4c127cc18295ce4d0735a;hb=9e214ee8ce7b87780d904111d359c74e90613e3
> > > > > >
> > > > > > The vote is formal, see voting guidelines
> > > > > > https://www.apache.org/foundation/voting.html
> > > > > >
> > > > > > +1 - to accept Apache Ignite 2.8.1-rc1
> > > > > > 0 - don't care either way
> > > > > > -1 - DO NOT accept Apache Ignite Ignite 2.8.1-rc1 (explain why)
> > > > > >
> > > > > > See notes on how to verify release here
> > > > > > https://www.apache.org/info/verification.html
> > > > > > and
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> > > > > >
> > > > > > Additional links:
> > > > > >
> > > > > > NuGet staging -
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
> > > > > > Comparsion of 2.8.0 and 2.8.1 -
> > > > > >
> > > > >
> > > >
> > >
> >
> https://ci.ignite.apache.org/repository/download/ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/5329458:id/results/result.log
> > > > > >
> > > > > >
> > > > > > This vote will be open for at least 3 days till Sun May 22,
> > 23:59:59
> > > > UTC.
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.timeanddate.com/countdown/generic?iso=20200525T235959=%3A=Ignite+2.8.1-rc1+Vote=cursive
> > > > >
> > > >
> > >
> >
>

Re: Re[4]: IGNITE-6499 Compact NULL fields

[steve.hostett...@gmail.com]

Ok, anyway I will finish my patch 2 unit tests still not working. I assume
anyway that if we would want to apply zstd compression we would reuse the
existing page compression algorithm in memory and not only for persistence.
That would probably be anyway simpler and more straightforward.

Will try to hurry to submit a patch for review.



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-13069) Rewrite creation of IgniteInClosure and IgniteOutClosure as lambda expression

[Nikolay Izhikov (Jira)]

Nikolay Izhikov created IGNITE-13069:


 Summary: Rewrite creation of IgniteInClosure and IgniteOutClosure 
as lambda expression
 Key: IGNITE-13069
 URL: https://issues.apache.org/jira/browse/IGNITE-13069
 Project: Ignite
  Issue Type: New Feature
Affects Versions: 2.8
Reporter: Nikolay Izhikov


We can use lambda expression instead of direct instantiation of 
IgniteInClosure, IgniteOutClosure.

This:
{code:java}
state = new DirectMessageState<>(StateItem.class, new 
IgniteOutClosure() {
@Override public StateItem apply() {
return new StateItem(msgFactory, protoVer);
}
});

{code}

Can be replaced with:
{code:java}
state = new DirectMessageState<>(StateItem.class, () -> new 
StateItem(msgFactory, protoVer));
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Prevent insertion of cache entry if the binary field type and the type of the query entity do not match.

[Pavel Pereslegin]

Hello Igniters.

If type of binary field does not match query entity field type we
still able to insert such entry into cache, but can't query it.
In the following example we have query entity with the UUID field
"name", but we insert String field "name" using binary object.

IgniteCache cache = grid(0).createCache(
new CacheConfiguration<>("testCache").setQueryEntities(
Collections.singletonList(
new QueryEntity()
.setKeyFieldName("id")
.setValueType("Person")
.setFields(new LinkedHashMap<>(
F.asMap("id", "java.lang.Integer",
"name", "java.util.UUID"));

BinaryObject obj = grid(0).binary().builder("Person")
.setField("id", 1)
.setField("name", UUID.randomUUID().toString())
.build();

cache.put(1, obj);
assertEquals(obj, cache.withKeepBinary().get(1));

String sql = "select id, name from Person where id=1";

grid(0).context().query()
.querySqlFields(new
SqlFieldsQuery(sql).setSchema("testCache"), true)
.getAll(); // java.lang.ClassCastException:
java.lang.String cannot be cast to java.util.UUID

The object was successfully inserted, but the "name"-field cannot be
read using sql.

Should it be better to prevent insertion of cache entry if the binary
field type and the type of the query entity field do not match?

Re[4]: IGNITE-6499 Compact NULL fields

[Zhenya Stanilovsky]

Compress of whole binary inside ignite.


 
>Sorry I do not actual get what are you opposing? the compress of the binary
>or the null compaction or both?
>And can you ellaborate on why you are opposing it?
>
>
>
>--
>Sent from:  http://apache-ignite-developers.2346864.n4.nabble.com/ 
 
 
 
 

Re: ExecutorServices hide assertions without logging and node stop

[Ivan Pavlukhin]

Maxim,

> Assertions must be disabled in production :-)

Since production deployments are out of our control we should think
about ones with enabled assertions as well (and they do occur in
practice).

> A contributor who makes a new patch must be able to see failed tests 
> immediately, he must not check all the logs with logged assertions. The 
> assertion is an error (not to be confused with an exception), all the errors 
> must fail the node.

I am not sure how practical it is today. I might have missed some
recent activities but I recall an effort to enable a node terminating
failure handler in tests. AFAIR, it was not successful and had lead to
troubles with TC build results interpretation.

2020-05-25 14:27 GMT+03:00, Maxim Muzafarov :
> Ivan,
>
>> 2. Assertions might be disabled in production.
> Assertions must be disabled in production :-)
>
> Here is another side of the same coin.
> A contributor who makes a new patch must be able to see failed tests
> immediately, he must not check all the logs with logged assertions.
> The assertion is an error (not to be confused with an exception), all
> the errors must fail the node.
>
> If we not enabling catching assertions for ExecutorServices this patch
> for all the cases, at least it must be done for all tests running the
> TeamCity environment. But I think we should enable fail node if an
> error occurs (assume assertions disabled in production).
>
> On Mon, 25 May 2020 at 12:36, Ivan Pavlukhin  wrote:
>>
>> Maxim,
>>
>> I looked into the code and checked how do we setup uncaught exception
>> handlers for internal executors. Indeed it looks not good (OOME leads
>> to failure handling, other exceptions are silently ignored). Logging
>> sounds good.
>>
>> Have doubts about failure handling (e.g. terminating a node). I
>> remember at least one related discussion [1]. And here are some
>> statements which causes my doubts:
>> 1. Not only assertion errors are a programming errors evidence, e.g. a
>> simple NPE often signals the same.
>> 2. Assertions might be disabled in production.
>> 3. Sometimes it seems that assertions are often overused in the Ignite
>> code. Intermittent concurrency-related assertion errors are not so
>> uncommon. And it seems impossible to figure out how critical a
>> particular error is.
>>
>> [1]
>> http://apache-ignite-developers.2346864.n4.nabble.com/Exceptions-thrown-in-IndexingSpi-and-quot-fail-fast-quot-principle-td38904.html
>>
>> 2020-05-24 12:13 GMT+03:00, Maxim Muzafarov :
>> > Ivan,
>> >
>> > Not sure that I've got your point. Let me give some example:
>> >
>> > IgniteEx ignite = startGrid();
>> > ignite.context().getSystemExecutorService().execute(() -> { assert
>> > false;
>> > });
>> > ignite.context().getSystemExecutorService().submit(() -> { assert
>> > false;
>> > });
>> >
>> > In both of these cases there are no exceptions logged and the node not
>> > stopped (I assume that failing assertion means a code contract is
>> > violated). Of course, I'll get an assertion error if fut.get() will be
>> > called but there a lot of cases where it's not. So, I think we must
>> > log errors here or fail the node.
>> >
>> > On Fri, 22 May 2020 at 10:51, Ivan Pavlukhin 
>> > wrote:
>> >>
>> >> Hi Maxim,
>> >>
>> >> I recently had similar troubles related to ExecutorService and
>> >> exception handling. It was an unexpected behavior for me that
>> >> exceptions of any kind* thrown from Runnable in a following code:
>> >> executor.submit(() -> throw new RuntimeException())
>> >> were silently ignored.
>> >>
>> >> I seemed to me really unexpected for me at first. But later on I
>> >> figured out the cause. There is a subtle detail which I forgot.
>> >> ExecutorService.submit returns a Future and leaves a responsibility to
>> >> handle exceptions of any kind to a caller of Future.get. And it was
>> >> really subtle for me, because I did not use a returned Future at all.
>> >> Actually I supposed ExecutorService.execute when used
>> >> ExecutorService.submit.
>> >>
>> >> Is it related to the subject? Can ExecutorService.execute help here?
>> >>
>> >> * In my case there were exceptions caused by missing classes on a
>> >> classpath.
>> >>
>> >> 2020-05-21 22:22 GMT+03:00, Maxim Muzafarov :
>> >> > Folks,
>> >> >
>> >> > I've created an issue.
>> >> > https://issues.apache.org/jira/browse/IGNITE-13055
>> >> >
>> >> > On Wed, 6 May 2020 at 10:28, Nikolay Izhikov 
>> >> > wrote:
>> >> >>
>> >> >> Hello, Maxim.
>> >> >>
>> >> >> I can confirm this itching issue.
>> >> >> It also happens when some custom Security plugin throws an
>> >> >> exception
>> >> >> while
>> >> >> processing a communication message.
>> >> >>
>> >> >> ```
>> >> >> UUID newSecSubjId = secSubjId != null ? secSubjId : nodeId;
>> >> >>
>> >> >> try (OperationSecurityContext s =
>> >> >> ctx.security().withContext(newSecSubjId)) {
>> >> >> lsnr.onMessage(nodeId, msg, plc);
>> >> >> }
>> >> >> finally {
>> >> >> if (change)
>> >> >> CUR_PLC.set(oldPlc);

Re: Re[2]: IGNITE-6499 Compact NULL fields

[steve.hostett...@gmail.com]

Sorry I do not actual get what are you opposing? the compress of the binary
or the null compaction or both?
And can you ellaborate on why you are opposing it?



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Re: ExecutorServices hide assertions without logging and node stop

[Maxim Muzafarov]

Ivan,

> 2. Assertions might be disabled in production.
Assertions must be disabled in production :-)

Here is another side of the same coin.
A contributor who makes a new patch must be able to see failed tests
immediately, he must not check all the logs with logged assertions.
The assertion is an error (not to be confused with an exception), all
the errors must fail the node.

If we not enabling catching assertions for ExecutorServices this patch
for all the cases, at least it must be done for all tests running the
TeamCity environment. But I think we should enable fail node if an
error occurs (assume assertions disabled in production).

On Mon, 25 May 2020 at 12:36, Ivan Pavlukhin  wrote:
>
> Maxim,
>
> I looked into the code and checked how do we setup uncaught exception
> handlers for internal executors. Indeed it looks not good (OOME leads
> to failure handling, other exceptions are silently ignored). Logging
> sounds good.
>
> Have doubts about failure handling (e.g. terminating a node). I
> remember at least one related discussion [1]. And here are some
> statements which causes my doubts:
> 1. Not only assertion errors are a programming errors evidence, e.g. a
> simple NPE often signals the same.
> 2. Assertions might be disabled in production.
> 3. Sometimes it seems that assertions are often overused in the Ignite
> code. Intermittent concurrency-related assertion errors are not so
> uncommon. And it seems impossible to figure out how critical a
> particular error is.
>
> [1] 
> http://apache-ignite-developers.2346864.n4.nabble.com/Exceptions-thrown-in-IndexingSpi-and-quot-fail-fast-quot-principle-td38904.html
>
> 2020-05-24 12:13 GMT+03:00, Maxim Muzafarov :
> > Ivan,
> >
> > Not sure that I've got your point. Let me give some example:
> >
> > IgniteEx ignite = startGrid();
> > ignite.context().getSystemExecutorService().execute(() -> { assert false;
> > });
> > ignite.context().getSystemExecutorService().submit(() -> { assert false;
> > });
> >
> > In both of these cases there are no exceptions logged and the node not
> > stopped (I assume that failing assertion means a code contract is
> > violated). Of course, I'll get an assertion error if fut.get() will be
> > called but there a lot of cases where it's not. So, I think we must
> > log errors here or fail the node.
> >
> > On Fri, 22 May 2020 at 10:51, Ivan Pavlukhin  wrote:
> >>
> >> Hi Maxim,
> >>
> >> I recently had similar troubles related to ExecutorService and
> >> exception handling. It was an unexpected behavior for me that
> >> exceptions of any kind* thrown from Runnable in a following code:
> >> executor.submit(() -> throw new RuntimeException())
> >> were silently ignored.
> >>
> >> I seemed to me really unexpected for me at first. But later on I
> >> figured out the cause. There is a subtle detail which I forgot.
> >> ExecutorService.submit returns a Future and leaves a responsibility to
> >> handle exceptions of any kind to a caller of Future.get. And it was
> >> really subtle for me, because I did not use a returned Future at all.
> >> Actually I supposed ExecutorService.execute when used
> >> ExecutorService.submit.
> >>
> >> Is it related to the subject? Can ExecutorService.execute help here?
> >>
> >> * In my case there were exceptions caused by missing classes on a
> >> classpath.
> >>
> >> 2020-05-21 22:22 GMT+03:00, Maxim Muzafarov :
> >> > Folks,
> >> >
> >> > I've created an issue.
> >> > https://issues.apache.org/jira/browse/IGNITE-13055
> >> >
> >> > On Wed, 6 May 2020 at 10:28, Nikolay Izhikov 
> >> > wrote:
> >> >>
> >> >> Hello, Maxim.
> >> >>
> >> >> I can confirm this itching issue.
> >> >> It also happens when some custom Security plugin throws an exception
> >> >> while
> >> >> processing a communication message.
> >> >>
> >> >> ```
> >> >> UUID newSecSubjId = secSubjId != null ? secSubjId : nodeId;
> >> >>
> >> >> try (OperationSecurityContext s =
> >> >> ctx.security().withContext(newSecSubjId)) {
> >> >> lsnr.onMessage(nodeId, msg, plc);
> >> >> }
> >> >> finally {
> >> >> if (change)
> >> >> CUR_PLC.set(oldPlc);
> >> >> }
> >> >> ```
> >> >>
> >> >> If an exception thrown from `withContext` it is hidden from the user.
> >> >>
> >> >> > 4 мая 2020 г., в 19:15, Maxim Muzafarov 
> >> >> > написал(а):
> >> >> >
> >> >> > Igniters,
> >> >> >
> >> >> >
> >> >> > I've spent a couple of days in debug mode and found that some of the
> >> >> > configured ExecutorServices of an Ignite instance completely hide
> >> >> > assertion errors without any logging and node killing. This may
> >> >> > violate some internal guarantees and hide serious errors.
> >> >> >
> >> >> > Let's consider, for instance, GridDhtPartitionsExchangeFuture [1].
> >> >> > It
> >> >> > has three places of submitting some Runnable on system executor
> >> >> > service. If an assertion error (or even any uncached exception)
> >> >> > occurs
> >> >> > in the code block below it will be swallowed without logging,
> >> >> > exchange
> 

Re[2]: IGNITE-6499 Compact NULL fields

[Zhenya Stanilovsky]

I`m currently against this approach, everyone can previously compress Binary 
Object for further using,  no additional code need here. This discussion only 
about currently not optimal null storing and looks like we can improve it 
without performance pay.  

  
>Понедельник, 25 мая 2020, 13:42 +03:00 от Ilya Kasnacheev 
>:
> 
>Hello!
>
>That would be nice! My preferred compression method is zstd (it also has
>dictionary generation built in).
>
>Regards,
>--
>Ilya Kasnacheev
>
>
>пн, 25 мая 2020 г. в 13:25, Hostettler, Steve <
>steve.hostett...@wolterskluwer.com >:
> 
>> I like the idea, especially because it also would apply across the board.
>> So you propose to build the binary object and to apply dictionary based
>> compression on top.
>>
>> I could quickly generate a bunch of binary objects from the tests and
>> apply java compress/deflate with a dictionary based on the BinaryUtils
>> elements.
>> To compare with the null compaction and the varint.
>>
>>
>> -Original Message-
>> From: Ilya Kasnacheev < ilya.kasnach...@gmail.com >
>> Sent: Monday, May 25, 2020 12:05 PM
>> To: dev < dev@ignite.apache.org >
>> Subject: Re: IGNITE-6499 Compact NULL fields
>>
>> Caution, this email may be from a sender outside Wolters Kluwer. Verify
>> the sender and know the content is safe.
>>
>> Hello!
>>
>> My take is the following: if conserving memory is needed at all, then we
>> better invest in compression (such as dictionary-based row compression)
>> rather than implementing varint, compact nulls, etc.
>>
>> Dictionary-based compression can easily tackle varints, null patterns
>> while also compressing strings and repeated values and even things we would
>> never think out on our own.
>>
>> It also has low complexity of our own code, no compatibility issues
>> (people store binary objects in 3rd party storage, they do indeed) and low
>> incidence of bugs.
>>
>> Regards,
>> --
>> Ilya Kasnacheev
>>
>>
>> пн, 25 мая 2020 г. в 12:51, Hostettler, Steve <
>>  steve.hostett...@wolterskluwer.com >:
>>
>> > I went for a simpler approach (only with null mask( and yes the gain
>> > is high for smaller object but low otherwise. I gain between 5-20% on
>> > my objects. But to me it is the step stone to easily implement other
>> > optimisations like varint and schemaless without using raw. Trying to
>> > solve the latest unit tests to give you a better idea. If not worth
>> > then let's not do it but it is worth a try I think.
>> >
>> >
>> > -Original Message-
>> > From: Ilya Kasnacheev < ilya.kasnach...@gmail.com >
>> > Sent: Monday, May 25, 2020 11:48 AM
>> > To: dev < dev@ignite.apache.org >
>> > Subject: Re: IGNITE-6499 Compact NULL fields
>> >
>> > Caution, this email may be from a sender outside Wolters Kluwer.
>> > Verify the sender and know the content is safe.
>> >
>> > Hello!
>> >
>> > I can't help myself but wonder how large of a benefit will it give. I
>> > have checked the ticket description, it looks the proposed scheme is
>> > elaborate and benefit for non-extreme binary objects rather tiny.
>> >
>> > WDYT?
>> >
>> > Regards,
>> > --
>> > Ilya Kasnacheev
>> >
>> >
>> > пн, 18 мая 2020 г. в 22:54,  steve.hostett...@gmail.com <
>> >  steve.hostett...@gmail.com >:
>> >
>> > > Hello igniters,
>> > >
>> > > while I would like to help on the calcite because H2 optimiser (or
>> > > the lack
>> > > thereof) is really killing us, I think that it would be wiser to
>> > > start by contributing on something easier.
>> > >
>> > > Therefore I will tackle another problem that we have which is the
>> > > memory consumption. I stumbled upon this IEP
>> > >
>> > >  https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
>> > > ik
>> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
>> > > Bo
>> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
>> > > wo
>> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
>> > > fa
>> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
>> > > R3
>> > > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0
>> > > <
>> > >  https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
>> > > ik
>> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
>> > > Bo
>> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
>> > > wo
>> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
>> > > fa
>> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
>> > > R3 HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0>
>> > >
>> > > that is about optimising the binary marshaller.
>> > >
>> > > The low hanging fruit seemed to be the null compaction so I decided
>> > > to start with it. Though I am sure I do see some hidden complexity.
>> > >
>> > > Here a couple of questions:
>> > > - Can I assign myself IGNITE-6499 and attach a patch?
>> > > - Who can I contact to help with the review. In the following page
>> > >  

Re: IGNITE-6499 Compact NULL fields

[Ilya Kasnacheev]

Hello!

That would be nice! My preferred compression method is zstd (it also has
dictionary generation built in).

Regards,
-- 
Ilya Kasnacheev


пн, 25 мая 2020 г. в 13:25, Hostettler, Steve <
steve.hostett...@wolterskluwer.com>:

> I like the idea, especially because it also would apply across the board.
> So you propose to build the binary object and to apply dictionary based
> compression on top.
>
> I could quickly generate a bunch of binary objects from the tests and
> apply java compress/deflate with a dictionary based on the BinaryUtils
> elements.
> To compare with the null compaction and the varint.
>
>
> -Original Message-
> From: Ilya Kasnacheev 
> Sent: Monday, May 25, 2020 12:05 PM
> To: dev 
> Subject: Re: IGNITE-6499 Compact NULL fields
>
> Caution, this email may be from a sender outside Wolters Kluwer. Verify
> the sender and know the content is safe.
>
> Hello!
>
> My take is the following: if conserving memory is needed at all, then we
> better invest in compression (such as dictionary-based row compression)
> rather than implementing varint, compact nulls, etc.
>
> Dictionary-based compression can easily tackle varints, null patterns
> while also compressing strings and repeated values and even things we would
> never think out on our own.
>
> It also has low complexity of our own code, no compatibility issues
> (people store binary objects in 3rd party storage, they do indeed) and low
> incidence of bugs.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 25 мая 2020 г. в 12:51, Hostettler, Steve <
> steve.hostett...@wolterskluwer.com>:
>
> > I went for a simpler approach (only with null mask( and yes the gain
> > is high for smaller object but low otherwise. I gain between 5-20% on
> > my objects. But to me it is the step stone to easily implement other
> > optimisations like varint and schemaless without using raw. Trying to
> > solve the latest unit tests to give you a better idea. If not worth
> > then let's not do it but it is worth a try I think.
> >
> >
> > -Original Message-
> > From: Ilya Kasnacheev 
> > Sent: Monday, May 25, 2020 11:48 AM
> > To: dev 
> > Subject: Re: IGNITE-6499 Compact NULL fields
> >
> > Caution, this email may be from a sender outside Wolters Kluwer.
> > Verify the sender and know the content is safe.
> >
> > Hello!
> >
> > I can't help myself but wonder how large of a benefit will it give.  I
> > have checked the ticket description, it looks the proposed scheme is
> > elaborate and benefit for non-extreme binary objects rather tiny.
> >
> > WDYT?
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com <
> > steve.hostett...@gmail.com>:
> >
> > > Hello igniters,
> > >
> > > while I would like to help on the calcite because H2 optimiser (or
> > > the lack
> > > thereof) is really killing us, I think that it would be wiser to
> > > start by contributing on something easier.
> > >
> > > Therefore I will tackle another problem that we have which is the
> > > memory consumption. I stumbled upon this IEP
> > >
> > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > > ik
> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
> > > Bo
> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
> > > wo
> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
> > > fa
> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
> > > R3
> > > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0
> > > <
> > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > > ik
> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
> > > Bo
> > > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
> > > wo
> > > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
> > > fa
> > > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
> > > R3 HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0>
> > >
> > > that is about optimising the binary marshaller.
> > >
> > > The low hanging fruit seemed to be the null compaction so I decided
> > > to start with it. Though I am sure I do see some hidden complexity.
> > >
> > > Here a couple of questions:
> > > - Can I assign myself IGNITE-6499 and attach a patch?
> > > - Who can I contact to help with the review. In the following page
> > > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > > ik
> > > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute
> > > 
> > > mp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C75681484874
> > > 34
> > > 617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637
> > > 25
> > > 9968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03F
> > > Q%
> > > 3Dreserved=0
> > >  > > wi
> > > ki.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribut
> > > e&
> > > 

RE: IGNITE-6499 Compact NULL fields

[Hostettler, Steve]

I like the idea, especially because it also would apply across the board.
So you propose to build the binary object and to apply dictionary based 
compression on top.

I could quickly generate a bunch of binary objects from the tests and apply 
java compress/deflate with a dictionary based on the BinaryUtils elements.
To compare with the null compaction and the varint.


-Original Message-
From: Ilya Kasnacheev  
Sent: Monday, May 25, 2020 12:05 PM
To: dev 
Subject: Re: IGNITE-6499 Compact NULL fields

Caution, this email may be from a sender outside Wolters Kluwer. Verify the 
sender and know the content is safe.

Hello!

My take is the following: if conserving memory is needed at all, then we better 
invest in compression (such as dictionary-based row compression) rather than 
implementing varint, compact nulls, etc.

Dictionary-based compression can easily tackle varints, null patterns while 
also compressing strings and repeated values and even things we would never 
think out on our own.

It also has low complexity of our own code, no compatibility issues (people 
store binary objects in 3rd party storage, they do indeed) and low incidence of 
bugs.

Regards,
--
Ilya Kasnacheev


пн, 25 мая 2020 г. в 12:51, Hostettler, Steve <
steve.hostett...@wolterskluwer.com>:

> I went for a simpler approach (only with null mask( and yes the gain 
> is high for smaller object but low otherwise. I gain between 5-20% on 
> my objects. But to me it is the step stone to easily implement other 
> optimisations like varint and schemaless without using raw. Trying to 
> solve the latest unit tests to give you a better idea. If not worth 
> then let's not do it but it is worth a try I think.
>
>
> -Original Message-
> From: Ilya Kasnacheev 
> Sent: Monday, May 25, 2020 11:48 AM
> To: dev 
> Subject: Re: IGNITE-6499 Compact NULL fields
>
> Caution, this email may be from a sender outside Wolters Kluwer. 
> Verify the sender and know the content is safe.
>
> Hello!
>
> I can't help myself but wonder how large of a benefit will it give.  I 
> have checked the ticket description, it looks the proposed scheme is 
> elaborate and benefit for non-extreme binary objects rather tiny.
>
> WDYT?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com <
> steve.hostett...@gmail.com>:
>
> > Hello igniters,
> >
> > while I would like to help on the calcite because H2 optimiser (or 
> > the lack
> > thereof) is really killing us, I think that it would be wiser to 
> > start by contributing on something easier.
> >
> > Therefore I will tackle another problem that we have which is the 
> > memory consumption. I stumbled upon this IEP
> >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > ik 
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
> > Bo 
> > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
> > wo 
> > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
> > fa
> > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
> > R3
> > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0
> > <
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > ik 
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2
> > Bo 
> > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40
> > wo 
> > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141f
> > fa
> > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5K
> > R3 HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0>
> >
> > that is about optimising the binary marshaller.
> >
> > The low hanging fruit seemed to be the null compaction so I decided 
> > to start with it. Though I am sure I do see some hidden complexity.
> >
> > Here a couple of questions:
> > - Can I assign myself IGNITE-6499 and attach a patch?
> > - Who can I contact to help with the review. In the following page 
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcw
> > ik 
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute
> > 
> > mp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C75681484874
> > 34
> > 617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C637
> > 25 
> > 9968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03F
> > Q%
> > 3Dreserved=0
> >  > wi 
> > ki.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribut
> > e&
> > amp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C7568148487
> > 43
> > 4617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C63
> > 72 
> > 59968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03
> > FQ %3Dreserved=0> there is no one assigned for marshalling.
> >
> > On the details:
> > The compression is disabled by default as it is not compatible with 
> > objects previously marshalled.
> >
> > My approach was to go a bit 

Re: IGNITE-6499 Compact NULL fields

[Ilya Kasnacheev]

Hello!

My take is the following: if conserving memory is needed at all, then we
better invest in compression (such as dictionary-based row compression)
rather than implementing varint, compact nulls, etc.

Dictionary-based compression can easily tackle varints, null patterns while
also compressing strings and repeated values and even things we would never
think out on our own.

It also has low complexity of our own code, no compatibility issues (people
store binary objects in 3rd party storage, they do indeed) and low
incidence of bugs.

Regards,
-- 
Ilya Kasnacheev


пн, 25 мая 2020 г. в 12:51, Hostettler, Steve <
steve.hostett...@wolterskluwer.com>:

> I went for a simpler approach (only with null mask( and yes the gain is
> high for smaller object but low otherwise. I gain between 5-20% on my
> objects. But to me it is the step stone to easily implement other
> optimisations like varint and schemaless without using raw. Trying to solve
> the latest unit tests to give you a better idea. If not worth then let's
> not do it but it is worth a try I think.
>
>
> -Original Message-
> From: Ilya Kasnacheev 
> Sent: Monday, May 25, 2020 11:48 AM
> To: dev 
> Subject: Re: IGNITE-6499 Compact NULL fields
>
> Caution, this email may be from a sender outside Wolters Kluwer. Verify
> the sender and know the content is safe.
>
> Hello!
>
> I can't help myself but wonder how large of a benefit will it give.  I
> have checked the ticket description, it looks the proposed scheme is
> elaborate and benefit for non-extreme binary objects rather tiny.
>
> WDYT?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com <
> steve.hostett...@gmail.com>:
>
> > Hello igniters,
> >
> > while I would like to help on the calcite because H2 optimiser (or the
> > lack
> > thereof) is really killing us, I think that it would be wiser to start
> > by contributing on something easier.
> >
> > Therefore I will tackle another problem that we have which is the
> > memory consumption. I stumbled upon this IEP
> >
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2Bo
> > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40wo
> > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141ffa
> > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5KR3
> > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0
> > <
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2Bo
> > bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40wo
> > lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141ffa
> > 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5KR3
> > HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0>
> >
> > that is about optimising the binary marshaller.
> >
> > The low hanging fruit seemed to be the null compaction so I decided to
> > start with it. Though I am sure I do see some hidden complexity.
> >
> > Here a couple of questions:
> > - Can I assign myself IGNITE-6499 and attach a patch?
> > - Who can I contact to help with the review. In the following page
> > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> > i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute
> > mp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C7568148487434
> > 617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C63725
> > 9968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03FQ%
> > 3Dreserved=0
> >  > ki.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute&
> > amp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C756814848743
> > 4617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C6372
> > 59968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03FQ
> > %3Dreserved=0> there is no one assigned for marshalling.
> >
> > On the details:
> > The compression is disabled by default as it is not compatible with
> > objects previously marshalled.
> >
> > My approach was to go a bit beyond the JIRA. No only do I remove the
> > indexes to null fields in the footer, I also remove the 0x65 in the
> > objects. I did not remove them fro the collections and arrays because
> > they are using absolute positioning.
> >
> > I gain between 5% to 20% depending of my test cases. Obviously the
> > smaller the object and the higher the number of nulls, the higher the
> > compression rate.
> >
> > Based on that I can quite easily add var int compression which is
> > IGNITE-6418 and should significantly increase the compression rate
> > with a lot of integers and longs when only using small numbers.
> >
> > Next step is to add JMH micro-benchmark to check the impact in terms
> > of performances.
> >
> >
> > Example on a simple object w/ 

RE: IGNITE-6499 Compact NULL fields

[Hostettler, Steve]

I went for a simpler approach (only with null mask( and yes the gain is high 
for smaller object but low otherwise. I gain between 5-20% on my objects. But 
to me it is the step stone to easily implement other optimisations like varint 
and schemaless without using raw. Trying to solve the latest unit tests to give 
you a better idea. If not worth then let's not do it but it is worth a try I 
think.


-Original Message-
From: Ilya Kasnacheev  
Sent: Monday, May 25, 2020 11:48 AM
To: dev 
Subject: Re: IGNITE-6499 Compact NULL fields

Caution, this email may be from a sender outside Wolters Kluwer. Verify the 
sender and know the content is safe.

Hello!

I can't help myself but wonder how large of a benefit will it give.  I have 
checked the ticket description, it looks the proposed scheme is elaborate and 
benefit for non-extreme binary objects rather tiny.

WDYT?

Regards,
--
Ilya Kasnacheev


пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com <
steve.hostett...@gmail.com>:

> Hello igniters,
>
> while I would like to help on the calcite because H2 optimiser (or the 
> lack
> thereof) is really killing us, I think that it would be wiser to start 
> by contributing on something easier.
>
> Therefore I will tackle another problem that we have which is the 
> memory consumption. I stumbled upon this IEP
>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2Bo
> bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40wo
> lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141ffa
> 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5KR3
> HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FIEP-2%253A%2BBinary%2Bo
> bject%2Bformat%2Bimprovementsdata=02%7C01%7CSteve.Hostettler%40wo
> lterskluwer.com%7C7568148487434617407b08d80090b1f2%7C8ac76c91e7f141ffa
> 89c3553b2da2c17%7C0%7C0%7C637259968758509764sdata=ZNFJ5gqEXRv5KR3
> HJUfYZ4rmnGwCiFVGg4IrWTORT2k%3Dreserved=0>
>
> that is about optimising the binary marshaller.
>
> The low hanging fruit seemed to be the null compaction so I decided to 
> start with it. Though I am sure I do see some hidden complexity.
>
> Here a couple of questions:
> - Can I assign myself IGNITE-6499 and attach a patch?
> - Who can I contact to help with the review. In the following page
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwik
> i.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute
> mp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C7568148487434
> 617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C63725
> 9968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03FQ%
> 3Dreserved=0 
>  ki.apache.org%2Fconfluence%2Fdisplay%2FIGNITE%2FHow%2Bto%2BContribute&
> amp;data=02%7C01%7CSteve.Hostettler%40wolterskluwer.com%7C756814848743
> 4617407b08d80090b1f2%7C8ac76c91e7f141ffa89c3553b2da2c17%7C0%7C0%7C6372
> 59968758519763sdata=1Uzz8DLO%2B9nd1FPQ14efFeL35QsYE6tT3BvhIKf03FQ
> %3Dreserved=0> there is no one assigned for marshalling.
>
> On the details:
> The compression is disabled by default as it is not compatible with 
> objects previously marshalled.
>
> My approach was to go a bit beyond the JIRA. No only do I remove the 
> indexes to null fields in the footer, I also remove the 0x65 in the 
> objects. I did not remove them fro the collections and arrays because 
> they are using absolute positioning.
>
> I gain between 5% to 20% depending of my test cases. Obviously the 
> smaller the object and the higher the number of nulls, the higher the 
> compression rate.
>
> Based on that I can quite easily add var int compression which is
> IGNITE-6418 and should significantly increase the compression rate 
> with a lot of integers and longs when only using small numbers.
>
> Next step is to add JMH micro-benchmark to check the impact in terms 
> of performances.
>
>
> Example on a simple object w/ null compaction
>
> Length=55 FooterPosition=50
> 0x67 // ValueType
> 0x01 // FormatVersion
> 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 
> compactFooter=true
> 0x78 0x66 0xbe 0x44 //TypeId
> 0xf9 0xcd 0x07 0x57 //Hashcode
> 0x37 0x00 0x00 0x00 //Length
> 0x3d 0xa8 0x15 0xe4 //SchemaId
> 0x32 0x00 0x00 0x00 //Footer position = 50
> 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 
> 0x00
> 0x61 0x62 0x63 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63 Footer length=5
> 0x18 0x1d 0x22 0x2a 0x47
>
> and w/o null compaction
> Length=60 FooterPosition=53
> 0x67 // ValueType
> 0x01 // FormatVersion
> 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 
> compactFooter=true
> 0x78 0x66 0xbe 0x44 //TypeId
> 0xa4 0x43 0x0e 0xf5 //Hashcode
> 0x3c 0x00 0x00 0x00 //Length
> 0x3d 0xa8 0x15 0xe4 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Pavel Pereslegin]

Nikolay, Alexei,

thanks for your suggestions.

Offline re-encryption does not seem so simple, we need to read/replace
the existing encryption keys on all nodes (therefore, we should be
able to read/write metastore/WAL and exchange data between the
baseline nodes). Re-encryption in maintenance mode (for example, in a
stable read-only cluster) will be simple, but it still looks very
inconvenient, at least because users will need to interrupt all
operations.

The main advantage of online "in place" re-encryption is that we'll
support multiple keys for reading, and this procedure does not
directly depend on background re-encryption.

So, the first step is similar to rotating the master key when the new
key was set for writing on all nodes - that’s it, the cache group key
rotation is complete (this is what PCI DSS requires - encrypt new
updates with new keys).
The second step is to re-encrypt the existing data, As I said
previously I thought about scanning all partition pages in some
background mode (store progress on the metapage to continue after
restart), but rebalance approach should also work here if I figure out
how to automate this process.

пн, 25 мая 2020 г. в 12:22, Alexei Scherbakov :
>
>
>
> пн, 25 мая 2020 г. в 12:00, Nikolay Izhikov :
>>
>> > This willl takes us to the re-encryption using full rebalancing
>>
>> Rebalance will require 2x efforts for reencryption
>>
>> 1. Read and send data from supplier node.
>> 2. Reencrypt and write data on demander node.
>>
>> Instead of
>>
>> 1. Read, reencrypt and write data on «demander» node.
>
>
> Usually, reading and sending is not a bottleneck. And don't forget we can run 
> out of WAL history and fall back to full rebalancing with partition eviction 
> eliminating all efforts from offline re-encryption.
>
> On the other side, for a grid having many nodes one-by-one re-encryption can 
> take a long time.
> It should also be possible to re-encrypt all data as fast as possible if, for 
> example, if a load can be switched to another grid, where offline encryption 
> will come in handy.
>
> So, I suggest to implement offline re-encryption and online re-encryption 
> using rebalancing as a first step.
>
> Next step can be online in-place re-encryption. It's important to measure 
> business impact from it on online grid.
>
>>
>>
>>
>> > 25 мая 2020 г., в 11:46, Alexei Scherbakov  
>> > написал(а):
>> >
>> > For me, the one big disadvantage for offline re-encryption is the
>> > possibility to run out of WAL history.
>> > If an re-encryption takes a long time we will get full rebalancing with
>> > partition eviction.
>> > This willl takes us to the re-encryption using full rebalancing, proposed
>> > by me earlier.
>> >
>> >
>> >
>> > пн, 25 мая 2020 г. в 11:27, Nikolay Izhikov :
>> >
>> >>> And definitely this approach is much simplier to implement
>> >>
>> >> I agree.
>> >>
>> >> If we allow to made nodes offline for reencryption then we can implement a
>> >> fully offline procedure:
>> >>
>> >> 1. Stop node.
>> >> 2. Execute some control.sh command that will reencrypt all data without
>> >> starting node
>> >> 3. Start node.
>> >>
>> >> Pavel, can you, please, write it one more time - what disadvantages in
>> >> offline procedure?
>> >>
>> >>> 25 мая 2020 г., в 11:20, Alexei Scherbakov 
>> >> написал(а):
>> >>>
>> >>> And definitely this approach is much simplier to implement because all
>> >>> corner cases are handled by rebalancing code.
>> >>>
>> >>> пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov <
>> >> alexey.scherbak...@gmail.com
>>  :
>> >>>
>>  I mean: serving supply requests.
>> 
>>  пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
>>  alexey.scherbak...@gmail.com>:
>> 
>> > Nikolay,
>> >
>> > Can you explain why such restriction is necessary ?
>> > Most likely having a currently re-encrypting node serving only demand
>> > requests will have least preformance impact on a grid.
>> >
>> > пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
>> >
>> >> Hello, Alexei.
>> >>
>> >> I think we want to implement this feature without nodes restart.
>> >> In the ideal scenario all nodes will stay alive and respond to the
>> >> user
>> >> requests.
>> >>
>> >>> 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>> >> alexey.scherbak...@gmail.com> написал(а):
>> >>>
>> >>> Pavel Pereslegin,
>> >>>
>> >>> I see another opportunity.
>> >>> We can use rebalancing to re-encrypt node data with a new key.
>> >>> It's a trivial procedure for me: stop a node, clear database, change
>> >> a
>> >> key,
>> >>> start node and wait for rebalancing to complete.
>> >>> Data will be re-encrypted during rebalancing.
>> >>>
>> >>> Did I miss something ?
>> >>>
>> >>> пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
>> >>>
>>  Folks,
>> 
>>  Just keeping you informed: I and my colleagues are highly interested
>> >> in TDE
>>  

Re: IGNITE-6499 Compact NULL fields

[Ilya Kasnacheev]

Hello!

I can't help myself but wonder how large of a benefit will it give.  I have
checked the ticket description, it looks the proposed scheme is elaborate
and benefit for non-extreme binary objects rather tiny.

WDYT?

Regards,
-- 
Ilya Kasnacheev


пн, 18 мая 2020 г. в 22:54, steve.hostett...@gmail.com <
steve.hostett...@gmail.com>:

> Hello igniters,
>
> while I would like to help on the calcite because H2 optimiser (or the lack
> thereof) is really killing us, I think that it would be wiser to start by
> contributing on something easier.
>
> Therefore I will tackle another problem that we have which is the memory
> consumption. I stumbled upon this IEP
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements
> <
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-2%3A+Binary+object+format+improvements>
>
> that is about optimising the binary marshaller.
>
> The low hanging fruit seemed to be the null compaction so I decided to
> start
> with it. Though I am sure I do see some hidden complexity.
>
> Here a couple of questions:
> - Can I assign myself IGNITE-6499 and attach a patch?
> - Who can I contact to help with the review. In the following page
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> 
> there is no one assigned for marshalling.
>
> On the details:
> The compression is disabled by default as it is not compatible with objects
> previously marshalled.
>
> My approach was to go a bit beyond the JIRA. No only do I remove the
> indexes
> to null fields in the footer, I also remove the 0x65 in the objects. I did
> not remove them fro the collections and arrays because they are using
> absolute positioning.
>
> I gain between 5% to 20% depending of my test cases. Obviously the smaller
> the object and the higher the number of nulls, the higher the compression
> rate.
>
> Based on that I can quite easily add var int compression which is
> IGNITE-6418 and should significantly increase the compression rate with a
> lot of integers and longs when only using small numbers.
>
> Next step is to add JMH micro-benchmark to check the impact in terms of
> performances.
>
>
> Example on a simple object w/ null compaction
>
> Length=55 FooterPosition=50
> 0x67 // ValueType
> 0x01 // FormatVersion
> 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true
> 0x78 0x66 0xbe 0x44 //TypeId
> 0xf9 0xcd 0x07 0x57 //Hashcode
> 0x37 0x00 0x00 0x00 //Length
> 0x3d 0xa8 0x15 0xe4 //SchemaId
> 0x32 0x00 0x00 0x00 //Footer position = 50
> 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00
> 0x61 0x62 0x63 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63
> Footer length=5
> 0x18 0x1d 0x22 0x2a 0x47
>
> and w/o null compaction
> Length=60 FooterPosition=53
> 0x67 // ValueType
> 0x01 // FormatVersion
> 0x2b 0x00 //Flags userType=true hasSchema=true offset=1 compactFooter=true
> 0x78 0x66 0xbe 0x44 //TypeId
> 0xa4 0x43 0x0e 0xf5 //Hashcode
> 0x3c 0x00 0x00 0x00 //Length
> 0x3d 0xa8 0x15 0xe4 //SchemaId
> 0x35 0x00 0x00 0x00 //Footer position = 53
> 0x03 0x01 0x00 0x00 0x00 0x03 0x01 0x00 0x00 0x00 0x09 0x03 0x00 0x00 0x00
> 0x61 0x62 0x63 0x65 0x65 0x65 0x09 0x03 0x00 0x00 0x00 0x61 0x62 0x63
> Footer length=7
> 0x18 0x1d 0x22 0x2a 0x2b 0x2c 0x2d
>
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: ExecutorServices hide assertions without logging and node stop

[Ivan Pavlukhin]

Maxim,

I looked into the code and checked how do we setup uncaught exception
handlers for internal executors. Indeed it looks not good (OOME leads
to failure handling, other exceptions are silently ignored). Logging
sounds good.

Have doubts about failure handling (e.g. terminating a node). I
remember at least one related discussion [1]. And here are some
statements which causes my doubts:
1. Not only assertion errors are a programming errors evidence, e.g. a
simple NPE often signals the same.
2. Assertions might be disabled in production.
3. Sometimes it seems that assertions are often overused in the Ignite
code. Intermittent concurrency-related assertion errors are not so
uncommon. And it seems impossible to figure out how critical a
particular error is.

[1] 
http://apache-ignite-developers.2346864.n4.nabble.com/Exceptions-thrown-in-IndexingSpi-and-quot-fail-fast-quot-principle-td38904.html

2020-05-24 12:13 GMT+03:00, Maxim Muzafarov :
> Ivan,
>
> Not sure that I've got your point. Let me give some example:
>
> IgniteEx ignite = startGrid();
> ignite.context().getSystemExecutorService().execute(() -> { assert false;
> });
> ignite.context().getSystemExecutorService().submit(() -> { assert false;
> });
>
> In both of these cases there are no exceptions logged and the node not
> stopped (I assume that failing assertion means a code contract is
> violated). Of course, I'll get an assertion error if fut.get() will be
> called but there a lot of cases where it's not. So, I think we must
> log errors here or fail the node.
>
> On Fri, 22 May 2020 at 10:51, Ivan Pavlukhin  wrote:
>>
>> Hi Maxim,
>>
>> I recently had similar troubles related to ExecutorService and
>> exception handling. It was an unexpected behavior for me that
>> exceptions of any kind* thrown from Runnable in a following code:
>> executor.submit(() -> throw new RuntimeException())
>> were silently ignored.
>>
>> I seemed to me really unexpected for me at first. But later on I
>> figured out the cause. There is a subtle detail which I forgot.
>> ExecutorService.submit returns a Future and leaves a responsibility to
>> handle exceptions of any kind to a caller of Future.get. And it was
>> really subtle for me, because I did not use a returned Future at all.
>> Actually I supposed ExecutorService.execute when used
>> ExecutorService.submit.
>>
>> Is it related to the subject? Can ExecutorService.execute help here?
>>
>> * In my case there were exceptions caused by missing classes on a
>> classpath.
>>
>> 2020-05-21 22:22 GMT+03:00, Maxim Muzafarov :
>> > Folks,
>> >
>> > I've created an issue.
>> > https://issues.apache.org/jira/browse/IGNITE-13055
>> >
>> > On Wed, 6 May 2020 at 10:28, Nikolay Izhikov 
>> > wrote:
>> >>
>> >> Hello, Maxim.
>> >>
>> >> I can confirm this itching issue.
>> >> It also happens when some custom Security plugin throws an exception
>> >> while
>> >> processing a communication message.
>> >>
>> >> ```
>> >> UUID newSecSubjId = secSubjId != null ? secSubjId : nodeId;
>> >>
>> >> try (OperationSecurityContext s =
>> >> ctx.security().withContext(newSecSubjId)) {
>> >> lsnr.onMessage(nodeId, msg, plc);
>> >> }
>> >> finally {
>> >> if (change)
>> >> CUR_PLC.set(oldPlc);
>> >> }
>> >> ```
>> >>
>> >> If an exception thrown from `withContext` it is hidden from the user.
>> >>
>> >> > 4 мая 2020 г., в 19:15, Maxim Muzafarov 
>> >> > написал(а):
>> >> >
>> >> > Igniters,
>> >> >
>> >> >
>> >> > I've spent a couple of days in debug mode and found that some of the
>> >> > configured ExecutorServices of an Ignite instance completely hide
>> >> > assertion errors without any logging and node killing. This may
>> >> > violate some internal guarantees and hide serious errors.
>> >> >
>> >> > Let's consider, for instance, GridDhtPartitionsExchangeFuture [1].
>> >> > It
>> >> > has three places of submitting some Runnable on system executor
>> >> > service. If an assertion error (or even any uncached exception)
>> >> > occurs
>> >> > in the code block below it will be swallowed without logging,
>> >> > exchange
>> >> > future completion or node stoping.
>> >> >
>> >> > cctx.kernalContext().getSystemExecutorService().submit(new Runnable()
>> >> > {
>> >> >@Override public void run() {
>> >> >sendPartitions(newCrd);
>> >> >}
>> >> > });
>> >> >
>> >> > I've checked that these executor services and most of them
>> >> > configured
>> >> > to catch only OutOfMemoryError.
>> >> >
>> >> > Should we consider catching AssertionErrors as well and treat them
>> >> > as
>> >> > CRITICAL_ERRORS for the Failure Handler?
>> >> > Should we log uncached errors on each of them?
>> >> >
>> >> >
>> >> > The most important list of executor services configured with OOM
>> >> > handler
>> >> > only:
>> >> > execSvc
>> >> > svcExecSvc
>> >> > sysExecSvc
>> >> > p2pExecSvc
>> >> > restExecSvc
>> >> > utilityCacheExecSvc
>> >> > affExecSvc
>> >> > qryExecSvc
>> >> >
>> >> > [1]
>> >> > 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Alexei Scherbakov]

пн, 25 мая 2020 г. в 12:00, Nikolay Izhikov :

> > This willl takes us to the re-encryption using full rebalancing
>
> Rebalance will require 2x efforts for reencryption
>
> 1. Read and send data from supplier node.
> 2. Reencrypt and write data on demander node.
>
> Instead of
>
> 1. Read, reencrypt and write data on «demander» node.
>

Usually, reading and sending is not a bottleneck. And don't forget we can
run out of WAL history and fall back to full rebalancing with partition
eviction eliminating all efforts from offline re-encryption.

On the other side, for a grid having many nodes one-by-one re-encryption
can take a long time.
It should also be possible to re-encrypt all data as fast as possible if,
for example, if a load can be switched to another grid, where offline
encryption will come in handy.

So, I suggest to implement offline re-encryption and online re-encryption
using rebalancing as a first step.

Next step can be online in-place re-encryption. It's important to measure
business impact from it on online grid.


>
>
> > 25 мая 2020 г., в 11:46, Alexei Scherbakov 
> написал(а):
> >
> > For me, the one big disadvantage for offline re-encryption is the
> > possibility to run out of WAL history.
> > If an re-encryption takes a long time we will get full rebalancing with
> > partition eviction.
> > This willl takes us to the re-encryption using full rebalancing, proposed
> > by me earlier.
> >
> >
> >
> > пн, 25 мая 2020 г. в 11:27, Nikolay Izhikov :
> >
> >>> And definitely this approach is much simplier to implement
> >>
> >> I agree.
> >>
> >> If we allow to made nodes offline for reencryption then we can
> implement a
> >> fully offline procedure:
> >>
> >> 1. Stop node.
> >> 2. Execute some control.sh command that will reencrypt all data without
> >> starting node
> >> 3. Start node.
> >>
> >> Pavel, can you, please, write it one more time - what disadvantages in
> >> offline procedure?
> >>
> >>> 25 мая 2020 г., в 11:20, Alexei Scherbakov <
> alexey.scherbak...@gmail.com>
> >> написал(а):
> >>>
> >>> And definitely this approach is much simplier to implement because all
> >>> corner cases are handled by rebalancing code.
> >>>
> >>> пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov <
> >> alexey.scherbak...@gmail.com
>  :
> >>>
>  I mean: serving supply requests.
> 
>  пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
>  alexey.scherbak...@gmail.com>:
> 
> > Nikolay,
> >
> > Can you explain why such restriction is necessary ?
> > Most likely having a currently re-encrypting node serving only demand
> > requests will have least preformance impact on a grid.
> >
> > пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
> >
> >> Hello, Alexei.
> >>
> >> I think we want to implement this feature without nodes restart.
> >> In the ideal scenario all nodes will stay alive and respond to the
> >> user
> >> requests.
> >>
> >>> 24 мая 2020 г., в 15:24, Alexei Scherbakov <
> >> alexey.scherbak...@gmail.com> написал(а):
> >>>
> >>> Pavel Pereslegin,
> >>>
> >>> I see another opportunity.
> >>> We can use rebalancing to re-encrypt node data with a new key.
> >>> It's a trivial procedure for me: stop a node, clear database,
> change
> >> a
> >> key,
> >>> start node and wait for rebalancing to complete.
> >>> Data will be re-encrypted during rebalancing.
> >>>
> >>> Did I miss something ?
> >>>
> >>> пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
> >>>
>  Folks,
> 
>  Just keeping you informed: I and my colleagues are highly
> interested
> >> in TDE
>  in general and keys rotations specifically, but we don't have
> enough
> >> time
>  so far.
>  We'll dive into this feature and participate in reviews next
> month.
> 
>  --
>  Best Regards,
>  Ivan Rakov
> 
>  On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin <
> xxt...@gmail.com
> >>>
>  wrote:
> 
> > Hello, Alexey.
> >
> >> is the encryption key for the data the same on all nodes in the
>  cluster?
> > Yes, each encrypted cache group has its own encryption key, the
> key
> >> is
> > the same on all nodes.
> >
> >> Clearly, during the re-encryption there will exist pages
> >> encrypted with both new and old keys at the same time.
> > Yes, there will be pages encrypted with different keys at the
> same
> >> time.
> > Currently, we only store one key for one cache group. To rotate a
> >> key,
> > at a certain point in time it is necessary to support several
> keys
> >> (at
> > least for reading the WAL).
> > For the "in place" strategy, we'll store the encryption key
> >> identifier
> > on each encrypted page (we currently have some unused space on
> > encrypted 

[jira] [Created] (IGNITE-13068) SQL: use cache.localSize instead of index scan to calculate row count statistic

[Taras Ledkov (Jira)]

Taras Ledkov created IGNITE-13068:
-

 Summary: SQL: use cache.localSize instead of index scan to 
calculate row count statistic 
 Key: IGNITE-13068
 URL: https://issues.apache.org/jira/browse/IGNITE-13068
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.9


The scan of index is slow on big data set on setup where disc i/o requires warm 
up.
This is the reason that restarting the node can take a long time.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Nikolay Izhikov]

> This willl takes us to the re-encryption using full rebalancing

Rebalance will require 2x efforts for reencryption

1. Read and send data from supplier node.
2. Reencrypt and write data on demander node.

Instead of

1. Read, reencrypt and write data on «demander» node.


> 25 мая 2020 г., в 11:46, Alexei Scherbakov  
> написал(а):
> 
> For me, the one big disadvantage for offline re-encryption is the
> possibility to run out of WAL history.
> If an re-encryption takes a long time we will get full rebalancing with
> partition eviction.
> This willl takes us to the re-encryption using full rebalancing, proposed
> by me earlier.
> 
> 
> 
> пн, 25 мая 2020 г. в 11:27, Nikolay Izhikov :
> 
>>> And definitely this approach is much simplier to implement
>> 
>> I agree.
>> 
>> If we allow to made nodes offline for reencryption then we can implement a
>> fully offline procedure:
>> 
>> 1. Stop node.
>> 2. Execute some control.sh command that will reencrypt all data without
>> starting node
>> 3. Start node.
>> 
>> Pavel, can you, please, write it one more time - what disadvantages in
>> offline procedure?
>> 
>>> 25 мая 2020 г., в 11:20, Alexei Scherbakov 
>> написал(а):
>>> 
>>> And definitely this approach is much simplier to implement because all
>>> corner cases are handled by rebalancing code.
>>> 
>>> пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com
 :
>>> 
 I mean: serving supply requests.
 
 пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
 alexey.scherbak...@gmail.com>:
 
> Nikolay,
> 
> Can you explain why such restriction is necessary ?
> Most likely having a currently re-encrypting node serving only demand
> requests will have least preformance impact on a grid.
> 
> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
> 
>> Hello, Alexei.
>> 
>> I think we want to implement this feature without nodes restart.
>> In the ideal scenario all nodes will stay alive and respond to the
>> user
>> requests.
>> 
>>> 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com> написал(а):
>>> 
>>> Pavel Pereslegin,
>>> 
>>> I see another opportunity.
>>> We can use rebalancing to re-encrypt node data with a new key.
>>> It's a trivial procedure for me: stop a node, clear database, change
>> a
>> key,
>>> start node and wait for rebalancing to complete.
>>> Data will be re-encrypted during rebalancing.
>>> 
>>> Did I miss something ?
>>> 
>>> пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
>>> 
 Folks,
 
 Just keeping you informed: I and my colleagues are highly interested
>> in TDE
 in general and keys rotations specifically, but we don't have enough
>> time
 so far.
 We'll dive into this feature and participate in reviews next month.
 
 --
 Best Regards,
 Ivan Rakov
 
 On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin >> 
 wrote:
 
> Hello, Alexey.
> 
>> is the encryption key for the data the same on all nodes in the
 cluster?
> Yes, each encrypted cache group has its own encryption key, the key
>> is
> the same on all nodes.
> 
>> Clearly, during the re-encryption there will exist pages
>> encrypted with both new and old keys at the same time.
> Yes, there will be pages encrypted with different keys at the same
>> time.
> Currently, we only store one key for one cache group. To rotate a
>> key,
> at a certain point in time it is necessary to support several keys
>> (at
> least for reading the WAL).
> For the "in place" strategy, we'll store the encryption key
>> identifier
> on each encrypted page (we currently have some unused space on
> encrypted page, so I don't expect any memory overhead here). Thus,
>> we
> will have several keys for reading and one key for writing. I
>> assume
> that the old key will be automatically deleted when a specific WAL
> segment is deleted (and re-encryption is finished).
> 
>> Will a node continue to re-encrypt the data after it restarts?
> Yes.
> 
>> If a node goes down during the re-encryption, but the rest of the
>> cluster finishes re-encryption, will we consider the procedure
 complete?
> I'm not sure, but it looks like the key rotation is complete when
>> we
> set the new key on all nodes so that the updates will be encrypted
> with the new key (as required by PCI DSS).
> Status of re-encryption can be obtained separately (locally or
>> cluster
> wide).
> 
> I forgot to mention that with “in place” re-encryption it will be
> impossible to quickly cancel re-encryption, because by 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Alexei Scherbakov]

For me, the one big disadvantage for offline re-encryption is the
possibility to run out of WAL history.
If an re-encryption takes a long time we will get full rebalancing with
partition eviction.
This willl takes us to the re-encryption using full rebalancing, proposed
by me earlier.



пн, 25 мая 2020 г. в 11:27, Nikolay Izhikov :

> > And definitely this approach is much simplier to implement
>
> I agree.
>
> If we allow to made nodes offline for reencryption then we can implement a
> fully offline procedure:
>
> 1. Stop node.
> 2. Execute some control.sh command that will reencrypt all data without
> starting node
> 3. Start node.
>
> Pavel, can you, please, write it one more time - what disadvantages in
> offline procedure?
>
> > 25 мая 2020 г., в 11:20, Alexei Scherbakov 
> написал(а):
> >
> > And definitely this approach is much simplier to implement because all
> > corner cases are handled by rebalancing code.
> >
> > пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov <
> alexey.scherbak...@gmail.com
> >> :
> >
> >> I mean: serving supply requests.
> >>
> >> пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
> >> alexey.scherbak...@gmail.com>:
> >>
> >>> Nikolay,
> >>>
> >>> Can you explain why such restriction is necessary ?
> >>> Most likely having a currently re-encrypting node serving only demand
> >>> requests will have least preformance impact on a grid.
> >>>
> >>> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
> >>>
>  Hello, Alexei.
> 
>  I think we want to implement this feature without nodes restart.
>  In the ideal scenario all nodes will stay alive and respond to the
> user
>  requests.
> 
> > 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>  alexey.scherbak...@gmail.com> написал(а):
> >
> > Pavel Pereslegin,
> >
> > I see another opportunity.
> > We can use rebalancing to re-encrypt node data with a new key.
> > It's a trivial procedure for me: stop a node, clear database, change
> a
>  key,
> > start node and wait for rebalancing to complete.
> > Data will be re-encrypted during rebalancing.
> >
> > Did I miss something ?
> >
> > пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
> >
> >> Folks,
> >>
> >> Just keeping you informed: I and my colleagues are highly interested
>  in TDE
> >> in general and keys rotations specifically, but we don't have enough
>  time
> >> so far.
> >> We'll dive into this feature and participate in reviews next month.
> >>
> >> --
> >> Best Regards,
> >> Ivan Rakov
> >>
> >> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin  >
> >> wrote:
> >>
> >>> Hello, Alexey.
> >>>
>  is the encryption key for the data the same on all nodes in the
> >> cluster?
> >>> Yes, each encrypted cache group has its own encryption key, the key
>  is
> >>> the same on all nodes.
> >>>
>  Clearly, during the re-encryption there will exist pages
>  encrypted with both new and old keys at the same time.
> >>> Yes, there will be pages encrypted with different keys at the same
>  time.
> >>> Currently, we only store one key for one cache group. To rotate a
>  key,
> >>> at a certain point in time it is necessary to support several keys
>  (at
> >>> least for reading the WAL).
> >>> For the "in place" strategy, we'll store the encryption key
>  identifier
> >>> on each encrypted page (we currently have some unused space on
> >>> encrypted page, so I don't expect any memory overhead here). Thus,
> we
> >>> will have several keys for reading and one key for writing. I
> assume
> >>> that the old key will be automatically deleted when a specific WAL
> >>> segment is deleted (and re-encryption is finished).
> >>>
>  Will a node continue to re-encrypt the data after it restarts?
> >>> Yes.
> >>>
>  If a node goes down during the re-encryption, but the rest of the
>  cluster finishes re-encryption, will we consider the procedure
> >> complete?
> >>> I'm not sure, but it looks like the key rotation is complete when
> we
> >>> set the new key on all nodes so that the updates will be encrypted
> >>> with the new key (as required by PCI DSS).
> >>> Status of re-encryption can be obtained separately (locally or
>  cluster
> >>> wide).
> >>>
> >>> I forgot to mention that with “in place” re-encryption it will be
> >>> impossible to quickly cancel re-encryption, because by canceling we
> >>> mean re-encryption with the old key.
> >>>
>  How do you see the whole key rotation procedure will work?
> >>> Initial design for re-encryption with "partition copying" is
>  described
> >>> here [1]. I'll prepare detailed design for "in place" re-encryption
>  if
> >>> we'll go this way. In short, send the new encryption key
>  cluster-wide,
> >>> each 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Nikolay Izhikov]

> And definitely this approach is much simplier to implement

I agree.

If we allow to made nodes offline for reencryption then we can implement a 
fully offline procedure:

1. Stop node.
2. Execute some control.sh command that will reencrypt all data without 
starting node
3. Start node.

Pavel, can you, please, write it one more time - what disadvantages in offline 
procedure?

> 25 мая 2020 г., в 11:20, Alexei Scherbakov  
> написал(а):
> 
> And definitely this approach is much simplier to implement because all
> corner cases are handled by rebalancing code.
> 
> пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov > :
> 
>> I mean: serving supply requests.
>> 
>> пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com>:
>> 
>>> Nikolay,
>>> 
>>> Can you explain why such restriction is necessary ?
>>> Most likely having a currently re-encrypting node serving only demand
>>> requests will have least preformance impact on a grid.
>>> 
>>> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
>>> 
 Hello, Alexei.
 
 I think we want to implement this feature without nodes restart.
 In the ideal scenario all nodes will stay alive and respond to the user
 requests.
 
> 24 мая 2020 г., в 15:24, Alexei Scherbakov <
 alexey.scherbak...@gmail.com> написал(а):
> 
> Pavel Pereslegin,
> 
> I see another opportunity.
> We can use rebalancing to re-encrypt node data with a new key.
> It's a trivial procedure for me: stop a node, clear database, change a
 key,
> start node and wait for rebalancing to complete.
> Data will be re-encrypted during rebalancing.
> 
> Did I miss something ?
> 
> пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
> 
>> Folks,
>> 
>> Just keeping you informed: I and my colleagues are highly interested
 in TDE
>> in general and keys rotations specifically, but we don't have enough
 time
>> so far.
>> We'll dive into this feature and participate in reviews next month.
>> 
>> --
>> Best Regards,
>> Ivan Rakov
>> 
>> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
>> wrote:
>> 
>>> Hello, Alexey.
>>> 
 is the encryption key for the data the same on all nodes in the
>> cluster?
>>> Yes, each encrypted cache group has its own encryption key, the key
 is
>>> the same on all nodes.
>>> 
 Clearly, during the re-encryption there will exist pages
 encrypted with both new and old keys at the same time.
>>> Yes, there will be pages encrypted with different keys at the same
 time.
>>> Currently, we only store one key for one cache group. To rotate a
 key,
>>> at a certain point in time it is necessary to support several keys
 (at
>>> least for reading the WAL).
>>> For the "in place" strategy, we'll store the encryption key
 identifier
>>> on each encrypted page (we currently have some unused space on
>>> encrypted page, so I don't expect any memory overhead here). Thus, we
>>> will have several keys for reading and one key for writing. I assume
>>> that the old key will be automatically deleted when a specific WAL
>>> segment is deleted (and re-encryption is finished).
>>> 
 Will a node continue to re-encrypt the data after it restarts?
>>> Yes.
>>> 
 If a node goes down during the re-encryption, but the rest of the
 cluster finishes re-encryption, will we consider the procedure
>> complete?
>>> I'm not sure, but it looks like the key rotation is complete when we
>>> set the new key on all nodes so that the updates will be encrypted
>>> with the new key (as required by PCI DSS).
>>> Status of re-encryption can be obtained separately (locally or
 cluster
>>> wide).
>>> 
>>> I forgot to mention that with “in place” re-encryption it will be
>>> impossible to quickly cancel re-encryption, because by canceling we
>>> mean re-encryption with the old key.
>>> 
 How do you see the whole key rotation procedure will work?
>>> Initial design for re-encryption with "partition copying" is
 described
>>> here [1]. I'll prepare detailed design for "in place" re-encryption
 if
>>> we'll go this way. In short, send the new encryption key
 cluster-wide,
>>> each node adds a new key and starts background re-encryption.
>>> 
>>> [1]
>>> 
>> 
 https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
>>> .
>>> 
>>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk <
 alexey.goncha...@gmail.com
>>> :
 
 Pavel, Anton,
 
 How do you see the whole key rotation procedure will work? Clearly,
>>> during
 the re-encryption there will exist pages encrypted with both new and
>> old
 keys at the same 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Nikolay Izhikov]

> Can you explain why such restriction is necessary ?

Reencryption should have a minimum impact on the cluster.

> Most likely having a currently re-encrypting node serving only demand 
> requests will have least preformance impact on a grid.

Current design assumes that reencryption will started on all noes 
simultaneously.


Makes sense?

> 25 мая 2020 г., в 11:16, Alexei Scherbakov  
> написал(а):
> 
> I mean: serving supply requests.
> 
> пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov > :
> 
>> Nikolay,
>> 
>> Can you explain why such restriction is necessary ?
>> Most likely having a currently re-encrypting node serving only demand
>> requests will have least preformance impact on a grid.
>> 
>> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
>> 
>>> Hello, Alexei.
>>> 
>>> I think we want to implement this feature without nodes restart.
>>> In the ideal scenario all nodes will stay alive and respond to the user
>>> requests.
>>> 
 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com> написал(а):
 
 Pavel Pereslegin,
 
 I see another opportunity.
 We can use rebalancing to re-encrypt node data with a new key.
 It's a trivial procedure for me: stop a node, clear database, change a
>>> key,
 start node and wait for rebalancing to complete.
 Data will be re-encrypted during rebalancing.
 
 Did I miss something ?
 
 пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
 
> Folks,
> 
> Just keeping you informed: I and my colleagues are highly interested
>>> in TDE
> in general and keys rotations specifically, but we don't have enough
>>> time
> so far.
> We'll dive into this feature and participate in reviews next month.
> 
> --
> Best Regards,
> Ivan Rakov
> 
> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
> wrote:
> 
>> Hello, Alexey.
>> 
>>> is the encryption key for the data the same on all nodes in the
> cluster?
>> Yes, each encrypted cache group has its own encryption key, the key is
>> the same on all nodes.
>> 
>>> Clearly, during the re-encryption there will exist pages
>>> encrypted with both new and old keys at the same time.
>> Yes, there will be pages encrypted with different keys at the same
>>> time.
>> Currently, we only store one key for one cache group. To rotate a key,
>> at a certain point in time it is necessary to support several keys (at
>> least for reading the WAL).
>> For the "in place" strategy, we'll store the encryption key identifier
>> on each encrypted page (we currently have some unused space on
>> encrypted page, so I don't expect any memory overhead here). Thus, we
>> will have several keys for reading and one key for writing. I assume
>> that the old key will be automatically deleted when a specific WAL
>> segment is deleted (and re-encryption is finished).
>> 
>>> Will a node continue to re-encrypt the data after it restarts?
>> Yes.
>> 
>>> If a node goes down during the re-encryption, but the rest of the
>>> cluster finishes re-encryption, will we consider the procedure
> complete?
>> I'm not sure, but it looks like the key rotation is complete when we
>> set the new key on all nodes so that the updates will be encrypted
>> with the new key (as required by PCI DSS).
>> Status of re-encryption can be obtained separately (locally or cluster
>> wide).
>> 
>> I forgot to mention that with “in place” re-encryption it will be
>> impossible to quickly cancel re-encryption, because by canceling we
>> mean re-encryption with the old key.
>> 
>>> How do you see the whole key rotation procedure will work?
>> Initial design for re-encryption with "partition copying" is described
>> here [1]. I'll prepare detailed design for "in place" re-encryption if
>> we'll go this way. In short, send the new encryption key cluster-wide,
>> each node adds a new key and starts background re-encryption.
>> 
>> [1]
>> 
> 
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
>> .
>> 
>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk <
>>> alexey.goncha...@gmail.com
>> :
>>> 
>>> Pavel, Anton,
>>> 
>>> How do you see the whole key rotation procedure will work? Clearly,
>> during
>>> the re-encryption there will exist pages encrypted with both new and
> old
>>> keys at the same time. Will a node continue to re-encrypt the data
> after
>> it
>>> restarts? If a node goes down during the re-encryption, but the rest
>>> of
>> the
>>> cluster finishes re-encryption, will we consider the procedure
> complete?
>> By
>>> the way, is the encryption key for the data the same on all nodes in
> the
>>> cluster?
>>> 
>>> чт, 14 мая 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Alexei Scherbakov]

And definitely this approach is much simplier to implement because all
corner cases are handled by rebalancing code.

пн, 25 мая 2020 г. в 11:16, Alexei Scherbakov :

> I mean: serving supply requests.
>
> пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov <
> alexey.scherbak...@gmail.com>:
>
>> Nikolay,
>>
>> Can you explain why such restriction is necessary ?
>> Most likely having a currently re-encrypting node serving only demand
>> requests will have least preformance impact on a grid.
>>
>> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
>>
>>> Hello, Alexei.
>>>
>>> I think we want to implement this feature without nodes restart.
>>> In the ideal scenario all nodes will stay alive and respond to the user
>>> requests.
>>>
>>> > 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>>> alexey.scherbak...@gmail.com> написал(а):
>>> >
>>> > Pavel Pereslegin,
>>> >
>>> > I see another opportunity.
>>> > We can use rebalancing to re-encrypt node data with a new key.
>>> > It's a trivial procedure for me: stop a node, clear database, change a
>>> key,
>>> > start node and wait for rebalancing to complete.
>>> > Data will be re-encrypted during rebalancing.
>>> >
>>> > Did I miss something ?
>>> >
>>> > пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
>>> >
>>> >> Folks,
>>> >>
>>> >> Just keeping you informed: I and my colleagues are highly interested
>>> in TDE
>>> >> in general and keys rotations specifically, but we don't have enough
>>> time
>>> >> so far.
>>> >> We'll dive into this feature and participate in reviews next month.
>>> >>
>>> >> --
>>> >> Best Regards,
>>> >> Ivan Rakov
>>> >>
>>> >> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
>>> >> wrote:
>>> >>
>>> >>> Hello, Alexey.
>>> >>>
>>>  is the encryption key for the data the same on all nodes in the
>>> >> cluster?
>>> >>> Yes, each encrypted cache group has its own encryption key, the key
>>> is
>>> >>> the same on all nodes.
>>> >>>
>>>  Clearly, during the re-encryption there will exist pages
>>>  encrypted with both new and old keys at the same time.
>>> >>> Yes, there will be pages encrypted with different keys at the same
>>> time.
>>> >>> Currently, we only store one key for one cache group. To rotate a
>>> key,
>>> >>> at a certain point in time it is necessary to support several keys
>>> (at
>>> >>> least for reading the WAL).
>>> >>> For the "in place" strategy, we'll store the encryption key
>>> identifier
>>> >>> on each encrypted page (we currently have some unused space on
>>> >>> encrypted page, so I don't expect any memory overhead here). Thus, we
>>> >>> will have several keys for reading and one key for writing. I assume
>>> >>> that the old key will be automatically deleted when a specific WAL
>>> >>> segment is deleted (and re-encryption is finished).
>>> >>>
>>>  Will a node continue to re-encrypt the data after it restarts?
>>> >>> Yes.
>>> >>>
>>>  If a node goes down during the re-encryption, but the rest of the
>>>  cluster finishes re-encryption, will we consider the procedure
>>> >> complete?
>>> >>> I'm not sure, but it looks like the key rotation is complete when we
>>> >>> set the new key on all nodes so that the updates will be encrypted
>>> >>> with the new key (as required by PCI DSS).
>>> >>> Status of re-encryption can be obtained separately (locally or
>>> cluster
>>> >>> wide).
>>> >>>
>>> >>> I forgot to mention that with “in place” re-encryption it will be
>>> >>> impossible to quickly cancel re-encryption, because by canceling we
>>> >>> mean re-encryption with the old key.
>>> >>>
>>>  How do you see the whole key rotation procedure will work?
>>> >>> Initial design for re-encryption with "partition copying" is
>>> described
>>> >>> here [1]. I'll prepare detailed design for "in place" re-encryption
>>> if
>>> >>> we'll go this way. In short, send the new encryption key
>>> cluster-wide,
>>> >>> each node adds a new key and starts background re-encryption.
>>> >>>
>>> >>> [1]
>>> >>>
>>> >>
>>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
>>> >>> .
>>> >>>
>>> >>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk <
>>> alexey.goncha...@gmail.com
>>> >>> :
>>> 
>>>  Pavel, Anton,
>>> 
>>>  How do you see the whole key rotation procedure will work? Clearly,
>>> >>> during
>>>  the re-encryption there will exist pages encrypted with both new and
>>> >> old
>>>  keys at the same time. Will a node continue to re-encrypt the data
>>> >> after
>>> >>> it
>>>  restarts? If a node goes down during the re-encryption, but the
>>> rest of
>>> >>> the
>>>  cluster finishes re-encryption, will we consider the procedure
>>> >> complete?
>>> >>> By
>>>  the way, is the encryption key for the data the same on all nodes in
>>> >> the
>>>  cluster?
>>> 
>>>  чт, 14 мая 2020 г. в 11:30, Anton Vinogradov :
>>> 
>>> > +1 to "In place re-encryption".
>>> >
>>> > - It 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Alexei Scherbakov]

I mean: serving supply requests.

пн, 25 мая 2020 г. в 11:15, Alexei Scherbakov :

> Nikolay,
>
> Can you explain why such restriction is necessary ?
> Most likely having a currently re-encrypting node serving only demand
> requests will have least preformance impact on a grid.
>
> пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :
>
>> Hello, Alexei.
>>
>> I think we want to implement this feature without nodes restart.
>> In the ideal scenario all nodes will stay alive and respond to the user
>> requests.
>>
>> > 24 мая 2020 г., в 15:24, Alexei Scherbakov <
>> alexey.scherbak...@gmail.com> написал(а):
>> >
>> > Pavel Pereslegin,
>> >
>> > I see another opportunity.
>> > We can use rebalancing to re-encrypt node data with a new key.
>> > It's a trivial procedure for me: stop a node, clear database, change a
>> key,
>> > start node and wait for rebalancing to complete.
>> > Data will be re-encrypted during rebalancing.
>> >
>> > Did I miss something ?
>> >
>> > пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
>> >
>> >> Folks,
>> >>
>> >> Just keeping you informed: I and my colleagues are highly interested
>> in TDE
>> >> in general and keys rotations specifically, but we don't have enough
>> time
>> >> so far.
>> >> We'll dive into this feature and participate in reviews next month.
>> >>
>> >> --
>> >> Best Regards,
>> >> Ivan Rakov
>> >>
>> >> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
>> >> wrote:
>> >>
>> >>> Hello, Alexey.
>> >>>
>>  is the encryption key for the data the same on all nodes in the
>> >> cluster?
>> >>> Yes, each encrypted cache group has its own encryption key, the key is
>> >>> the same on all nodes.
>> >>>
>>  Clearly, during the re-encryption there will exist pages
>>  encrypted with both new and old keys at the same time.
>> >>> Yes, there will be pages encrypted with different keys at the same
>> time.
>> >>> Currently, we only store one key for one cache group. To rotate a key,
>> >>> at a certain point in time it is necessary to support several keys (at
>> >>> least for reading the WAL).
>> >>> For the "in place" strategy, we'll store the encryption key identifier
>> >>> on each encrypted page (we currently have some unused space on
>> >>> encrypted page, so I don't expect any memory overhead here). Thus, we
>> >>> will have several keys for reading and one key for writing. I assume
>> >>> that the old key will be automatically deleted when a specific WAL
>> >>> segment is deleted (and re-encryption is finished).
>> >>>
>>  Will a node continue to re-encrypt the data after it restarts?
>> >>> Yes.
>> >>>
>>  If a node goes down during the re-encryption, but the rest of the
>>  cluster finishes re-encryption, will we consider the procedure
>> >> complete?
>> >>> I'm not sure, but it looks like the key rotation is complete when we
>> >>> set the new key on all nodes so that the updates will be encrypted
>> >>> with the new key (as required by PCI DSS).
>> >>> Status of re-encryption can be obtained separately (locally or cluster
>> >>> wide).
>> >>>
>> >>> I forgot to mention that with “in place” re-encryption it will be
>> >>> impossible to quickly cancel re-encryption, because by canceling we
>> >>> mean re-encryption with the old key.
>> >>>
>>  How do you see the whole key rotation procedure will work?
>> >>> Initial design for re-encryption with "partition copying" is described
>> >>> here [1]. I'll prepare detailed design for "in place" re-encryption if
>> >>> we'll go this way. In short, send the new encryption key cluster-wide,
>> >>> each node adds a new key and starts background re-encryption.
>> >>>
>> >>> [1]
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
>> >>> .
>> >>>
>> >>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk <
>> alexey.goncha...@gmail.com
>> >>> :
>> 
>>  Pavel, Anton,
>> 
>>  How do you see the whole key rotation procedure will work? Clearly,
>> >>> during
>>  the re-encryption there will exist pages encrypted with both new and
>> >> old
>>  keys at the same time. Will a node continue to re-encrypt the data
>> >> after
>> >>> it
>>  restarts? If a node goes down during the re-encryption, but the rest
>> of
>> >>> the
>>  cluster finishes re-encryption, will we consider the procedure
>> >> complete?
>> >>> By
>>  the way, is the encryption key for the data the same on all nodes in
>> >> the
>>  cluster?
>> 
>>  чт, 14 мая 2020 г. в 11:30, Anton Vinogradov :
>> 
>> > +1 to "In place re-encryption".
>> >
>> > - It has a simple design.
>> > - Clusters under load may require just load to re-encrypt the data.
>> > (Friendly to load).
>> > - Easy to throttle.
>> > - Easy to continue.
>> > - Design compatible with the multi-key architecture.
>> > - It can be optimized to use own WAL buffer and to re-encrypt pages
>> >>> without
>> > restoring them 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Alexei Scherbakov]

Nikolay,

Can you explain why such restriction is necessary ?
Most likely having a currently re-encrypting node serving only demand
requests will have least preformance impact on a grid.

пн, 25 мая 2020 г. в 11:08, Nikolay Izhikov :

> Hello, Alexei.
>
> I think we want to implement this feature without nodes restart.
> In the ideal scenario all nodes will stay alive and respond to the user
> requests.
>
> > 24 мая 2020 г., в 15:24, Alexei Scherbakov 
> написал(а):
> >
> > Pavel Pereslegin,
> >
> > I see another opportunity.
> > We can use rebalancing to re-encrypt node data with a new key.
> > It's a trivial procedure for me: stop a node, clear database, change a
> key,
> > start node and wait for rebalancing to complete.
> > Data will be re-encrypted during rebalancing.
> >
> > Did I miss something ?
> >
> > пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
> >
> >> Folks,
> >>
> >> Just keeping you informed: I and my colleagues are highly interested in
> TDE
> >> in general and keys rotations specifically, but we don't have enough
> time
> >> so far.
> >> We'll dive into this feature and participate in reviews next month.
> >>
> >> --
> >> Best Regards,
> >> Ivan Rakov
> >>
> >> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
> >> wrote:
> >>
> >>> Hello, Alexey.
> >>>
>  is the encryption key for the data the same on all nodes in the
> >> cluster?
> >>> Yes, each encrypted cache group has its own encryption key, the key is
> >>> the same on all nodes.
> >>>
>  Clearly, during the re-encryption there will exist pages
>  encrypted with both new and old keys at the same time.
> >>> Yes, there will be pages encrypted with different keys at the same
> time.
> >>> Currently, we only store one key for one cache group. To rotate a key,
> >>> at a certain point in time it is necessary to support several keys (at
> >>> least for reading the WAL).
> >>> For the "in place" strategy, we'll store the encryption key identifier
> >>> on each encrypted page (we currently have some unused space on
> >>> encrypted page, so I don't expect any memory overhead here). Thus, we
> >>> will have several keys for reading and one key for writing. I assume
> >>> that the old key will be automatically deleted when a specific WAL
> >>> segment is deleted (and re-encryption is finished).
> >>>
>  Will a node continue to re-encrypt the data after it restarts?
> >>> Yes.
> >>>
>  If a node goes down during the re-encryption, but the rest of the
>  cluster finishes re-encryption, will we consider the procedure
> >> complete?
> >>> I'm not sure, but it looks like the key rotation is complete when we
> >>> set the new key on all nodes so that the updates will be encrypted
> >>> with the new key (as required by PCI DSS).
> >>> Status of re-encryption can be obtained separately (locally or cluster
> >>> wide).
> >>>
> >>> I forgot to mention that with “in place” re-encryption it will be
> >>> impossible to quickly cancel re-encryption, because by canceling we
> >>> mean re-encryption with the old key.
> >>>
>  How do you see the whole key rotation procedure will work?
> >>> Initial design for re-encryption with "partition copying" is described
> >>> here [1]. I'll prepare detailed design for "in place" re-encryption if
> >>> we'll go this way. In short, send the new encryption key cluster-wide,
> >>> each node adds a new key and starts background re-encryption.
> >>>
> >>> [1]
> >>>
> >>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
> >>> .
> >>>
> >>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk <
> alexey.goncha...@gmail.com
> >>> :
> 
>  Pavel, Anton,
> 
>  How do you see the whole key rotation procedure will work? Clearly,
> >>> during
>  the re-encryption there will exist pages encrypted with both new and
> >> old
>  keys at the same time. Will a node continue to re-encrypt the data
> >> after
> >>> it
>  restarts? If a node goes down during the re-encryption, but the rest
> of
> >>> the
>  cluster finishes re-encryption, will we consider the procedure
> >> complete?
> >>> By
>  the way, is the encryption key for the data the same on all nodes in
> >> the
>  cluster?
> 
>  чт, 14 мая 2020 г. в 11:30, Anton Vinogradov :
> 
> > +1 to "In place re-encryption".
> >
> > - It has a simple design.
> > - Clusters under load may require just load to re-encrypt the data.
> > (Friendly to load).
> > - Easy to throttle.
> > - Easy to continue.
> > - Design compatible with the multi-key architecture.
> > - It can be optimized to use own WAL buffer and to re-encrypt pages
> >>> without
> > restoring them to on-heap.
> >
> > On Thu, May 14, 2020 at 1:54 AM Pavel Pereslegin 
> >>> wrote:
> >
> >> Hello Igniters.
> >>
> >> Recently, master key rotation for Apache Ignite Transparent Data
> >> Encryption was implemented [1], 

Re: [DISCUSSION] Ignite integration testing framework.

[Nikolay Izhikov]

Hello, 

Branch with duck tape created - 
https://github.com/apache/ignite/tree/ignite-ducktape

Any who are willing to contribute to PoC are welcome.


> 21 мая 2020 г., в 22:33, Nikolay Izhikov  написал(а):
> 
> Hello, Denis.
> 
> There is no rush with these improvements.
> We can wait for Maxim proposal and compare two solutions :)
> 
>> 21 мая 2020 г., в 22:24, Denis Magda  написал(а):
>> 
>> Hi Nikolay,
>> 
>> Thanks for kicking off this conversation and sharing your findings with the
>> results. That's the right initiative. I do agree that Ignite needs to have
>> an integration testing framework with capabilities listed by you.
>> 
>> As we discussed privately, I would only check if instead of
>> Confluent's Ducktape library, we can use an integration testing framework
>> developed by GridGain for testing of Ignite/GridGain clusters. That
>> framework has been battle-tested and might be more convenient for
>> Ignite-specific workloads. Let's wait for @Maksim Shonichev
>>  who promised to join this thread once he finishes
>> preparing the usage examples of the framework. To my knowledge, Max has
>> already been working on that for several days.
>> 
>> -
>> Denis
>> 
>> 
>> On Thu, May 21, 2020 at 12:27 AM Nikolay Izhikov 
>> wrote:
>> 
>>> Hello, Igniters.
>>> 
>>> I created a PoC [1] for the integration tests of Ignite.
>>> 
>>> Let me briefly explain the gap I want to cover:
>>> 
>>> 1. For now, we don’t have a solution for automated testing of Ignite on
>>> «real cluster».
>>> By «real cluster» I mean cluster «like a production»:
>>>   * client and server nodes deployed on different hosts.
>>>   * thin clients perform queries from some other hosts
>>>   * etc.
>>> 
>>> 2. We don’t have a solution for automated benchmarks of some internal
>>> Ignite process
>>>   * PME
>>>   * rebalance.
>>> This means we don’t know - Do we perform rebalance(or PME) in 2.7.0 faster
>>> or slower than in 2.8.0 for the same cluster?
>>> 
>>> 3. We don’t have a solution for automated testing of Ignite integration in
>>> a real-world environment:
>>> Ignite-Spark integration can be taken as an example.
>>> I think some ML solutions also should be tested in real-world deployments.
>>> 
>>> Solution:
>>> 
>>> I propose to use duck tape library from confluent (apache 2.0 license)
>>> I tested it both on the real cluster(Yandex Cloud) and on the local
>>> environment(docker) and it works just fine.
>>> 
>>> PoC contains following services:
>>> 
>>>   * Simple rebalance test:
>>>   Start 2 server nodes,
>>>   Create some data with Ignite client,
>>>   Start one more server node,
>>>   Wait for rebalance finish
>>>   * Simple Ignite-Spark integration test:
>>>   Start 1 Spark master, start 1 Spark worker,
>>>   Start 1 Ignite server node
>>>   Create some data with Ignite client,
>>>   Check data in application that queries it from Spark.
>>> 
>>> All tests are fully automated.
>>> Logs collection works just fine.
>>> You can see an example of the tests report - [4].
>>> 
>>> Pros:
>>> 
>>> * Ability to test local changes(no need to public changes to some remote
>>> repository or similar).
>>> * Ability to parametrize test environment(run the same tests on different
>>> JDK, JVM params, config, etc.)
>>> * Isolation by default so system tests are as reliable as possible.
>>> * Utilities for pulling up and tearing down services easily in clusters in
>>> different environments (e.g. local, custom cluster, Vagrant, K8s, Mesos,
>>> Docker, cloud providers, etc.)
>>> * Easy to write unit tests for distributed systems
>>> * Adopted and successfully used by other distributed open source project -
>>> Apache Kafka.
>>> * Collect results (e.g. logs, console output)
>>> * Report results (e.g. expected conditions met, performance results, etc.)
>>> 
>>> WDYT?
>>> 
>>> [1] https://github.com/nizhikov/ignite/pull/15
>>> [2] https://github.com/confluentinc/ducktape
>>> [3] https://ducktape-docs.readthedocs.io/en/latest/run_tests.html
>>> [4] https://yadi.sk/d/JC8ciJZjrkdndg
> 

Re: [DISCUSS] Best way to re-encrypt existing data (TDE cache key rotation).

[Nikolay Izhikov]

Hello, Alexei.

I think we want to implement this feature without nodes restart.
In the ideal scenario all nodes will stay alive and respond to the user 
requests.

> 24 мая 2020 г., в 15:24, Alexei Scherbakov  
> написал(а):
> 
> Pavel Pereslegin,
> 
> I see another opportunity.
> We can use rebalancing to re-encrypt node data with a new key.
> It's a trivial procedure for me: stop a node, clear database, change a key,
> start node and wait for rebalancing to complete.
> Data will be re-encrypted during rebalancing.
> 
> Did I miss something ?
> 
> пт, 22 мая 2020 г. в 16:14, Ivan Rakov :
> 
>> Folks,
>> 
>> Just keeping you informed: I and my colleagues are highly interested in TDE
>> in general and keys rotations specifically, but we don't have enough time
>> so far.
>> We'll dive into this feature and participate in reviews next month.
>> 
>> --
>> Best Regards,
>> Ivan Rakov
>> 
>> On Sun, May 17, 2020 at 10:51 PM Pavel Pereslegin 
>> wrote:
>> 
>>> Hello, Alexey.
>>> 
 is the encryption key for the data the same on all nodes in the
>> cluster?
>>> Yes, each encrypted cache group has its own encryption key, the key is
>>> the same on all nodes.
>>> 
 Clearly, during the re-encryption there will exist pages
 encrypted with both new and old keys at the same time.
>>> Yes, there will be pages encrypted with different keys at the same time.
>>> Currently, we only store one key for one cache group. To rotate a key,
>>> at a certain point in time it is necessary to support several keys (at
>>> least for reading the WAL).
>>> For the "in place" strategy, we'll store the encryption key identifier
>>> on each encrypted page (we currently have some unused space on
>>> encrypted page, so I don't expect any memory overhead here). Thus, we
>>> will have several keys for reading and one key for writing. I assume
>>> that the old key will be automatically deleted when a specific WAL
>>> segment is deleted (and re-encryption is finished).
>>> 
 Will a node continue to re-encrypt the data after it restarts?
>>> Yes.
>>> 
 If a node goes down during the re-encryption, but the rest of the
 cluster finishes re-encryption, will we consider the procedure
>> complete?
>>> I'm not sure, but it looks like the key rotation is complete when we
>>> set the new key on all nodes so that the updates will be encrypted
>>> with the new key (as required by PCI DSS).
>>> Status of re-encryption can be obtained separately (locally or cluster
>>> wide).
>>> 
>>> I forgot to mention that with “in place” re-encryption it will be
>>> impossible to quickly cancel re-encryption, because by canceling we
>>> mean re-encryption with the old key.
>>> 
 How do you see the whole key rotation procedure will work?
>>> Initial design for re-encryption with "partition copying" is described
>>> here [1]. I'll prepare detailed design for "in place" re-encryption if
>>> we'll go this way. In short, send the new encryption key cluster-wide,
>>> each node adds a new key and starts background re-encryption.
>>> 
>>> [1]
>>> 
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=95652384#TDE.Phase-3.Cachekeyrotation.-Copywithre-encryptiondesign
>>> .
>>> 
>>> вс, 17 мая 2020 г. в 18:35, Alexey Goncharuk >> :
 
 Pavel, Anton,
 
 How do you see the whole key rotation procedure will work? Clearly,
>>> during
 the re-encryption there will exist pages encrypted with both new and
>> old
 keys at the same time. Will a node continue to re-encrypt the data
>> after
>>> it
 restarts? If a node goes down during the re-encryption, but the rest of
>>> the
 cluster finishes re-encryption, will we consider the procedure
>> complete?
>>> By
 the way, is the encryption key for the data the same on all nodes in
>> the
 cluster?
 
 чт, 14 мая 2020 г. в 11:30, Anton Vinogradov :
 
> +1 to "In place re-encryption".
> 
> - It has a simple design.
> - Clusters under load may require just load to re-encrypt the data.
> (Friendly to load).
> - Easy to throttle.
> - Easy to continue.
> - Design compatible with the multi-key architecture.
> - It can be optimized to use own WAL buffer and to re-encrypt pages
>>> without
> restoring them to on-heap.
> 
> On Thu, May 14, 2020 at 1:54 AM Pavel Pereslegin 
>>> wrote:
> 
>> Hello Igniters.
>> 
>> Recently, master key rotation for Apache Ignite Transparent Data
>> Encryption was implemented [1], but some security standards (PCI
>> DSS
>> at least) require rotation of all encryption keys [2]. Currently,
>> encryption occurs when reading/writing pages to disk, cache
>>> encryption
>> keys are stored in metastore.
>> 
>> I'm going to contribute cache encryption key rotation and want to
>> consult what is the best way to re-encrypting existing data, I see
>>> two
>> different strategies.
>> 
>> 1. In place re-encryption:
>> Using the old key, 

Apache Ignite and Mono

[Ilya Kasnacheev]

Hello!

Well, while we ship build-mono.sh I'll try to run it every time :) I'm not
running tests with it, just smoke testing to see if anything happens at all.

I admit it looks like Mono no longer has an ambition to be the stand-alone
dotnet runtime for Linux.

Regards,
-- 
Ilya Kasnacheev


пт, 22 мая 2020 г. в 19:47, Pavel Tupitsyn :

> Ilya,
>
> Mono 6.x is broken [1]. The fix is merged, but only included in nightly
> builds [2].
> Mono 5.x works.
>
> Ignite.NET does not have official Mono support.
> Please use .NET Core 2+ to test Ignite.NET (on any OS) [3].
>
> [1] https://github.com/mono/mono/issues/18418
> [2] https://github.com/mono/mono/pull/18430
> [3] https://dotnet.microsoft.com/download
>
> (sorry for hijacking the release vote thread)
>
> On Fri, May 22, 2020 at 7:21 PM Ilya Kasnacheev  >
> wrote:
>
> > Hello!
> >
> > +1 (binding)
> >
> > Checked that .deb package installs and runs.
> > Checked node start and C++ build in -bin.
> > Checked source compilation and mono compilation in -src (note that
> running
> > with Mono is still broken, but problem is not new and seems to be on Mono
> > side).
> >
> > Regards,
> > --
> > Ilya Kasnacheev
> >
> >
> > пт, 22 мая 2020 г. в 15:15, Pavel Tupitsyn :
> >
> > > +1 (binding)
> > > Checked .NET packages and examples.
> > >
> > > On Fri, May 22, 2020 at 2:38 PM Alexey Zinoviev <
> zaleslaw@gmail.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > пт, 22 мая 2020 г. в 13:57, Nikolay Izhikov :
> > > >
> > > > > Dear Community,
> > > > >
> > > > > I have uploaded release candidate to
> > > > > https://dist.apache.org/repos/dist/dev/ignite/2.8.1-rc1/
> > > > > https://dist.apache.org/repos/dist/dev/ignite/packages_2.8.1-rc1/
> > > > >
> > > > > The following staging can be used for testing:
> > > > >
> > >
> https://repository.apache.org/content/repositories/orgapacheignite-1481
> > > > >
> > > > > This is the first maintenance release for 2.8.x with a number of
> > fixes.
> > > > >
> > > > > Tag name is 2.8.1-rc1:
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=tag;h=refs/tags/2.8.1-rc1
> > > > >
> > > > > RELEASE NOTES:
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=RELEASE_NOTES.txt;h=531b2ff7d72bdec9e0d9130efb27bf64f2b4f3fd;hb=9e214ee8ce7b87780d904111d359c74e90613e3f#l5
> > > > >
> > > > > Complete list of resolved issues:
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20IGNITE%20AND%20fixVersion%20%3D%202.8.1
> > > > >
> > > > > DEVNOTES
> > > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=ignite.git;a=blob;f=DEVNOTES.txt;h=307189059ae70834ead4c127cc18295ce4d0735a;hb=9e214ee8ce7b87780d904111d359c74e90613e3
> > > > >
> > > > > The vote is formal, see voting guidelines
> > > > > https://www.apache.org/foundation/voting.html
> > > > >
> > > > > +1 - to accept Apache Ignite 2.8.1-rc1
> > > > > 0 - don't care either way
> > > > > -1 - DO NOT accept Apache Ignite Ignite 2.8.1-rc1 (explain why)
> > > > >
> > > > > See notes on how to verify release here
> > > > > https://www.apache.org/info/verification.html
> > > > > and
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Release+Process#ReleaseProcess-P5.VotingonReleaseandReleaseVerification
> > > > >
> > > > > Additional links:
> > > > >
> > > > > NuGet staging -
> > > > >
> > > >
> > >
> >
> https://www.myget.org/feed/apache-ignite-staging/package/nuget/Apache.Ignite
> > > > > Comparsion of 2.8.0 and 2.8.1 -
> > > > >
> > > >
> > >
> >
> https://ci.ignite.apache.org/repository/download/ApacheIgniteReleaseJava8_IgniteRelease72CheckFileConsistency/5329458:id/results/result.log
> > > > >
> > > > >
> > > > > This vote will be open for at least 3 days till Sun May 22,
> 23:59:59
> > > UTC.
> > > > >
> > > >
> > >
> >
> https://www.timeanddate.com/countdown/generic?iso=20200525T235959=%3A=Ignite+2.8.1-rc1+Vote=cursive
> > > >
> > >
> >
>