[MTCGA]: new failures in builds [6062758] needs to be handled

[dpavlov . tasks]

Hi Igniters,

 I've detected some new issue on TeamCity to be handled. You are more than 
welcomed to help.

 *Test with high flaky rate in master 
CacheExchangeMergeTest.testMergeStartRandomClientsServers 
https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=232193340230784387=%3Cdefault%3E=testDetails
 No changes in the build

 - Here's a reminder of what contributors were agreed to do 
https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute 
 - Should you have any questions please contact dev@ignite.apache.org 

Best Regards,
Apache Ignite TeamCity Bot 
https://github.com/apache/ignite-teamcity-bot
Notification generated at 21:37:25 01-07-2021 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Val, my understanding about it was exactly the same as yours, but, again, I
heard a different opinion.

But nevertheless, binary protocol should not be about objects, records aka
tuples are the best varii, simple and powerful.

As for me, I want to take part in implementing python and golang thin
clients for ignite 3, so consider my remarks using this info. I am not just
a rude critic,
I am just interested in consice and universal binary prorocol
чт, 1 июл. 2021 г., 20:23 Valentin Kulichenko :

> Ivan,
>
> KV view does work over the tuples. Nested objects and arbitrary structures
> can be stored as blobs. So if you need a basic KV cache, you can always
> create a table with two blob fields - one for key and one for value - and
> store anything there.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky 
> wrote:
>
> > Val, am I right, that kv view over the tuples is just simple mapping from
> > POJO to tuple? No collections, no nested objects? I have discussed this
> in
> > private with Igor and Pavel and they told me different info.
> >
> > чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Ivan,
> > >
> > > I was answering your question about the KV API. The API I provided has
> > been
> > > discussed and agreed upon. One of the goals of the protocol is to
> > implement
> > > this API, so it should give you a clear idea of what we're looking for.
> > >
> > > Of course, I agree with you that the protocol should be simple and
> > flexible
> > > enough to allow other implementations for different languages and
> > > platforms.
> > >
> > > -Val
> > >
> > > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> > > wrote:
> > >
> > > > Andrey, yep, you are right. This was just a quick idea. As for me, I
> > just
> > > > don't want to repeat the same problem with compactFooter in thin
> client
> > > api
> > > > of ignite 2.x.
> > > >
> > > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov <
> andrey.mashen...@gmail.com
> > >:
> > > >
> > > > > >
> > > > > > I suppose that we should describe this more verbose and
> explicit. I
> > > > > > nevertheless suggest to also consider writing values this way:
> > > > > > - arr of fields names (if name is missed, corresponding field is
> > nil)
> > > > > > - arr of rows (row as array, length equal to fields array)
> > > > >
> > > > >
> > > > > Ivan,
> > > > > I think GET and PUT operation parameters should be consistent.
> > > > > With PUT operation this way may be tricky.
> > > > >
> > > > > SQL INSERT operation (which is similar PUT operation) semantic
> allows
> > > > > skipping columns that have a default value.
> > > > > Assume we have smth like this:
> > > > >
> > > > > CREATE TABLE t1 (
> > > > >'id' INT;
> > > > >'colname' VARCHAR DEFAULT "abc";
> > > > > )
> > > > > INSERT INTO t1 VALUES(1)
> > > > >
> > > > > Actually, this will add a row (1, "abc")
> > > > >
> > > > > Your suggestion related to missed fields will not work this way as
> it
> > > is
> > > > > impossible to distinct
> > > > > case with 'null' value from the case with a default value.
> > > > >
> > > > >
> > > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > > - UUID: table ID
> > > > > > - int: schema ID
> > > > > > - arr of arr: array of rows with values for all columns in given
> > > schema
> > > > > >
> > > > > > I suppose that we should describe this more verbose and
> explicit. I
> > > > > > nevertheless suggest to also consider writing values this way:
> > > > > > - arr of fields names (if name is missed, corresponding field is
> > nil)
> > > > > > - arr of rows (row as array, length equal to fields array)
> > > > > >
> > > > > > It is quite simple and if we use str8 (it is more than enough for
> > any
> > > > > utf-8
> > > > > > reasonable field name), overhead will be negligible, but
> > realization
> > > > of a
> > > > > > client will be way simpler
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > > > > >
> > > > > > > > No it isn't, I have carefully read code and IEP, in your code
> > you
> > > > > write
> > > > > > > > schema id in each tuple.
> > > > > > >
> > > > > > > There is no code for batch operations yet.
> > > > > > >
> > > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > > > - UUID: table ID
> > > > > > > - int: schema ID
> > > > > > > - arr of arr: array of rows with values for all columns in
> given
> > > > schema
> > > > > > > (nil when value is missing for a column)
> > > > > > >
> > > > > > > As you can see, schema ID is written once for all rows.
> > > > > > > A row is just a set of values according to the schema.
> > > > > > >
> > > > > > >
> > > > > > > > Also, my biggest concern -- extra serde step. I suppose we
> > should
> > > > > pass
> > > > > > > > bytearray to internal api, and use msgpack throughout all
> wire
> > > > > > protocols,
> > > > > > > > as 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

KV view does work over the tuples. Nested objects and arbitrary structures
can be stored as blobs. So if you need a basic KV cache, you can always
create a table with two blob fields - one for key and one for value - and
store anything there.

-Val

On Thu, Jul 1, 2021 at 9:55 AM Ivan Daschinsky  wrote:

> Val, am I right, that kv view over the tuples is just simple mapping from
> POJO to tuple? No collections, no nested objects? I have discussed this in
> private with Igor and Pavel and they told me different info.
>
> чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > I was answering your question about the KV API. The API I provided has
> been
> > discussed and agreed upon. One of the goals of the protocol is to
> implement
> > this API, so it should give you a clear idea of what we're looking for.
> >
> > Of course, I agree with you that the protocol should be simple and
> flexible
> > enough to allow other implementations for different languages and
> > platforms.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> > wrote:
> >
> > > Andrey, yep, you are right. This was just a quick idea. As for me, I
> just
> > > don't want to repeat the same problem with compactFooter in thin client
> > api
> > > of ignite 2.x.
> > >
> > > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov  >:
> > >
> > > > >
> > > > > I suppose that we should describe this more verbose and explicit. I
> > > > > nevertheless suggest to also consider writing values this way:
> > > > > - arr of fields names (if name is missed, corresponding field is
> nil)
> > > > > - arr of rows (row as array, length equal to fields array)
> > > >
> > > >
> > > > Ivan,
> > > > I think GET and PUT operation parameters should be consistent.
> > > > With PUT operation this way may be tricky.
> > > >
> > > > SQL INSERT operation (which is similar PUT operation) semantic allows
> > > > skipping columns that have a default value.
> > > > Assume we have smth like this:
> > > >
> > > > CREATE TABLE t1 (
> > > >'id' INT;
> > > >'colname' VARCHAR DEFAULT "abc";
> > > > )
> > > > INSERT INTO t1 VALUES(1)
> > > >
> > > > Actually, this will add a row (1, "abc")
> > > >
> > > > Your suggestion related to missed fields will not work this way as it
> > is
> > > > impossible to distinct
> > > > case with 'null' value from the case with a default value.
> > > >
> > > >
> > > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > - UUID: table ID
> > > > > - int: schema ID
> > > > > - arr of arr: array of rows with values for all columns in given
> > schema
> > > > >
> > > > > I suppose that we should describe this more verbose and explicit. I
> > > > > nevertheless suggest to also consider writing values this way:
> > > > > - arr of fields names (if name is missed, corresponding field is
> nil)
> > > > > - arr of rows (row as array, length equal to fields array)
> > > > >
> > > > > It is quite simple and if we use str8 (it is more than enough for
> any
> > > > utf-8
> > > > > reasonable field name), overhead will be negligible, but
> realization
> > > of a
> > > > > client will be way simpler
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > > > >
> > > > > > > No it isn't, I have carefully read code and IEP, in your code
> you
> > > > write
> > > > > > > schema id in each tuple.
> > > > > >
> > > > > > There is no code for batch operations yet.
> > > > > >
> > > > > > Here is the description of TUPLE_GET_ALL:
> > > > > > - UUID: table ID
> > > > > > - int: schema ID
> > > > > > - arr of arr: array of rows with values for all columns in given
> > > schema
> > > > > > (nil when value is missing for a column)
> > > > > >
> > > > > > As you can see, schema ID is written once for all rows.
> > > > > > A row is just a set of values according to the schema.
> > > > > >
> > > > > >
> > > > > > > Also, my biggest concern -- extra serde step. I suppose we
> should
> > > > pass
> > > > > > > bytearray to internal api, and use msgpack throughout all wire
> > > > > protocols,
> > > > > > > as tarantool does.
> > > > > >
> > > > > > I agree. But this was decided before in IEP-54, and is out of
> scope
> > > for
> > > > > > current IEP.
> > > > > > Would you like to start a separate thread to discuss this? Or I
> can
> > > do
> > > > > this
> > > > > > a bit later.
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > > This is described in all operations that include multiple
> > tuples.
> > > > > > > No it isn't, I have carefully read code and IEP, in your code
> you
> > > > write
> > > > > > > schema id in each tuple.
> > > > > > >
> > > > > > > Also, my biggest concern -- extra serde step. I suppose we
> should
> > > > pass
> > > > > > > bytearray to internal api, and use msgpack throughout all wire
> > > > > 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

An extra argument for IgniteSystemProperty sounds reasonable.

-Val

On Thu, Jul 1, 2021 at 10:04 AM Ivan Daschinsky  wrote:

> Ok, this can be excluded using blocklist-jvm-params.properties or just by
> providing and extra arg to annotation, as I have just suggested
>
> чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Ivan,
> >
> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to
> be
> > sensitive.
> >
> > Once again - I would exclude any property that can contain user-specific
> > information. Only our internal settings (stuff
> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> > wrote:
> >
> > > We can add add an extra param in annotation, that blocks param to be
> > > printed, just set it to false by default and block it wheb set to true
> > >
> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> > >
> > > > What if we allowed a blocklist of parameters that are never printed?
> > > >
> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Not all of them are OK to be printed out. At the very least, we
> > should
> > > > have
> > > > > a mechanism to exclude some of them. I would still go with opt-in
> > > rather
> > > > > than opt-out though, but I guess that is up to a discussion.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is security through obscurity, an obvious and a well-known
> > anti
> > > > > > pattern. I suppose that printing jvm options, that is registered
> by
> > > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > *Anything* that a user provides to the system can potentially
> be
> > > > > > considered
> > > > > > > sensitive information. This includes the VM arguments. We can't
> > > > predict
> > > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > > >
> > > > > > > When dealing with security, we should be as conservative as
> > > possible.
> > > > > > That
> > > > > > > said, I do not even agree with the pattern approach - there
> might
> > > be
> > > > a
> > > > > > > user's system property named IGNITE_xxx. It is also possible
> for
> > > our
> > > > > > > internal properties to contain sensitive information (not all
> of
> > > them
> > > > > are
> > > > > > > boolean flags).
> > > > > > >
> > > > > > > The only option I see is to print out specific properties for
> > which
> > > > we
> > > > > > > agree that they are safe. For example, we can introduce an
> > > annotation
> > > > > > that
> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > > then
> > > > > > print
> > > > > > > out only those that are marked with the annotation.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ivan,
> > > > > > > >
> > > > > > > > > At least, we could just hide params that match a specific
> > > pattern
> > > > > > > > Yes, we can filter out all vm options that do not relate to
> > > Ignite,
> > > > > for
> > > > > > > > instance.
> > > > > > > >
> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > > Please do not rush. Let's listen to other community members.
> > This
> > > > > > > question
> > > > > > > > is about security and it should not be discussed in a hurry
> > (even
> > > > > > though
> > > > > > > it
> > > > > > > > looks like an obvious thing).
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > S.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > I suppose, that all normal users should not suffer from
> this
> > > > > > > > restrictions.
> > > > > > > > > Nobody will pass password using jvm options. It is
> absolutely
> > > > > insane,
> > > > > > > > > normal users pass passwords using environment variables.
> > > > > > > > >
> > > > > > > > > At least, we could just hide params that match specific
> > pattern
> > > > > > > > >
> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > > >
> > > > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >:
> > > > > > > > >
> > > > > > > > > > Hello,
> > > > > > > > > >
> > > > > > > > > > Unfortunately, the user can pass its own system

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

Ok, this can be excluded using blocklist-jvm-params.properties or just by
providing and extra arg to annotation, as I have just suggested

чт, 1 июл. 2021 г., 19:51 Valentin Kulichenko :

> Ivan,
>
> IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to be
> sensitive.
>
> Once again - I would exclude any property that can contain user-specific
> information. Only our internal settings (stuff
> like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> wrote:
>
> > We can add add an extra param in annotation, that blocks param to be
> > printed, just set it to false by default and block it wheb set to true
> >
> > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> >
> > > What if we allowed a blocklist of parameters that are never printed?
> > >
> > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Not all of them are OK to be printed out. At the very least, we
> should
> > > have
> > > > a mechanism to exclude some of them. I would still go with opt-in
> > rather
> > > > than opt-out though, but I guess that is up to a discussion.
> > > >
> > > > -Val
> > > >
> > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > This is security through obscurity, an obvious and a well-known
> anti
> > > > > pattern. I suppose that printing jvm options, that is registered by
> > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > >
> > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com
> > > > > >:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > *Anything* that a user provides to the system can potentially be
> > > > > considered
> > > > > > sensitive information. This includes the VM arguments. We can't
> > > predict
> > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > >
> > > > > > When dealing with security, we should be as conservative as
> > possible.
> > > > > That
> > > > > > said, I do not even agree with the pattern approach - there might
> > be
> > > a
> > > > > > user's system property named IGNITE_xxx. It is also possible for
> > our
> > > > > > internal properties to contain sensitive information (not all of
> > them
> > > > are
> > > > > > boolean flags).
> > > > > >
> > > > > > The only option I see is to print out specific properties for
> which
> > > we
> > > > > > agree that they are safe. For example, we can introduce an
> > annotation
> > > > > that
> > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > then
> > > > > print
> > > > > > out only those that are marked with the annotation.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Ivan,
> > > > > > >
> > > > > > > > At least, we could just hide params that match a specific
> > pattern
> > > > > > > Yes, we can filter out all vm options that do not relate to
> > Ignite,
> > > > for
> > > > > > > instance.
> > > > > > >
> > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > Please do not rush. Let's listen to other community members.
> This
> > > > > > question
> > > > > > > is about security and it should not be discussed in a hurry
> (even
> > > > > though
> > > > > > it
> > > > > > > looks like an obvious thing).
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S.
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky <
> ivanda...@gmail.com
> > >:
> > > > > > >
> > > > > > > > I suppose, that all normal users should not suffer from this
> > > > > > > restrictions.
> > > > > > > > Nobody will pass password using jvm options. It is absolutely
> > > > insane,
> > > > > > > > normal users pass passwords using environment variables.
> > > > > > > >
> > > > > > > > At least, we could just hide params that match specific
> pattern
> > > > > > > >
> > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > Unfortunately, the user can pass its own system properties
> > via
> > > > JVM
> > > > > > > > options
> > > > > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > > > > It does not seem, this approach is the best one. However,
> the
> > > > user
> > > > > > > should
> > > > > > > > > have a "kostyl" in order to hide these properties and
> values
> > in
> > > > the
> > > > > > log
> > > > > > > > > file, IMHO.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > S.
> > > > > > > > >
> > > > > > > > > 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Ilya,

I don't think this is the best approach because there are so many
properties that contain so many different types of information. "All or
nothing" doesn't really fit here. We want to have the ability to exclude
sensitive information, but still print out internal settings that are
useful for debugging, etc.

-Val

On Thu, Jul 1, 2021 at 9:54 AM Shishkov Ilya  wrote:

> Folks,
>
> > Maybe we should add an extra JVM option (e.g.
> IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> > but if set to 'true' then #ackVmOptions will print VM arguments even if
> sensitive data is restricted?
>
> What do you think about an extra JVM option?
>
> чт, 1 июл. 2021 г. в 19:51, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Ivan,
> >
> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to
> be
> > sensitive.
> >
> > Once again - I would exclude any property that can contain user-specific
> > information. Only our internal settings (stuff
> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> > wrote:
> >
> > > We can add add an extra param in annotation, that blocks param to be
> > > printed, just set it to false by default and block it wheb set to true
> > >
> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> > >
> > > > What if we allowed a blocklist of parameters that are never printed?
> > > >
> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Not all of them are OK to be printed out. At the very least, we
> > should
> > > > have
> > > > > a mechanism to exclude some of them. I would still go with opt-in
> > > rather
> > > > > than opt-out though, but I guess that is up to a discussion.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is security through obscurity, an obvious and a well-known
> > anti
> > > > > > pattern. I suppose that printing jvm options, that is registered
> by
> > > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > *Anything* that a user provides to the system can potentially
> be
> > > > > > considered
> > > > > > > sensitive information. This includes the VM arguments. We can't
> > > > predict
> > > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > > >
> > > > > > > When dealing with security, we should be as conservative as
> > > possible.
> > > > > > That
> > > > > > > said, I do not even agree with the pattern approach - there
> might
> > > be
> > > > a
> > > > > > > user's system property named IGNITE_xxx. It is also possible
> for
> > > our
> > > > > > > internal properties to contain sensitive information (not all
> of
> > > them
> > > > > are
> > > > > > > boolean flags).
> > > > > > >
> > > > > > > The only option I see is to print out specific properties for
> > which
> > > > we
> > > > > > > agree that they are safe. For example, we can introduce an
> > > annotation
> > > > > > that
> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > > then
> > > > > > print
> > > > > > > out only those that are marked with the annotation.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ivan,
> > > > > > > >
> > > > > > > > > At least, we could just hide params that match a specific
> > > pattern
> > > > > > > > Yes, we can filter out all vm options that do not relate to
> > > Ignite,
> > > > > for
> > > > > > > > instance.
> > > > > > > >
> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > > Please do not rush. Let's listen to other community members.
> > This
> > > > > > > question
> > > > > > > > is about security and it should not be discussed in a hurry
> > (even
> > > > > > though
> > > > > > > it
> > > > > > > > looks like an obvious thing).
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > S.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > I suppose, that all normal users should not suffer from
> this
> > > > > > > > restrictions.
> > > > > > > > > Nobody will pass password using jvm options. It is
> absolutely
> > > > > insane,
> > > > > > > > > normal users pass passwords using environment variables.
> > > > > > > > >
> > > > > > > > > At least, we could just hide 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Atri Sharma]

Why not mask the default known sensitive options using a blocklist?

On Thu, 1 Jul 2021, 22:24 Shishkov Ilya,  wrote:

> Folks,
>
> > Maybe we should add an extra JVM option (e.g.
> IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> > but if set to 'true' then #ackVmOptions will print VM arguments even if
> sensitive data is restricted?
>
> What do you think about an extra JVM option?
>
> чт, 1 июл. 2021 г. в 19:51, Valentin Kulichenko <
> valentin.kuliche...@gmail.com>:
>
> > Ivan,
> >
> > IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> > (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> > related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to
> be
> > sensitive.
> >
> > Once again - I would exclude any property that can contain user-specific
> > information. Only our internal settings (stuff
> > like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> > wrote:
> >
> > > We can add add an extra param in annotation, that blocks param to be
> > > printed, just set it to false by default and block it wheb set to true
> > >
> > > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> > >
> > > > What if we allowed a blocklist of parameters that are never printed?
> > > >
> > > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Not all of them are OK to be printed out. At the very least, we
> > should
> > > > have
> > > > > a mechanism to exclude some of them. I would still go with opt-in
> > > rather
> > > > > than opt-out though, but I guess that is up to a discussion.
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This is security through obscurity, an obvious and a well-known
> > anti
> > > > > > pattern. I suppose that printing jvm options, that is registered
> by
> > > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > > valentin.kuliche...@gmail.com
> > > > > > >:
> > > > > >
> > > > > > > Folks,
> > > > > > >
> > > > > > > *Anything* that a user provides to the system can potentially
> be
> > > > > > considered
> > > > > > > sensitive information. This includes the VM arguments. We can't
> > > > predict
> > > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > > >
> > > > > > > When dealing with security, we should be as conservative as
> > > possible.
> > > > > > That
> > > > > > > said, I do not even agree with the pattern approach - there
> might
> > > be
> > > > a
> > > > > > > user's system property named IGNITE_xxx. It is also possible
> for
> > > our
> > > > > > > internal properties to contain sensitive information (not all
> of
> > > them
> > > > > are
> > > > > > > boolean flags).
> > > > > > >
> > > > > > > The only option I see is to print out specific properties for
> > which
> > > > we
> > > > > > > agree that they are safe. For example, we can introduce an
> > > annotation
> > > > > > that
> > > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > > then
> > > > > > print
> > > > > > > out only those that are marked with the annotation.
> > > > > > >
> > > > > > > -Val
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hello Ivan,
> > > > > > > >
> > > > > > > > > At least, we could just hide params that match a specific
> > > pattern
> > > > > > > > Yes, we can filter out all vm options that do not relate to
> > > Ignite,
> > > > > for
> > > > > > > > instance.
> > > > > > > >
> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > > Please do not rush. Let's listen to other community members.
> > This
> > > > > > > question
> > > > > > > > is about security and it should not be discussed in a hurry
> > (even
> > > > > > though
> > > > > > > it
> > > > > > > > looks like an obvious thing).
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > S.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >:
> > > > > > > >
> > > > > > > > > I suppose, that all normal users should not suffer from
> this
> > > > > > > > restrictions.
> > > > > > > > > Nobody will pass password using jvm options. It is
> absolutely
> > > > > insane,
> > > > > > > > > normal users pass passwords using environment variables.
> > > > > > > > >
> > > > > > > > > At least, we could just hide params that match specific
> > pattern
> > > > > > > > >
> > > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > > >
> > > > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > > > > slava.kopti...@gmail.com
> > > > > > > >:
> 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Val, am I right, that kv view over the tuples is just simple mapping from
POJO to tuple? No collections, no nested objects? I have discussed this in
private with Igor and Pavel and they told me different info.

чт, 1 июл. 2021 г., 19:43 Valentin Kulichenko :

> Ivan,
>
> I was answering your question about the KV API. The API I provided has been
> discussed and agreed upon. One of the goals of the protocol is to implement
> this API, so it should give you a clear idea of what we're looking for.
>
> Of course, I agree with you that the protocol should be simple and flexible
> enough to allow other implementations for different languages and
> platforms.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky 
> wrote:
>
> > Andrey, yep, you are right. This was just a quick idea. As for me, I just
> > don't want to repeat the same problem with compactFooter in thin client
> api
> > of ignite 2.x.
> >
> > чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov :
> >
> > > >
> > > > I suppose that we should describe this more verbose and explicit. I
> > > > nevertheless suggest to also consider writing values this way:
> > > > - arr of fields names (if name is missed, corresponding field is nil)
> > > > - arr of rows (row as array, length equal to fields array)
> > >
> > >
> > > Ivan,
> > > I think GET and PUT operation parameters should be consistent.
> > > With PUT operation this way may be tricky.
> > >
> > > SQL INSERT operation (which is similar PUT operation) semantic allows
> > > skipping columns that have a default value.
> > > Assume we have smth like this:
> > >
> > > CREATE TABLE t1 (
> > >'id' INT;
> > >'colname' VARCHAR DEFAULT "abc";
> > > )
> > > INSERT INTO t1 VALUES(1)
> > >
> > > Actually, this will add a row (1, "abc")
> > >
> > > Your suggestion related to missed fields will not work this way as it
> is
> > > impossible to distinct
> > > case with 'null' value from the case with a default value.
> > >
> > >
> > > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > > Here is the description of TUPLE_GET_ALL:
> > > > - UUID: table ID
> > > > - int: schema ID
> > > > - arr of arr: array of rows with values for all columns in given
> schema
> > > >
> > > > I suppose that we should describe this more verbose and explicit. I
> > > > nevertheless suggest to also consider writing values this way:
> > > > - arr of fields names (if name is missed, corresponding field is nil)
> > > > - arr of rows (row as array, length equal to fields array)
> > > >
> > > > It is quite simple and if we use str8 (it is more than enough for any
> > > utf-8
> > > > reasonable field name), overhead will be negligible, but realization
> > of a
> > > > client will be way simpler
> > > >
> > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > > >
> > > > > > No it isn't, I have carefully read code and IEP, in your code you
> > > write
> > > > > > schema id in each tuple.
> > > > >
> > > > > There is no code for batch operations yet.
> > > > >
> > > > > Here is the description of TUPLE_GET_ALL:
> > > > > - UUID: table ID
> > > > > - int: schema ID
> > > > > - arr of arr: array of rows with values for all columns in given
> > schema
> > > > > (nil when value is missing for a column)
> > > > >
> > > > > As you can see, schema ID is written once for all rows.
> > > > > A row is just a set of values according to the schema.
> > > > >
> > > > >
> > > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > > pass
> > > > > > bytearray to internal api, and use msgpack throughout all wire
> > > > protocols,
> > > > > > as tarantool does.
> > > > >
> > > > > I agree. But this was decided before in IEP-54, and is out of scope
> > for
> > > > > current IEP.
> > > > > Would you like to start a separate thread to discuss this? Or I can
> > do
> > > > this
> > > > > a bit later.
> > > > >
> > > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > > This is described in all operations that include multiple
> tuples.
> > > > > > No it isn't, I have carefully read code and IEP, in your code you
> > > write
> > > > > > schema id in each tuple.
> > > > > >
> > > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > > pass
> > > > > > bytearray to internal api, and use msgpack throughout all wire
> > > > protocols,
> > > > > > as tarantool does.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > > > > >
> > > > > > > Ivan,
> > > > > > >
> > > > > > > >  that there is not neccesary to write schema versions in each
> > row
> > > > > > > > in collectionof tuples
> > > > > > >
> > > > > > > This is described in all operations that include multiple
> tuples.
> > > > > > >
> > > > > > >
> > > > > > > > it is not clear from your code (probably
> > > > > > > > mistake?) how differ key tuples and value tuples from each
> > other
> > > > > > >
> > > > > > > Key tuples include only key columns. Key columns come 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Shishkov Ilya]

Folks,

> Maybe we should add an extra JVM option (e.g.
IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default,
> but if set to 'true' then #ackVmOptions will print VM arguments even if
sensitive data is restricted?

What do you think about an extra JVM option?

чт, 1 июл. 2021 г. в 19:51, Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Ivan,
>
> IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
> (e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
> related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to be
> sensitive.
>
> Once again - I would exclude any property that can contain user-specific
> information. Only our internal settings (stuff
> like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky 
> wrote:
>
> > We can add add an extra param in annotation, that blocks param to be
> > printed, just set it to false by default and block it wheb set to true
> >
> > чт, 1 июл. 2021 г., 19:45 Atri Sharma :
> >
> > > What if we allowed a blocklist of parameters that are never printed?
> > >
> > > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > > valentin.kuliche...@gmail.com> wrote:
> > >
> > > > Not all of them are OK to be printed out. At the very least, we
> should
> > > have
> > > > a mechanism to exclude some of them. I would still go with opt-in
> > rather
> > > > than opt-out though, but I guess that is up to a discussion.
> > > >
> > > > -Val
> > > >
> > > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > This is security through obscurity, an obvious and a well-known
> anti
> > > > > pattern. I suppose that printing jvm options, that is registered by
> > > > > @IgniteSystemProperty annotation is an ideal variant
> > > > >
> > > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > > valentin.kuliche...@gmail.com
> > > > > >:
> > > > >
> > > > > > Folks,
> > > > > >
> > > > > > *Anything* that a user provides to the system can potentially be
> > > > > considered
> > > > > > sensitive information. This includes the VM arguments. We can't
> > > predict
> > > > > > what exactly one can put there, so let's not make assumptions.
> > > > > >
> > > > > > When dealing with security, we should be as conservative as
> > possible.
> > > > > That
> > > > > > said, I do not even agree with the pattern approach - there might
> > be
> > > a
> > > > > > user's system property named IGNITE_xxx. It is also possible for
> > our
> > > > > > internal properties to contain sensitive information (not all of
> > them
> > > > are
> > > > > > boolean flags).
> > > > > >
> > > > > > The only option I see is to print out specific properties for
> which
> > > we
> > > > > > agree that they are safe. For example, we can introduce an
> > annotation
> > > > > that
> > > > > > would mark "safe" properties in IgniteSystemProperties; we will
> > then
> > > > > print
> > > > > > out only those that are marked with the annotation.
> > > > > >
> > > > > > -Val
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hello Ivan,
> > > > > > >
> > > > > > > > At least, we could just hide params that match a specific
> > pattern
> > > > > > > Yes, we can filter out all vm options that do not relate to
> > Ignite,
> > > > for
> > > > > > > instance.
> > > > > > >
> > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > Please do not rush. Let's listen to other community members.
> This
> > > > > > question
> > > > > > > is about security and it should not be discussed in a hurry
> (even
> > > > > though
> > > > > > it
> > > > > > > looks like an obvious thing).
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S.
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky <
> ivanda...@gmail.com
> > >:
> > > > > > >
> > > > > > > > I suppose, that all normal users should not suffer from this
> > > > > > > restrictions.
> > > > > > > > Nobody will pass password using jvm options. It is absolutely
> > > > insane,
> > > > > > > > normal users pass passwords using environment variables.
> > > > > > > >
> > > > > > > > At least, we could just hide params that match specific
> pattern
> > > > > > > >
> > > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > > > slava.kopti...@gmail.com
> > > > > > >:
> > > > > > > >
> > > > > > > > > Hello,
> > > > > > > > >
> > > > > > > > > Unfortunately, the user can pass its own system properties
> > via
> > > > JVM
> > > > > > > > options
> > > > > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > > > > It does not seem, this approach is the best one. However,
> the
> > > > user
> > > > > > > should
> > > > > > > > > have a "kostyl" in order to hide these properties and
> values
> 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Ivan,

IP addresses (e.g. IGNITE_TCP_DISCOVERY_ADDRESSES) and file paths
(e.g. IGNITE_CONFIG_URL) are often considered sensitive information. Data
related to authentication (e.g. IGNITE_SSH_USER_NAME) is very likely to be
sensitive.

Once again - I would exclude any property that can contain user-specific
information. Only our internal settings (stuff
like IGNITE_SQL_MERGE_TABLE_MAX_SIZE) are OK to appear in the logs.

-Val

On Thu, Jul 1, 2021 at 9:47 AM Ivan Daschinsky  wrote:

> We can add add an extra param in annotation, that blocks param to be
> printed, just set it to false by default and block it wheb set to true
>
> чт, 1 июл. 2021 г., 19:45 Atri Sharma :
>
> > What if we allowed a blocklist of parameters that are never printed?
> >
> > On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> > valentin.kuliche...@gmail.com> wrote:
> >
> > > Not all of them are OK to be printed out. At the very least, we should
> > have
> > > a mechanism to exclude some of them. I would still go with opt-in
> rather
> > > than opt-out though, but I guess that is up to a discussion.
> > >
> > > -Val
> > >
> > > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > > wrote:
> > >
> > > > This is security through obscurity, an obvious and a well-known anti
> > > > pattern. I suppose that printing jvm options, that is registered by
> > > > @IgniteSystemProperty annotation is an ideal variant
> > > >
> > > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com
> > > > >:
> > > >
> > > > > Folks,
> > > > >
> > > > > *Anything* that a user provides to the system can potentially be
> > > > considered
> > > > > sensitive information. This includes the VM arguments. We can't
> > predict
> > > > > what exactly one can put there, so let's not make assumptions.
> > > > >
> > > > > When dealing with security, we should be as conservative as
> possible.
> > > > That
> > > > > said, I do not even agree with the pattern approach - there might
> be
> > a
> > > > > user's system property named IGNITE_xxx. It is also possible for
> our
> > > > > internal properties to contain sensitive information (not all of
> them
> > > are
> > > > > boolean flags).
> > > > >
> > > > > The only option I see is to print out specific properties for which
> > we
> > > > > agree that they are safe. For example, we can introduce an
> annotation
> > > > that
> > > > > would mark "safe" properties in IgniteSystemProperties; we will
> then
> > > > print
> > > > > out only those that are marked with the annotation.
> > > > >
> > > > > -Val
> > > > >
> > > > >
> > > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > > slava.kopti...@gmail.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Hello Ivan,
> > > > > >
> > > > > > > At least, we could just hide params that match a specific
> pattern
> > > > > > Yes, we can filter out all vm options that do not relate to
> Ignite,
> > > for
> > > > > > instance.
> > > > > >
> > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > Please do not rush. Let's listen to other community members. This
> > > > > question
> > > > > > is about security and it should not be discussed in a hurry (even
> > > > though
> > > > > it
> > > > > > looks like an obvious thing).
> > > > > >
> > > > > > Thanks,
> > > > > > S.
> > > > > >
> > > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky  >:
> > > > > >
> > > > > > > I suppose, that all normal users should not suffer from this
> > > > > > restrictions.
> > > > > > > Nobody will pass password using jvm options. It is absolutely
> > > insane,
> > > > > > > normal users pass passwords using environment variables.
> > > > > > >
> > > > > > > At least, we could just hide params that match specific pattern
> > > > > > >
> > > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > > slava.kopti...@gmail.com
> > > > > >:
> > > > > > >
> > > > > > > > Hello,
> > > > > > > >
> > > > > > > > Unfortunately, the user can pass its own system properties
> via
> > > JVM
> > > > > > > options
> > > > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > > > It does not seem, this approach is the best one. However, the
> > > user
> > > > > > should
> > > > > > > > have a "kostyl" in order to hide these properties and values
> in
> > > the
> > > > > log
> > > > > > > > file, IMHO.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > S.
> > > > > > > >
> > > > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya <
> > > shishkovi...@gmail.com
> > > > >:
> > > > > > > >
> > > > > > > > > Hi Igniters,
> > > > > > > > >
> > > > > > > > > This feature [1, 2] prevents logging of the VM arguments
> when
> > > > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false.
> > Till
> > > > > now,
> > > > > > > > method
> > > > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > > > > >
> > > > > > > > > Is this behaviour actual now? Often, we 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

We can add add an extra param in annotation, that blocks param to be
printed, just set it to false by default and block it wheb set to true

чт, 1 июл. 2021 г., 19:45 Atri Sharma :

> What if we allowed a blocklist of parameters that are never printed?
>
> On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
> valentin.kuliche...@gmail.com> wrote:
>
> > Not all of them are OK to be printed out. At the very least, we should
> have
> > a mechanism to exclude some of them. I would still go with opt-in rather
> > than opt-out though, but I guess that is up to a discussion.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > wrote:
> >
> > > This is security through obscurity, an obvious and a well-known anti
> > > pattern. I suppose that printing jvm options, that is registered by
> > > @IgniteSystemProperty annotation is an ideal variant
> > >
> > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Folks,
> > > >
> > > > *Anything* that a user provides to the system can potentially be
> > > considered
> > > > sensitive information. This includes the VM arguments. We can't
> predict
> > > > what exactly one can put there, so let's not make assumptions.
> > > >
> > > > When dealing with security, we should be as conservative as possible.
> > > That
> > > > said, I do not even agree with the pattern approach - there might be
> a
> > > > user's system property named IGNITE_xxx. It is also possible for our
> > > > internal properties to contain sensitive information (not all of them
> > are
> > > > boolean flags).
> > > >
> > > > The only option I see is to print out specific properties for which
> we
> > > > agree that they are safe. For example, we can introduce an annotation
> > > that
> > > > would mark "safe" properties in IgniteSystemProperties; we will then
> > > print
> > > > out only those that are marked with the annotation.
> > > >
> > > > -Val
> > > >
> > > >
> > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > slava.kopti...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello Ivan,
> > > > >
> > > > > > At least, we could just hide params that match a specific pattern
> > > > > Yes, we can filter out all vm options that do not relate to Ignite,
> > for
> > > > > instance.
> > > > >
> > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > Please do not rush. Let's listen to other community members. This
> > > > question
> > > > > is about security and it should not be discussed in a hurry (even
> > > though
> > > > it
> > > > > looks like an obvious thing).
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > > > >
> > > > > > I suppose, that all normal users should not suffer from this
> > > > > restrictions.
> > > > > > Nobody will pass password using jvm options. It is absolutely
> > insane,
> > > > > > normal users pass passwords using environment variables.
> > > > > >
> > > > > > At least, we could just hide params that match specific pattern
> > > > > >
> > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > slava.kopti...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Unfortunately, the user can pass its own system properties via
> > JVM
> > > > > > options
> > > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > > It does not seem, this approach is the best one. However, the
> > user
> > > > > should
> > > > > > > have a "kostyl" in order to hide these properties and values in
> > the
> > > > log
> > > > > > > file, IMHO.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S.
> > > > > > >
> > > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya <
> > shishkovi...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false.
> Till
> > > > now,
> > > > > > > method
> > > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > > > >
> > > > > > > > Is this behaviour actual now? Often, we should be able to get
> > > from
> > > > > logs
> > > > > > > the
> > > > > > > > actual VM options used at startup even if output of sensitive
> > > data
> > > > is
> > > > > > > > restricted.
> > > > > > > >
> > > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > > > 2.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > > > 3.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Atri Sharma]

What if we allowed a blocklist of parameters that are never printed?

On Thu, 1 Jul 2021, 22:06 Valentin Kulichenko, <
valentin.kuliche...@gmail.com> wrote:

> Not all of them are OK to be printed out. At the very least, we should have
> a mechanism to exclude some of them. I would still go with opt-in rather
> than opt-out though, but I guess that is up to a discussion.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> wrote:
>
> > This is security through obscurity, an obvious and a well-known anti
> > pattern. I suppose that printing jvm options, that is registered by
> > @IgniteSystemProperty annotation is an ideal variant
> >
> > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Folks,
> > >
> > > *Anything* that a user provides to the system can potentially be
> > considered
> > > sensitive information. This includes the VM arguments. We can't predict
> > > what exactly one can put there, so let's not make assumptions.
> > >
> > > When dealing with security, we should be as conservative as possible.
> > That
> > > said, I do not even agree with the pattern approach - there might be a
> > > user's system property named IGNITE_xxx. It is also possible for our
> > > internal properties to contain sensitive information (not all of them
> are
> > > boolean flags).
> > >
> > > The only option I see is to print out specific properties for which we
> > > agree that they are safe. For example, we can introduce an annotation
> > that
> > > would mark "safe" properties in IgniteSystemProperties; we will then
> > print
> > > out only those that are marked with the annotation.
> > >
> > > -Val
> > >
> > >
> > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > slava.kopti...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello Ivan,
> > > >
> > > > > At least, we could just hide params that match a specific pattern
> > > > Yes, we can filter out all vm options that do not relate to Ignite,
> for
> > > > instance.
> > > >
> > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > Please do not rush. Let's listen to other community members. This
> > > question
> > > > is about security and it should not be discussed in a hurry (even
> > though
> > > it
> > > > looks like an obvious thing).
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > > >
> > > > > I suppose, that all normal users should not suffer from this
> > > > restrictions.
> > > > > Nobody will pass password using jvm options. It is absolutely
> insane,
> > > > > normal users pass passwords using environment variables.
> > > > >
> > > > > At least, we could just hide params that match specific pattern
> > > > >
> > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > slava.kopti...@gmail.com
> > > >:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Unfortunately, the user can pass its own system properties via
> JVM
> > > > > options
> > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > It does not seem, this approach is the best one. However, the
> user
> > > > should
> > > > > > have a "kostyl" in order to hide these properties and values in
> the
> > > log
> > > > > > file, IMHO.
> > > > > >
> > > > > > Thanks,
> > > > > > S.
> > > > > >
> > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya <
> shishkovi...@gmail.com
> > >:
> > > > > >
> > > > > > > Hi Igniters,
> > > > > > >
> > > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till
> > > now,
> > > > > > method
> > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > > >
> > > > > > > Is this behaviour actual now? Often, we should be able to get
> > from
> > > > logs
> > > > > > the
> > > > > > > actual VM options used at startup even if output of sensitive
> > data
> > > is
> > > > > > > restricted.
> > > > > > >
> > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > > 2.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > > 3.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Shishkov Ilya]

Hi,
Maybe we should add an extra JVM option (e.g.
IGNITE_FORCE_PRINT_VM_ARGUMENTS) which is 'false' by default, but if set to
'true' then #ackVmOptions will print VM arguments even if sensitive data is
restricted?

чт, 1 июл. 2021 г. в 19:41, Ivan Daschinsky :

> > Not all of them are OK to be printed out
> Could you give an example please? As for me, all of them are pretty
> harmless
>
> чт, 1 июл. 2021 г., 19:36 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Not all of them are OK to be printed out. At the very least, we should
> have
> > a mechanism to exclude some of them. I would still go with opt-in rather
> > than opt-out though, but I guess that is up to a discussion.
> >
> > -Val
> >
> > On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> > wrote:
> >
> > > This is security through obscurity, an obvious and a well-known anti
> > > pattern. I suppose that printing jvm options, that is registered by
> > > @IgniteSystemProperty annotation is an ideal variant
> > >
> > > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > > valentin.kuliche...@gmail.com
> > > >:
> > >
> > > > Folks,
> > > >
> > > > *Anything* that a user provides to the system can potentially be
> > > considered
> > > > sensitive information. This includes the VM arguments. We can't
> predict
> > > > what exactly one can put there, so let's not make assumptions.
> > > >
> > > > When dealing with security, we should be as conservative as possible.
> > > That
> > > > said, I do not even agree with the pattern approach - there might be
> a
> > > > user's system property named IGNITE_xxx. It is also possible for our
> > > > internal properties to contain sensitive information (not all of them
> > are
> > > > boolean flags).
> > > >
> > > > The only option I see is to print out specific properties for which
> we
> > > > agree that they are safe. For example, we can introduce an annotation
> > > that
> > > > would mark "safe" properties in IgniteSystemProperties; we will then
> > > print
> > > > out only those that are marked with the annotation.
> > > >
> > > > -Val
> > > >
> > > >
> > > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > > slava.kopti...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hello Ivan,
> > > > >
> > > > > > At least, we could just hide params that match a specific pattern
> > > > > Yes, we can filter out all vm options that do not relate to Ignite,
> > for
> > > > > instance.
> > > > >
> > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > Please do not rush. Let's listen to other community members. This
> > > > question
> > > > > is about security and it should not be discussed in a hurry (even
> > > though
> > > > it
> > > > > looks like an obvious thing).
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > > > >
> > > > > > I suppose, that all normal users should not suffer from this
> > > > > restrictions.
> > > > > > Nobody will pass password using jvm options. It is absolutely
> > insane,
> > > > > > normal users pass passwords using environment variables.
> > > > > >
> > > > > > At least, we could just hide params that match specific pattern
> > > > > >
> > > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > > slava.kopti...@gmail.com
> > > > >:
> > > > > >
> > > > > > > Hello,
> > > > > > >
> > > > > > > Unfortunately, the user can pass its own system properties via
> > JVM
> > > > > > options
> > > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > > It does not seem, this approach is the best one. However, the
> > user
> > > > > should
> > > > > > > have a "kostyl" in order to hide these properties and values in
> > the
> > > > log
> > > > > > > file, IMHO.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > S.
> > > > > > >
> > > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya <
> > shishkovi...@gmail.com
> > > >:
> > > > > > >
> > > > > > > > Hi Igniters,
> > > > > > > >
> > > > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false.
> Till
> > > > now,
> > > > > > > method
> > > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > > > >
> > > > > > > > Is this behaviour actual now? Often, we should be able to get
> > > from
> > > > > logs
> > > > > > > the
> > > > > > > > actual VM options used at startup even if output of sensitive
> > > data
> > > > is
> > > > > > > > restricted.
> > > > > > > >
> > > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > > > 2.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > > > 3.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

I was answering your question about the KV API. The API I provided has been
discussed and agreed upon. One of the goals of the protocol is to implement
this API, so it should give you a clear idea of what we're looking for.

Of course, I agree with you that the protocol should be simple and flexible
enough to allow other implementations for different languages and platforms.

-Val

On Thu, Jul 1, 2021 at 9:38 AM Ivan Daschinsky  wrote:

> Andrey, yep, you are right. This was just a quick idea. As for me, I just
> don't want to repeat the same problem with compactFooter in thin client api
> of ignite 2.x.
>
> чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov :
>
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> >
> >
> > Ivan,
> > I think GET and PUT operation parameters should be consistent.
> > With PUT operation this way may be tricky.
> >
> > SQL INSERT operation (which is similar PUT operation) semantic allows
> > skipping columns that have a default value.
> > Assume we have smth like this:
> >
> > CREATE TABLE t1 (
> >'id' INT;
> >'colname' VARCHAR DEFAULT "abc";
> > )
> > INSERT INTO t1 VALUES(1)
> >
> > Actually, this will add a row (1, "abc")
> >
> > Your suggestion related to missed fields will not work this way as it is
> > impossible to distinct
> > case with 'null' value from the case with a default value.
> >
> >
> > On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> > wrote:
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > utf-8
> > > reasonable field name), overhead will be negligible, but realization
> of a
> > > client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > >
> > > > > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > > > > schema id in each tuple.
> > > >
> > > > There is no code for batch operations yet.
> > > >
> > > > Here is the description of TUPLE_GET_ALL:
> > > > - UUID: table ID
> > > > - int: schema ID
> > > > - arr of arr: array of rows with values for all columns in given
> schema
> > > > (nil when value is missing for a column)
> > > >
> > > > As you can see, schema ID is written once for all rows.
> > > > A row is just a set of values according to the schema.
> > > >
> > > >
> > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > > > bytearray to internal api, and use msgpack throughout all wire
> > > protocols,
> > > > > as tarantool does.
> > > >
> > > > I agree. But this was decided before in IEP-54, and is out of scope
> for
> > > > current IEP.
> > > > Would you like to start a separate thread to discuss this? Or I can
> do
> > > this
> > > > a bit later.
> > > >
> > > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > > This is described in all operations that include multiple tuples.
> > > > > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > > > > schema id in each tuple.
> > > > >
> > > > > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > > > bytearray to internal api, and use msgpack throughout all wire
> > > protocols,
> > > > > as tarantool does.
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan,
> > > > > >
> > > > > > >  that there is not neccesary to write schema versions in each
> row
> > > > > > > in collectionof tuples
> > > > > >
> > > > > > This is described in all operations that include multiple tuples.
> > > > > >
> > > > > >
> > > > > > > it is not clear from your code (probably
> > > > > > > mistake?) how differ key tuples and value tuples from each
> other
> > > > > >
> > > > > > Key tuples include only key columns. Key columns come first in
> the
> > > > > schema.
> > > > > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > > > > > section.
> > > > > >
> > > > > >
> > > > > > > As for me, these excercises with schema's doesn't worth a lot
> > > > > >
> > > > > > I'll add a benchmark and we'll see.
> > > > > >
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I suppose, that there is not neccesary to write schema versions
> > in
> > > > each
> > > > > > row
> 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

> Not all of them are OK to be printed out
Could you give an example please? As for me, all of them are pretty harmless

чт, 1 июл. 2021 г., 19:36 Valentin Kulichenko :

> Not all of them are OK to be printed out. At the very least, we should have
> a mechanism to exclude some of them. I would still go with opt-in rather
> than opt-out though, but I guess that is up to a discussion.
>
> -Val
>
> On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky 
> wrote:
>
> > This is security through obscurity, an obvious and a well-known anti
> > pattern. I suppose that printing jvm options, that is registered by
> > @IgniteSystemProperty annotation is an ideal variant
> >
> > чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> > valentin.kuliche...@gmail.com
> > >:
> >
> > > Folks,
> > >
> > > *Anything* that a user provides to the system can potentially be
> > considered
> > > sensitive information. This includes the VM arguments. We can't predict
> > > what exactly one can put there, so let's not make assumptions.
> > >
> > > When dealing with security, we should be as conservative as possible.
> > That
> > > said, I do not even agree with the pattern approach - there might be a
> > > user's system property named IGNITE_xxx. It is also possible for our
> > > internal properties to contain sensitive information (not all of them
> are
> > > boolean flags).
> > >
> > > The only option I see is to print out specific properties for which we
> > > agree that they are safe. For example, we can introduce an annotation
> > that
> > > would mark "safe" properties in IgniteSystemProperties; we will then
> > print
> > > out only those that are marked with the annotation.
> > >
> > > -Val
> > >
> > >
> > > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> > slava.kopti...@gmail.com
> > > >
> > > wrote:
> > >
> > > > Hello Ivan,
> > > >
> > > > > At least, we could just hide params that match a specific pattern
> > > > Yes, we can filter out all vm options that do not relate to Ignite,
> for
> > > > instance.
> > > >
> > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > Please do not rush. Let's listen to other community members. This
> > > question
> > > > is about security and it should not be discussed in a hurry (even
> > though
> > > it
> > > > looks like an obvious thing).
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > > >
> > > > > I suppose, that all normal users should not suffer from this
> > > > restrictions.
> > > > > Nobody will pass password using jvm options. It is absolutely
> insane,
> > > > > normal users pass passwords using environment variables.
> > > > >
> > > > > At least, we could just hide params that match specific pattern
> > > > >
> > > > > Ilya, go ahead, file ticket and prepare a PR.
> > > > >
> > > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> > slava.kopti...@gmail.com
> > > >:
> > > > >
> > > > > > Hello,
> > > > > >
> > > > > > Unfortunately, the user can pass its own system properties via
> JVM
> > > > > options
> > > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > > It does not seem, this approach is the best one. However, the
> user
> > > > should
> > > > > > have a "kostyl" in order to hide these properties and values in
> the
> > > log
> > > > > > file, IMHO.
> > > > > >
> > > > > > Thanks,
> > > > > > S.
> > > > > >
> > > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya <
> shishkovi...@gmail.com
> > >:
> > > > > >
> > > > > > > Hi Igniters,
> > > > > > >
> > > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till
> > > now,
> > > > > > method
> > > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > > >
> > > > > > > Is this behaviour actual now? Often, we should be able to get
> > from
> > > > logs
> > > > > > the
> > > > > > > actual VM options used at startup even if output of sensitive
> > data
> > > is
> > > > > > > restricted.
> > > > > > >
> > > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > > 2.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > > 3.
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Andrey, yep, you are right. This was just a quick idea. As for me, I just
don't want to repeat the same problem with compactFooter in thin client api
of ignite 2.x.

чт, 1 июл. 2021 г., 19:22 Andrey Mashenkov :

> >
> > I suppose that we should describe this more verbose and explicit. I
> > nevertheless suggest to also consider writing values this way:
> > - arr of fields names (if name is missed, corresponding field is nil)
> > - arr of rows (row as array, length equal to fields array)
>
>
> Ivan,
> I think GET and PUT operation parameters should be consistent.
> With PUT operation this way may be tricky.
>
> SQL INSERT operation (which is similar PUT operation) semantic allows
> skipping columns that have a default value.
> Assume we have smth like this:
>
> CREATE TABLE t1 (
>'id' INT;
>'colname' VARCHAR DEFAULT "abc";
> )
> INSERT INTO t1 VALUES(1)
>
> Actually, this will add a row (1, "abc")
>
> Your suggestion related to missed fields will not work this way as it is
> impossible to distinct
> case with 'null' value from the case with a default value.
>
>
> On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky 
> wrote:
>
> > > Here is the description of TUPLE_GET_ALL:
> > - UUID: table ID
> > - int: schema ID
> > - arr of arr: array of rows with values for all columns in given schema
> >
> > I suppose that we should describe this more verbose and explicit. I
> > nevertheless suggest to also consider writing values this way:
> > - arr of fields names (if name is missed, corresponding field is nil)
> > - arr of rows (row as array, length equal to fields array)
> >
> > It is quite simple and if we use str8 (it is more than enough for any
> utf-8
> > reasonable field name), overhead will be negligible, but realization of a
> > client will be way simpler
> >
> > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> >
> > > > No it isn't, I have carefully read code and IEP, in your code you
> write
> > > > schema id in each tuple.
> > >
> > > There is no code for batch operations yet.
> > >
> > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > > (nil when value is missing for a column)
> > >
> > > As you can see, schema ID is written once for all rows.
> > > A row is just a set of values according to the schema.
> > >
> > >
> > > > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > > > bytearray to internal api, and use msgpack throughout all wire
> > protocols,
> > > > as tarantool does.
> > >
> > > I agree. But this was decided before in IEP-54, and is out of scope for
> > > current IEP.
> > > Would you like to start a separate thread to discuss this? Or I can do
> > this
> > > a bit later.
> > >
> > > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > > This is described in all operations that include multiple tuples.
> > > > No it isn't, I have carefully read code and IEP, in your code you
> write
> > > > schema id in each tuple.
> > > >
> > > > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > > > bytearray to internal api, and use msgpack throughout all wire
> > protocols,
> > > > as tarantool does.
> > > >
> > > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > > >
> > > > > Ivan,
> > > > >
> > > > > >  that there is not neccesary to write schema versions in each row
> > > > > > in collectionof tuples
> > > > >
> > > > > This is described in all operations that include multiple tuples.
> > > > >
> > > > >
> > > > > > it is not clear from your code (probably
> > > > > > mistake?) how differ key tuples and value tuples from each other
> > > > >
> > > > > Key tuples include only key columns. Key columns come first in the
> > > > schema.
> > > > > Value tuples include all columns, key and value. Added "Key tuples"
> > > > > section.
> > > > >
> > > > >
> > > > > > As for me, these excercises with schema's doesn't worth a lot
> > > > >
> > > > > I'll add a benchmark and we'll see.
> > > > >
> > > > >
> > > > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I suppose, that there is not neccesary to write schema versions
> in
> > > each
> > > > > row
> > > > > > in collectionof tuples. Also it is not clear from your code
> > (probably
> > > > > > mistake?) how differ key tuples and value tuples from each other.
> > In
> > > > > > readTuple you always read full schema and check for full length.
> As
> > > for
> > > > > me,
> > > > > > these excercises with schema's doesn't worth a lot. I.e. postgres
> > > just
> > > > > > writes field names and then simpy rows with data. Saving few
> bytes
> > > > > doesn't
> > > > > > make much deal. Btw, msgpack has special types for short strings
> > > (i.e.
> > > > > > str8). It is much easier use it and write field name as is.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> > > > > >
> > > > > > > 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Not all of them are OK to be printed out. At the very least, we should have
a mechanism to exclude some of them. I would still go with opt-in rather
than opt-out though, but I guess that is up to a discussion.

-Val

On Thu, Jul 1, 2021 at 9:29 AM Ivan Daschinsky  wrote:

> This is security through obscurity, an obvious and a well-known anti
> pattern. I suppose that printing jvm options, that is registered by
> @IgniteSystemProperty annotation is an ideal variant
>
> чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko <
> valentin.kuliche...@gmail.com
> >:
>
> > Folks,
> >
> > *Anything* that a user provides to the system can potentially be
> considered
> > sensitive information. This includes the VM arguments. We can't predict
> > what exactly one can put there, so let's not make assumptions.
> >
> > When dealing with security, we should be as conservative as possible.
> That
> > said, I do not even agree with the pattern approach - there might be a
> > user's system property named IGNITE_xxx. It is also possible for our
> > internal properties to contain sensitive information (not all of them are
> > boolean flags).
> >
> > The only option I see is to print out specific properties for which we
> > agree that they are safe. For example, we can introduce an annotation
> that
> > would mark "safe" properties in IgniteSystemProperties; we will then
> print
> > out only those that are marked with the annotation.
> >
> > -Val
> >
> >
> > On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин <
> slava.kopti...@gmail.com
> > >
> > wrote:
> >
> > > Hello Ivan,
> > >
> > > > At least, we could just hide params that match a specific pattern
> > > Yes, we can filter out all vm options that do not relate to Ignite, for
> > > instance.
> > >
> > > > Ilya, go ahead, file ticket and prepare a PR.
> > > Please do not rush. Let's listen to other community members. This
> > question
> > > is about security and it should not be discussed in a hurry (even
> though
> > it
> > > looks like an obvious thing).
> > >
> > > Thanks,
> > > S.
> > >
> > > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> > >
> > > > I suppose, that all normal users should not suffer from this
> > > restrictions.
> > > > Nobody will pass password using jvm options. It is absolutely insane,
> > > > normal users pass passwords using environment variables.
> > > >
> > > > At least, we could just hide params that match specific pattern
> > > >
> > > > Ilya, go ahead, file ticket and prepare a PR.
> > > >
> > > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин <
> slava.kopti...@gmail.com
> > >:
> > > >
> > > > > Hello,
> > > > >
> > > > > Unfortunately, the user can pass its own system properties via JVM
> > > > options
> > > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > > It does not seem, this approach is the best one. However, the user
> > > should
> > > > > have a "kostyl" in order to hide these properties and values in the
> > log
> > > > > file, IMHO.
> > > > >
> > > > > Thanks,
> > > > > S.
> > > > >
> > > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya  >:
> > > > >
> > > > > > Hi Igniters,
> > > > > >
> > > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till
> > now,
> > > > > method
> > > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > > >
> > > > > > Is this behaviour actual now? Often, we should be able to get
> from
> > > logs
> > > > > the
> > > > > > actual VM options used at startup even if output of sensitive
> data
> > is
> > > > > > restricted.
> > > > > >
> > > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > > 2.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > > 3.
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

This is security through obscurity, an obvious and a well-known anti
pattern. I suppose that printing jvm options, that is registered by
@IgniteSystemProperty annotation is an ideal variant

чт, 1 июл. 2021 г., 19:25 Valentin Kulichenko :

> Folks,
>
> *Anything* that a user provides to the system can potentially be considered
> sensitive information. This includes the VM arguments. We can't predict
> what exactly one can put there, so let's not make assumptions.
>
> When dealing with security, we should be as conservative as possible. That
> said, I do not even agree with the pattern approach - there might be a
> user's system property named IGNITE_xxx. It is also possible for our
> internal properties to contain sensitive information (not all of them are
> boolean flags).
>
> The only option I see is to print out specific properties for which we
> agree that they are safe. For example, we can introduce an annotation that
> would mark "safe" properties in IgniteSystemProperties; we will then print
> out only those that are marked with the annotation.
>
> -Val
>
>
> On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин  >
> wrote:
>
> > Hello Ivan,
> >
> > > At least, we could just hide params that match a specific pattern
> > Yes, we can filter out all vm options that do not relate to Ignite, for
> > instance.
> >
> > > Ilya, go ahead, file ticket and prepare a PR.
> > Please do not rush. Let's listen to other community members. This
> question
> > is about security and it should not be discussed in a hurry (even though
> it
> > looks like an obvious thing).
> >
> > Thanks,
> > S.
> >
> > чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
> >
> > > I suppose, that all normal users should not suffer from this
> > restrictions.
> > > Nobody will pass password using jvm options. It is absolutely insane,
> > > normal users pass passwords using environment variables.
> > >
> > > At least, we could just hide params that match specific pattern
> > >
> > > Ilya, go ahead, file ticket and prepare a PR.
> > >
> > > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин  >:
> > >
> > > > Hello,
> > > >
> > > > Unfortunately, the user can pass its own system properties via JVM
> > > options
> > > > as follows: -DMY_SECRET_PASSWORD=123
> > > > It does not seem, this approach is the best one. However, the user
> > should
> > > > have a "kostyl" in order to hide these properties and values in the
> log
> > > > file, IMHO.
> > > >
> > > > Thanks,
> > > > S.
> > > >
> > > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :
> > > >
> > > > > Hi Igniters,
> > > > >
> > > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till
> now,
> > > > method
> > > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > > >
> > > > > Is this behaviour actual now? Often, we should be able to get from
> > logs
> > > > the
> > > > > actual VM options used at startup even if output of sensitive data
> is
> > > > > restricted.
> > > > >
> > > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > > 2.
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > > 3.
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Val, when we think about binary protocol, using java interface as an
example is a not good idea. Please, see a tarantool binary api as an
example. Implementor of the client can write it immediately (1)
[1] --
 https://www.tarantool.io/en/doc/latest/dev_guide/internals/box_protocol/


чт, 1 июл. 2021 г., 19:15 Valentin Kulichenko :

> Ivan,
>
> Regarding the API, please take a look at this package:
>
> https://github.com/apache/ignite-3/tree/main/modules/api/src/main/java/org/apache/ignite/table
>
> 'Table' is the primary API, which works with raw tuples. There are also
> multiple views on top of it, including KeyValueView and KeyValueBinaryView
> - these provide the key-value API.
>
> The ultimate goal is for the thin client to implement all these APIs.
>
> -Val
>
> On Thu, Jul 1, 2021 at 8:42 AM Igor Sapego  wrote:
>
> > Ivan, what are extra serde steps you are talking about?
> >
> > Best Regards,
> > Igor
> >
> >
> > On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky 
> > wrote:
> >
> > > > I agree. But this was decided before in IEP-54, and is out of scope
> for
> > > current IEP.
> > > Would you like to start a separate thread to discuss this? Or I can do
> > this
> > > a bit later.
> > >
> > > Great idea, let's discuss it. I suppose this will simplify many aspects
> > of
> > > realization and improve performance a lot
> > >
> > > чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :
> > >
> > > > > Here is the description of TUPLE_GET_ALL:
> > > > - UUID: table ID
> > > > - int: schema ID
> > > > - arr of arr: array of rows with values for all columns in given
> schema
> > > >
> > > > I suppose that we should describe this more verbose and explicit. I
> > > > nevertheless suggest to also consider writing values this way:
> > > > - arr of fields names (if name is missed, corresponding field is nil)
> > > > - arr of rows (row as array, length equal to fields array)
> > > >
> > > > It is quite simple and if we use str8 (it is more than enough for any
> > > > utf-8 reasonable field name), overhead will be negligible, but
> > > realization
> > > > of a client will be way simpler
> > > >
> > > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > > >
> > > >> > No it isn't, I have carefully read code and IEP, in your code you
> > > write
> > > >> > schema id in each tuple.
> > > >>
> > > >> There is no code for batch operations yet.
> > > >>
> > > >> Here is the description of TUPLE_GET_ALL:
> > > >> - UUID: table ID
> > > >> - int: schema ID
> > > >> - arr of arr: array of rows with values for all columns in given
> > schema
> > > >> (nil when value is missing for a column)
> > > >>
> > > >> As you can see, schema ID is written once for all rows.
> > > >> A row is just a set of values according to the schema.
> > > >>
> > > >>
> > > >> > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > >> > bytearray to internal api, and use msgpack throughout all wire
> > > >> protocols,
> > > >> > as tarantool does.
> > > >>
> > > >> I agree. But this was decided before in IEP-54, and is out of scope
> > for
> > > >> current IEP.
> > > >> Would you like to start a separate thread to discuss this? Or I can
> do
> > > >> this
> > > >> a bit later.
> > > >>
> > > >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky  >
> > > >> wrote:
> > > >>
> > > >> > > This is described in all operations that include multiple
> tuples.
> > > >> > No it isn't, I have carefully read code and IEP, in your code you
> > > write
> > > >> > schema id in each tuple.
> > > >> >
> > > >> > Also, my biggest concern -- extra serde step. I suppose we should
> > pass
> > > >> > bytearray to internal api, and use msgpack throughout all wire
> > > >> protocols,
> > > >> > as tarantool does.
> > > >> >
> > > >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > > >> >
> > > >> > > Ivan,
> > > >> > >
> > > >> > > >  that there is not neccesary to write schema versions in each
> > row
> > > >> > > > in collectionof tuples
> > > >> > >
> > > >> > > This is described in all operations that include multiple
> tuples.
> > > >> > >
> > > >> > >
> > > >> > > > it is not clear from your code (probably
> > > >> > > > mistake?) how differ key tuples and value tuples from each
> other
> > > >> > >
> > > >> > > Key tuples include only key columns. Key columns come first in
> the
> > > >> > schema.
> > > >> > > Value tuples include all columns, key and value. Added "Key
> > tuples"
> > > >> > > section.
> > > >> > >
> > > >> > >
> > > >> > > > As for me, these excercises with schema's doesn't worth a lot
> > > >> > >
> > > >> > > I'll add a benchmark and we'll see.
> > > >> > >
> > > >> > >
> > > >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> > ivanda...@gmail.com
> > > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > I suppose, that there is not neccesary to write schema
> versions
> > in
> > > >> each
> > > >> > > row
> > > >> > > > in collectionof tuples. Also it is not clear from your code
> > > >> (probably
> > > >> > > > mistake?) how differ 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Valentin Kulichenko]

Folks,

*Anything* that a user provides to the system can potentially be considered
sensitive information. This includes the VM arguments. We can't predict
what exactly one can put there, so let's not make assumptions.

When dealing with security, we should be as conservative as possible. That
said, I do not even agree with the pattern approach - there might be a
user's system property named IGNITE_xxx. It is also possible for our
internal properties to contain sensitive information (not all of them are
boolean flags).

The only option I see is to print out specific properties for which we
agree that they are safe. For example, we can introduce an annotation that
would mark "safe" properties in IgniteSystemProperties; we will then print
out only those that are marked with the annotation.

-Val


On Thu, Jul 1, 2021 at 7:07 AM Вячеслав Коптилин 
wrote:

> Hello Ivan,
>
> > At least, we could just hide params that match a specific pattern
> Yes, we can filter out all vm options that do not relate to Ignite, for
> instance.
>
> > Ilya, go ahead, file ticket and prepare a PR.
> Please do not rush. Let's listen to other community members. This question
> is about security and it should not be discussed in a hurry (even though it
> looks like an obvious thing).
>
> Thanks,
> S.
>
> чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :
>
> > I suppose, that all normal users should not suffer from this
> restrictions.
> > Nobody will pass password using jvm options. It is absolutely insane,
> > normal users pass passwords using environment variables.
> >
> > At least, we could just hide params that match specific pattern
> >
> > Ilya, go ahead, file ticket and prepare a PR.
> >
> > чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин :
> >
> > > Hello,
> > >
> > > Unfortunately, the user can pass its own system properties via JVM
> > options
> > > as follows: -DMY_SECRET_PASSWORD=123
> > > It does not seem, this approach is the best one. However, the user
> should
> > > have a "kostyl" in order to hide these properties and values in the log
> > > file, IMHO.
> > >
> > > Thanks,
> > > S.
> > >
> > > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :
> > >
> > > > Hi Igniters,
> > > >
> > > > This feature [1, 2] prevents logging of the VM arguments when
> > > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now,
> > > method
> > > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > > >
> > > > Is this behaviour actual now? Often, we should be able to get from
> logs
> > > the
> > > > actual VM options used at startup even if output of sensitive data is
> > > > restricted.
> > > >
> > > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > > 2.
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > > 3.
> > > >
> > > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Andrey Mashenkov]

>
> I suppose that we should describe this more verbose and explicit. I
> nevertheless suggest to also consider writing values this way:
> - arr of fields names (if name is missed, corresponding field is nil)
> - arr of rows (row as array, length equal to fields array)


Ivan,
I think GET and PUT operation parameters should be consistent.
With PUT operation this way may be tricky.

SQL INSERT operation (which is similar PUT operation) semantic allows
skipping columns that have a default value.
Assume we have smth like this:

CREATE TABLE t1 (
   'id' INT;
   'colname' VARCHAR DEFAULT "abc";
)
INSERT INTO t1 VALUES(1)

Actually, this will add a row (1, "abc")

Your suggestion related to missed fields will not work this way as it is
impossible to distinct
case with 'null' value from the case with a default value.


On Thu, Jul 1, 2021 at 5:51 PM Ivan Daschinsky  wrote:

> > Here is the description of TUPLE_GET_ALL:
> - UUID: table ID
> - int: schema ID
> - arr of arr: array of rows with values for all columns in given schema
>
> I suppose that we should describe this more verbose and explicit. I
> nevertheless suggest to also consider writing values this way:
> - arr of fields names (if name is missed, corresponding field is nil)
> - arr of rows (row as array, length equal to fields array)
>
> It is quite simple and if we use str8 (it is more than enough for any utf-8
> reasonable field name), overhead will be negligible, but realization of a
> client will be way simpler
>
> чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
>
> > > No it isn't, I have carefully read code and IEP, in your code you write
> > > schema id in each tuple.
> >
> > There is no code for batch operations yet.
> >
> > Here is the description of TUPLE_GET_ALL:
> > - UUID: table ID
> > - int: schema ID
> > - arr of arr: array of rows with values for all columns in given schema
> > (nil when value is missing for a column)
> >
> > As you can see, schema ID is written once for all rows.
> > A row is just a set of values according to the schema.
> >
> >
> > > Also, my biggest concern -- extra serde step. I suppose we should pass
> > > bytearray to internal api, and use msgpack throughout all wire
> protocols,
> > > as tarantool does.
> >
> > I agree. But this was decided before in IEP-54, and is out of scope for
> > current IEP.
> > Would you like to start a separate thread to discuss this? Or I can do
> this
> > a bit later.
> >
> > On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > wrote:
> >
> > > > This is described in all operations that include multiple tuples.
> > > No it isn't, I have carefully read code and IEP, in your code you write
> > > schema id in each tuple.
> > >
> > > Also, my biggest concern -- extra serde step. I suppose we should pass
> > > bytearray to internal api, and use msgpack throughout all wire
> protocols,
> > > as tarantool does.
> > >
> > > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > >  that there is not neccesary to write schema versions in each row
> > > > > in collectionof tuples
> > > >
> > > > This is described in all operations that include multiple tuples.
> > > >
> > > >
> > > > > it is not clear from your code (probably
> > > > > mistake?) how differ key tuples and value tuples from each other
> > > >
> > > > Key tuples include only key columns. Key columns come first in the
> > > schema.
> > > > Value tuples include all columns, key and value. Added "Key tuples"
> > > > section.
> > > >
> > > >
> > > > > As for me, these excercises with schema's doesn't worth a lot
> > > >
> > > > I'll add a benchmark and we'll see.
> > > >
> > > >
> > > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > I suppose, that there is not neccesary to write schema versions in
> > each
> > > > row
> > > > > in collectionof tuples. Also it is not clear from your code
> (probably
> > > > > mistake?) how differ key tuples and value tuples from each other.
> In
> > > > > readTuple you always read full schema and check for full length. As
> > for
> > > > me,
> > > > > these excercises with schema's doesn't worth a lot. I.e. postgres
> > just
> > > > > writes field names and then simpy rows with data. Saving few bytes
> > > > doesn't
> > > > > make much deal. Btw, msgpack has special types for short strings
> > (i.e.
> > > > > str8). It is much easier use it and write field name as is.
> > > > >
> > > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan, tuple serialization section added to the IEP, let me know
> if
> > it
> > > > is
> > > > > > clear enough.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I can't find any description of tuple serialization in IEP,
> only
> > in
> > > > > code
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn  >:
> > > > > > >
> > > > > > > > Ivan,
> > > > > > > >
> > > > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Igor, as you can see from the code, we read data from msgpack and build the
tuple from scratch. Then we serialize this tuple again when send it to
backups. I presume this is absolutely unnecessary

чт, 1 июл. 2021 г., 18:42 Igor Sapego :

> Ivan, what are extra serde steps you are talking about?
>
> Best Regards,
> Igor
>
>
> On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky 
> wrote:
>
> > > I agree. But this was decided before in IEP-54, and is out of scope for
> > current IEP.
> > Would you like to start a separate thread to discuss this? Or I can do
> this
> > a bit later.
> >
> > Great idea, let's discuss it. I suppose this will simplify many aspects
> of
> > realization and improve performance a lot
> >
> > чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > > utf-8 reasonable field name), overhead will be negligible, but
> > realization
> > > of a client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > >
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >>
> > >> There is no code for batch operations yet.
> > >>
> > >> Here is the description of TUPLE_GET_ALL:
> > >> - UUID: table ID
> > >> - int: schema ID
> > >> - arr of arr: array of rows with values for all columns in given
> schema
> > >> (nil when value is missing for a column)
> > >>
> > >> As you can see, schema ID is written once for all rows.
> > >> A row is just a set of values according to the schema.
> > >>
> > >>
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >>
> > >> I agree. But this was decided before in IEP-54, and is out of scope
> for
> > >> current IEP.
> > >> Would you like to start a separate thread to discuss this? Or I can do
> > >> this
> > >> a bit later.
> > >>
> > >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > >> wrote:
> > >>
> > >> > > This is described in all operations that include multiple tuples.
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >> >
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >> >
> > >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > > >  that there is not neccesary to write schema versions in each
> row
> > >> > > > in collectionof tuples
> > >> > >
> > >> > > This is described in all operations that include multiple tuples.
> > >> > >
> > >> > >
> > >> > > > it is not clear from your code (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each other
> > >> > >
> > >> > > Key tuples include only key columns. Key columns come first in the
> > >> > schema.
> > >> > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > >> > > section.
> > >> > >
> > >> > >
> > >> > > > As for me, these excercises with schema's doesn't worth a lot
> > >> > >
> > >> > > I'll add a benchmark and we'll see.
> > >> > >
> > >> > >
> > >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > I suppose, that there is not neccesary to write schema versions
> in
> > >> each
> > >> > > row
> > >> > > > in collectionof tuples. Also it is not clear from your code
> > >> (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each
> other.
> > In
> > >> > > > readTuple you always read full schema and check for full length.
> > As
> > >> for
> > >> > > me,
> > >> > > > these excercises with schema's doesn't worth a lot. I.e.
> postgres
> > >> just
> > >> > > > writes field names and then simpy rows with data. Saving few
> bytes
> > >> > > doesn't
> > >> > > > make much deal. Btw, msgpack has special types for short strings
> > >> (i.e.
> > >> > > > str8). It is much easier use it and write field name as is.
> > >> > > >
> > >> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn  >:
> > >> > > >
> > >> > > > > Ivan, tuple serialization section added to the IEP, let me
> know
> > >> if it
> > >> > > is
> > >> > > > > clear enough.
> > >> > > > >
> > >> > > > > Thanks!
> > >> > > > >
> > >> > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Valentin Kulichenko]

Ivan,

Regarding the API, please take a look at this package:
https://github.com/apache/ignite-3/tree/main/modules/api/src/main/java/org/apache/ignite/table

'Table' is the primary API, which works with raw tuples. There are also
multiple views on top of it, including KeyValueView and KeyValueBinaryView
- these provide the key-value API.

The ultimate goal is for the thin client to implement all these APIs.

-Val

On Thu, Jul 1, 2021 at 8:42 AM Igor Sapego  wrote:

> Ivan, what are extra serde steps you are talking about?
>
> Best Regards,
> Igor
>
>
> On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky 
> wrote:
>
> > > I agree. But this was decided before in IEP-54, and is out of scope for
> > current IEP.
> > Would you like to start a separate thread to discuss this? Or I can do
> this
> > a bit later.
> >
> > Great idea, let's discuss it. I suppose this will simplify many aspects
> of
> > realization and improve performance a lot
> >
> > чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :
> >
> > > > Here is the description of TUPLE_GET_ALL:
> > > - UUID: table ID
> > > - int: schema ID
> > > - arr of arr: array of rows with values for all columns in given schema
> > >
> > > I suppose that we should describe this more verbose and explicit. I
> > > nevertheless suggest to also consider writing values this way:
> > > - arr of fields names (if name is missed, corresponding field is nil)
> > > - arr of rows (row as array, length equal to fields array)
> > >
> > > It is quite simple and if we use str8 (it is more than enough for any
> > > utf-8 reasonable field name), overhead will be negligible, but
> > realization
> > > of a client will be way simpler
> > >
> > > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> > >
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >>
> > >> There is no code for batch operations yet.
> > >>
> > >> Here is the description of TUPLE_GET_ALL:
> > >> - UUID: table ID
> > >> - int: schema ID
> > >> - arr of arr: array of rows with values for all columns in given
> schema
> > >> (nil when value is missing for a column)
> > >>
> > >> As you can see, schema ID is written once for all rows.
> > >> A row is just a set of values according to the schema.
> > >>
> > >>
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >>
> > >> I agree. But this was decided before in IEP-54, and is out of scope
> for
> > >> current IEP.
> > >> Would you like to start a separate thread to discuss this? Or I can do
> > >> this
> > >> a bit later.
> > >>
> > >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> > >> wrote:
> > >>
> > >> > > This is described in all operations that include multiple tuples.
> > >> > No it isn't, I have carefully read code and IEP, in your code you
> > write
> > >> > schema id in each tuple.
> > >> >
> > >> > Also, my biggest concern -- extra serde step. I suppose we should
> pass
> > >> > bytearray to internal api, and use msgpack throughout all wire
> > >> protocols,
> > >> > as tarantool does.
> > >> >
> > >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> > >> >
> > >> > > Ivan,
> > >> > >
> > >> > > >  that there is not neccesary to write schema versions in each
> row
> > >> > > > in collectionof tuples
> > >> > >
> > >> > > This is described in all operations that include multiple tuples.
> > >> > >
> > >> > >
> > >> > > > it is not clear from your code (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each other
> > >> > >
> > >> > > Key tuples include only key columns. Key columns come first in the
> > >> > schema.
> > >> > > Value tuples include all columns, key and value. Added "Key
> tuples"
> > >> > > section.
> > >> > >
> > >> > >
> > >> > > > As for me, these excercises with schema's doesn't worth a lot
> > >> > >
> > >> > > I'll add a benchmark and we'll see.
> > >> > >
> > >> > >
> > >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky <
> ivanda...@gmail.com
> > >
> > >> > > wrote:
> > >> > >
> > >> > > > I suppose, that there is not neccesary to write schema versions
> in
> > >> each
> > >> > > row
> > >> > > > in collectionof tuples. Also it is not clear from your code
> > >> (probably
> > >> > > > mistake?) how differ key tuples and value tuples from each
> other.
> > In
> > >> > > > readTuple you always read full schema and check for full length.
> > As
> > >> for
> > >> > > me,
> > >> > > > these excercises with schema's doesn't worth a lot. I.e.
> postgres
> > >> just
> > >> > > > writes field names and then simpy rows with data. Saving few
> bytes
> > >> > > doesn't
> > >> > > > make much deal. Btw, msgpack has special types for short strings
> > >> (i.e.
> > >> > > > str8). It is much easier use it and write field name as is.
> > >> > > >
> > >> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn  >:
> > >> > > >
> > >> > > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Igor Sapego]

Ivan, what are extra serde steps you are talking about?

Best Regards,
Igor


On Thu, Jul 1, 2021 at 5:52 PM Ivan Daschinsky  wrote:

> > I agree. But this was decided before in IEP-54, and is out of scope for
> current IEP.
> Would you like to start a separate thread to discuss this? Or I can do this
> a bit later.
>
> Great idea, let's discuss it. I suppose this will simplify many aspects of
> realization and improve performance a lot
>
> чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :
>
> > > Here is the description of TUPLE_GET_ALL:
> > - UUID: table ID
> > - int: schema ID
> > - arr of arr: array of rows with values for all columns in given schema
> >
> > I suppose that we should describe this more verbose and explicit. I
> > nevertheless suggest to also consider writing values this way:
> > - arr of fields names (if name is missed, corresponding field is nil)
> > - arr of rows (row as array, length equal to fields array)
> >
> > It is quite simple and if we use str8 (it is more than enough for any
> > utf-8 reasonable field name), overhead will be negligible, but
> realization
> > of a client will be way simpler
> >
> > чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
> >
> >> > No it isn't, I have carefully read code and IEP, in your code you
> write
> >> > schema id in each tuple.
> >>
> >> There is no code for batch operations yet.
> >>
> >> Here is the description of TUPLE_GET_ALL:
> >> - UUID: table ID
> >> - int: schema ID
> >> - arr of arr: array of rows with values for all columns in given schema
> >> (nil when value is missing for a column)
> >>
> >> As you can see, schema ID is written once for all rows.
> >> A row is just a set of values according to the schema.
> >>
> >>
> >> > Also, my biggest concern -- extra serde step. I suppose we should pass
> >> > bytearray to internal api, and use msgpack throughout all wire
> >> protocols,
> >> > as tarantool does.
> >>
> >> I agree. But this was decided before in IEP-54, and is out of scope for
> >> current IEP.
> >> Would you like to start a separate thread to discuss this? Or I can do
> >> this
> >> a bit later.
> >>
> >> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> >> wrote:
> >>
> >> > > This is described in all operations that include multiple tuples.
> >> > No it isn't, I have carefully read code and IEP, in your code you
> write
> >> > schema id in each tuple.
> >> >
> >> > Also, my biggest concern -- extra serde step. I suppose we should pass
> >> > bytearray to internal api, and use msgpack throughout all wire
> >> protocols,
> >> > as tarantool does.
> >> >
> >> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> >> >
> >> > > Ivan,
> >> > >
> >> > > >  that there is not neccesary to write schema versions in each row
> >> > > > in collectionof tuples
> >> > >
> >> > > This is described in all operations that include multiple tuples.
> >> > >
> >> > >
> >> > > > it is not clear from your code (probably
> >> > > > mistake?) how differ key tuples and value tuples from each other
> >> > >
> >> > > Key tuples include only key columns. Key columns come first in the
> >> > schema.
> >> > > Value tuples include all columns, key and value. Added "Key tuples"
> >> > > section.
> >> > >
> >> > >
> >> > > > As for me, these excercises with schema's doesn't worth a lot
> >> > >
> >> > > I'll add a benchmark and we'll see.
> >> > >
> >> > >
> >> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky  >
> >> > > wrote:
> >> > >
> >> > > > I suppose, that there is not neccesary to write schema versions in
> >> each
> >> > > row
> >> > > > in collectionof tuples. Also it is not clear from your code
> >> (probably
> >> > > > mistake?) how differ key tuples and value tuples from each other.
> In
> >> > > > readTuple you always read full schema and check for full length.
> As
> >> for
> >> > > me,
> >> > > > these excercises with schema's doesn't worth a lot. I.e. postgres
> >> just
> >> > > > writes field names and then simpy rows with data. Saving few bytes
> >> > > doesn't
> >> > > > make much deal. Btw, msgpack has special types for short strings
> >> (i.e.
> >> > > > str8). It is much easier use it and write field name as is.
> >> > > >
> >> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> >> > > >
> >> > > > > Ivan, tuple serialization section added to the IEP, let me know
> >> if it
> >> > > is
> >> > > > > clear enough.
> >> > > > >
> >> > > > > Thanks!
> >> > > > >
> >> > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky <
> >> ivanda...@gmail.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > I can't find any description of tuple serialization in IEP,
> >> only in
> >> > > > code
> >> > > > > >
> >> > > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn <
> ptupit...@apache.org
> >> >:
> >> > > > > >
> >> > > > > > > Ivan,
> >> > > > > > >
> >> > > > > > > 0. The IEP is not in progress, it is ready for review and
> >> > > discussion.
> >> > > > > > > 1. Tuple serialization is described in the IEP and
> >> demonstrated
> >> > in
> >> > > > the
> >> > > > > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

> I agree. But this was decided before in IEP-54, and is out of scope for
current IEP.
Would you like to start a separate thread to discuss this? Or I can do this
a bit later.

Great idea, let's discuss it. I suppose this will simplify many aspects of
realization and improve performance a lot

чт, 1 июл. 2021 г., 17:50 Ivan Daschinsky :

> > Here is the description of TUPLE_GET_ALL:
> - UUID: table ID
> - int: schema ID
> - arr of arr: array of rows with values for all columns in given schema
>
> I suppose that we should describe this more verbose and explicit. I
> nevertheless suggest to also consider writing values this way:
> - arr of fields names (if name is missed, corresponding field is nil)
> - arr of rows (row as array, length equal to fields array)
>
> It is quite simple and if we use str8 (it is more than enough for any
> utf-8 reasonable field name), overhead will be negligible, but realization
> of a client will be way simpler
>
> чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :
>
>> > No it isn't, I have carefully read code and IEP, in your code you write
>> > schema id in each tuple.
>>
>> There is no code for batch operations yet.
>>
>> Here is the description of TUPLE_GET_ALL:
>> - UUID: table ID
>> - int: schema ID
>> - arr of arr: array of rows with values for all columns in given schema
>> (nil when value is missing for a column)
>>
>> As you can see, schema ID is written once for all rows.
>> A row is just a set of values according to the schema.
>>
>>
>> > Also, my biggest concern -- extra serde step. I suppose we should pass
>> > bytearray to internal api, and use msgpack throughout all wire
>> protocols,
>> > as tarantool does.
>>
>> I agree. But this was decided before in IEP-54, and is out of scope for
>> current IEP.
>> Would you like to start a separate thread to discuss this? Or I can do
>> this
>> a bit later.
>>
>> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
>> wrote:
>>
>> > > This is described in all operations that include multiple tuples.
>> > No it isn't, I have carefully read code and IEP, in your code you write
>> > schema id in each tuple.
>> >
>> > Also, my biggest concern -- extra serde step. I suppose we should pass
>> > bytearray to internal api, and use msgpack throughout all wire
>> protocols,
>> > as tarantool does.
>> >
>> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
>> >
>> > > Ivan,
>> > >
>> > > >  that there is not neccesary to write schema versions in each row
>> > > > in collectionof tuples
>> > >
>> > > This is described in all operations that include multiple tuples.
>> > >
>> > >
>> > > > it is not clear from your code (probably
>> > > > mistake?) how differ key tuples and value tuples from each other
>> > >
>> > > Key tuples include only key columns. Key columns come first in the
>> > schema.
>> > > Value tuples include all columns, key and value. Added "Key tuples"
>> > > section.
>> > >
>> > >
>> > > > As for me, these excercises with schema's doesn't worth a lot
>> > >
>> > > I'll add a benchmark and we'll see.
>> > >
>> > >
>> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
>> > > wrote:
>> > >
>> > > > I suppose, that there is not neccesary to write schema versions in
>> each
>> > > row
>> > > > in collectionof tuples. Also it is not clear from your code
>> (probably
>> > > > mistake?) how differ key tuples and value tuples from each other. In
>> > > > readTuple you always read full schema and check for full length. As
>> for
>> > > me,
>> > > > these excercises with schema's doesn't worth a lot. I.e. postgres
>> just
>> > > > writes field names and then simpy rows with data. Saving few bytes
>> > > doesn't
>> > > > make much deal. Btw, msgpack has special types for short strings
>> (i.e.
>> > > > str8). It is much easier use it and write field name as is.
>> > > >
>> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
>> > > >
>> > > > > Ivan, tuple serialization section added to the IEP, let me know
>> if it
>> > > is
>> > > > > clear enough.
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky <
>> ivanda...@gmail.com>
>> > > > > wrote:
>> > > > >
>> > > > > > I can't find any description of tuple serialization in IEP,
>> only in
>> > > > code
>> > > > > >
>> > > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn > >:
>> > > > > >
>> > > > > > > Ivan,
>> > > > > > >
>> > > > > > > 0. The IEP is not in progress, it is ready for review and
>> > > discussion.
>> > > > > > > 1. Tuple serialization is described in the IEP and
>> demonstrated
>> > in
>> > > > the
>> > > > > > PoC
>> > > > > > > (see ClientMessageHandler#readTuple), let me know if more
>> details
>> > > are
>> > > > > > > required
>> > > > > > > 2. Tuple schema serialization is described in SCHEMAS_GET
>> > section.
>> > > > > Table
>> > > > > > > schema (configuration) needs more details, you are right -
>> I'll
>> > add
>> > > > > them.
>> > > > > > > 3. This IEP is about tables (tuple-based) API only, since it
>> is
>> > the
>> > > > > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

> Here is the description of TUPLE_GET_ALL:
- UUID: table ID
- int: schema ID
- arr of arr: array of rows with values for all columns in given schema

I suppose that we should describe this more verbose and explicit. I
nevertheless suggest to also consider writing values this way:
- arr of fields names (if name is missed, corresponding field is nil)
- arr of rows (row as array, length equal to fields array)

It is quite simple and if we use str8 (it is more than enough for any utf-8
reasonable field name), overhead will be negligible, but realization of a
client will be way simpler

чт, 1 июл. 2021 г., 16:57 Pavel Tupitsyn :

> > No it isn't, I have carefully read code and IEP, in your code you write
> > schema id in each tuple.
>
> There is no code for batch operations yet.
>
> Here is the description of TUPLE_GET_ALL:
> - UUID: table ID
> - int: schema ID
> - arr of arr: array of rows with values for all columns in given schema
> (nil when value is missing for a column)
>
> As you can see, schema ID is written once for all rows.
> A row is just a set of values according to the schema.
>
>
> > Also, my biggest concern -- extra serde step. I suppose we should pass
> > bytearray to internal api, and use msgpack throughout all wire protocols,
> > as tarantool does.
>
> I agree. But this was decided before in IEP-54, and is out of scope for
> current IEP.
> Would you like to start a separate thread to discuss this? Or I can do this
> a bit later.
>
> On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky 
> wrote:
>
> > > This is described in all operations that include multiple tuples.
> > No it isn't, I have carefully read code and IEP, in your code you write
> > schema id in each tuple.
> >
> > Also, my biggest concern -- extra serde step. I suppose we should pass
> > bytearray to internal api, and use msgpack throughout all wire protocols,
> > as tarantool does.
> >
> > чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
> >
> > > Ivan,
> > >
> > > >  that there is not neccesary to write schema versions in each row
> > > > in collectionof tuples
> > >
> > > This is described in all operations that include multiple tuples.
> > >
> > >
> > > > it is not clear from your code (probably
> > > > mistake?) how differ key tuples and value tuples from each other
> > >
> > > Key tuples include only key columns. Key columns come first in the
> > schema.
> > > Value tuples include all columns, key and value. Added "Key tuples"
> > > section.
> > >
> > >
> > > > As for me, these excercises with schema's doesn't worth a lot
> > >
> > > I'll add a benchmark and we'll see.
> > >
> > >
> > > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > I suppose, that there is not neccesary to write schema versions in
> each
> > > row
> > > > in collectionof tuples. Also it is not clear from your code (probably
> > > > mistake?) how differ key tuples and value tuples from each other. In
> > > > readTuple you always read full schema and check for full length. As
> for
> > > me,
> > > > these excercises with schema's doesn't worth a lot. I.e. postgres
> just
> > > > writes field names and then simpy rows with data. Saving few bytes
> > > doesn't
> > > > make much deal. Btw, msgpack has special types for short strings
> (i.e.
> > > > str8). It is much easier use it and write field name as is.
> > > >
> > > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> > > >
> > > > > Ivan, tuple serialization section added to the IEP, let me know if
> it
> > > is
> > > > > clear enough.
> > > > >
> > > > > Thanks!
> > > > >
> > > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I can't find any description of tuple serialization in IEP, only
> in
> > > > code
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > > > > >
> > > > > > > Ivan,
> > > > > > >
> > > > > > > 0. The IEP is not in progress, it is ready for review and
> > > discussion.
> > > > > > > 1. Tuple serialization is described in the IEP and demonstrated
> > in
> > > > the
> > > > > > PoC
> > > > > > > (see ClientMessageHandler#readTuple), let me know if more
> details
> > > are
> > > > > > > required
> > > > > > > 2. Tuple schema serialization is described in SCHEMAS_GET
> > section.
> > > > > Table
> > > > > > > schema (configuration) needs more details, you are right - I'll
> > add
> > > > > them.
> > > > > > > 3. This IEP is about tables (tuple-based) API only, since it is
> > the
> > > > > only
> > > > > > > API that we have right now, as noted in Risks and Assumptions.
> > > > > > >
> > > > > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky <
> > > ivanda...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Also, is there any clear information about KV api? Is there
> any
> > > > plan
> > > > > to
> > > > > > > > implement it? Or is there any proposal about it?
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky <
> ivanda...@gmail.com
> > >:
> > > > > > > 

Re: Apache Ignite 2.11

[Nikita Safonov]

Guys, I've created the following ticket:
https://issues.apache.org/jira/browse/IGNITE-15046

Please feel free to comment or edit it.

If there are no objections, I'll create a PR soon.

Regards,
Nikita

чт, 1 июл. 2021 г. в 16:54, Nikita Safonov :

> Ivan, thank you for the links!
>
> Well, I can still find the docs on Python Thin Client under the "Thin
> Сlients"->"Python Thin Client" in 2.10:
> https://ignite.apache.org/docs/2.10.0/thin-clients/python-thin-client
>
> So, I can replace the contents of  "Thin Сlients"->"Python Thin Client"
> page with a link to the
> https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/ and
> a short note as follows:
> "To see the documentation on Python Thin Client, please click here[link to
> the readthedocs.io
> ].
> +
> And I can also change the link here
> 
>  (after
> "Ignite provides the following thin clients:", so that it would lead to the
> readthedocs.io
> 
>  too).
>
> Alex, my bad, I misread the first question and thought that all the docs
> on thin clients have been migrated to the readthedocs.io
> .
> The articles are absolutely OK :)
>
> Regards,
> Nikita
>
> чт, 1 июл. 2021 г. в 16:44, Alex Plehanov :
>
>> Nikita,
>>
>> What's wrong with platform-specific thin-client articles? These articles
>> describe how to configure and use specific thin client implementation, why
>> should we remove them?
>>
>> чт, 1 июл. 2021 г. в 16:06, Ivan Daschinsky :
>>
>>> Binary prorocol description should stand and improved (there are some
>>> mistakes in this article)
>>>
>>> чт, 1 июл. 2021 г., 16:05 Ivan Daschinsky :
>>>
>>> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
>>> >
>>> > AFAIK, thin clients have been already removed since 2.10
>>> >
>>> > чт, 1 июл. 2021 г., 15:57 Nikita Safonov :
>>> >
>>> >> Hi guys!
>>> >>
>>> >> Sorry, totally missed this email.
>>> >>
>>> >> I can do this right away.
>>> >>
>>> >> Just a couple of questions first:
>>> >>
>>> >> [1] On the website, under the Thin Clients tab, we have the following
>>> >> articles:
>>> >>
>>> >> Thin Clients
>>> >> Thin Clients Overview
>>> >> Java Thin Client
>>> >> .NET Thin Client
>>> >> C++ Thin Client
>>> >> Python Thin Client
>>> >> PHP Thin Client
>>> >> Node.js Thin Client
>>> >> Binary Client Protocol
>>> >>
>>> >> I do believe that the introduction ( "Thin Clients" - the first one)
>>> >> should
>>> >> remain, while we can remove the rest (by the way, what about Binary
>>> Client
>>> >> Protocol? Should it remain too?).
>>> >> What do you think about this?
>>> >>
>>> >> [2] Could you please indicate the place in the docs where the link to
>>> the
>>> >> actual readthedocs.io documentation is? I didn't manage to find it.
>>> >> If we don't have it, I would place it at the beginning of the Thin
>>> >> clients'
>>> >> introduction article.
>>> >>
>>> >> Regards,
>>> >> Nikita
>>> >>
>>> >>
>>> >> ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :
>>> >>
>>> >> > Hi, Nikita!
>>> >> >
>>> >> > Your help with documentation is really needed. Can you describe how
>>> we
>>> >> can
>>> >> > start with it?
>>> >> >
>>> >> > On 2021/06/09 10:06:09, Nikita Safonov 
>>> >> wrote:
>>> >> > > Hello Igniters!
>>> >> > >
>>> >> > > I guess that AI 2.11 will include documentation work.
>>> >> > > I'll be glad to help with this.
>>> >> > > And I'll be happy to know the scope of features that we need to
>>> >> document.
>>> >> > >
>>> >> > > Regards,
>>> >> > > Nikita
>>> >> > >
>>> >> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
>>> >> > >
>>> >> > > > hello,
>>> >> > > >
>>> >> > > > can someone can help to merge the following patch into master?
>>> >> > > >
>>> >> > > > https://issues.apache.org/jira/browse/IGNITE-12852
>>> >> > > >
>>> >> > > > This patch has been completed for a long time, and I think it
>>> is a
>>> >> > > > valuable one.
>>> >> > > >
>>> >> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
>>> >> > > > > Sure, thanks for your reply
>>> >> > > > >
>>> >> > > > > Folks, could you please advice how to setup JIRA account
>>> >> integration
>>> >> > for
>>> >> > > > non-committers?
>>> >> > > > >
>>> >> > > > > For the page
>>> >> > > > >
>>> >> >
>>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
>>> >> > > > > I can see all issues, but Alexey's login seems to be not
>>> >> sufficient.
>>> >> > Can
>>> >> > > > wiki admins set up smth? Or Alexey should do some setup on his
>>> own?
>>> >> > > > >
>>> >> > > > > On 2021/06/03 14:55:25, Maksim Timonin <
>>> timonin.ma...@gmail.com>
>>> >> > wrote:
>>> >> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear
>>> >> there
>>> >> > are
>>> >> > > > >> chances for 

Re: [Announcement] Apache Ignite 2.11 Code Freeze started

[Alexey Gidaspov]

Hi, Pavel. 

I think, it looks like blocker. Please cherry-pick it to 2.11 release branch 

On 2021/07/01 09:29:57, Pavel Pereslegin  wrote: 
> Hello, Alexey!
> 
> Is it possible to include a hotfix that corrects the command syntax
> output in the control script? [1]
> 
> This bug can significantly complicate the use of the snapshot restore
> function (one of the important features of 2.11). In addition, this
> may raise a number of questions to the product support, which we can
> prevent by adding this patch in 2.11.
> 
> This patch does not affect any functions other than the "help" output
> of the control script.
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-14989
> 
> чт, 1 июл. 2021 г. в 11:56, Alexey Gidaspov :
> >
> > Hi, Iilya!
> >
> > As I can see, this feature highly improves debugging during incidents. So I 
> > think we can call it blocker and cherry-pick to ignite-2.11 branch
> >
> > On 2021/06/30 20:26:43, Shishkov Ilya  wrote:
> > > Hello, Alexey!
> > > Is it possible to add system views for BaselineNode attributes [1] and
> > > corresponding documentation [2] to 2.11?
> > >
> > > 1. https://issues.apache.org/jira/browse/IGNITE-15007
> > > 2. https://issues.apache.org/jira/browse/IGNITE-15028
> > >
> > > ср, 30 июн. 2021 г. в 11:07, Nikita Amelchev :
> > >
> > > > Thanks! I have cherry-picked the commit to the 2.11 branch.
> > > >
> > > > ср, 30 июн. 2021 г. в 11:00, Alexey Gidaspov :
> > > > >
> > > > > Hi, Nikita!
> > > > >
> > > > > I think it's important fix and should be included in 2.11 release. 
> > > > > I've
> > > > tagged ticket by fixVersion 2.11. Can you cherry-pick it to ignite-2.11
> > > > branch? And please fill release notes or delete flag.
> > > > >
> > > > > On 2021/06/30 07:55:04, Nikita Amelchev  wrote:
> > > > > > Hello, Alexey.
> > > > > >
> > > > > > I suggest adding to the 2.11 scope the resolved issue [1]. It fixes
> > > > > > incorrect values of cache, cache groups, data region metrics after
> > > > > > cluster reactivation.
> > > > > > WDYT?
> > > > > >
> > > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14990
> > > > > >
> > > > > > вт, 29 июн. 2021 г. в 15:09, Alexey Gidaspov :
> > > > > > >
> > > > > > > Ok, we can add this fix to release scope.
> > > > > > >
> > > > > > > On 2021/06/29 08:36:00, Сурков Александр
> > > > Викторович wrote:
> > > > > > > > Hi Alexey.
> > > > > > > >
> > > > > > > > I think need add ticket: JmxMetricExporter fails to export
> > > > discovery metrics - https://issues.apache.org/jira/browse/IGNITE-14376
> > > > > > > >
> > > > > > > > On 2021/06/15 14:43:55, Alexey Gidaspov  wrote:
> > > > > > > > > Apache Ignite 2.11 Code Freeze started now>
> > > > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Best wishes,
> > > > > > Amelchev Nikita
> > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best wishes,
> > > > Amelchev Nikita
> > > >
> > >
> 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Вячеслав Коптилин]

Hello Ivan,

> At least, we could just hide params that match a specific pattern
Yes, we can filter out all vm options that do not relate to Ignite, for
instance.

> Ilya, go ahead, file ticket and prepare a PR.
Please do not rush. Let's listen to other community members. This question
is about security and it should not be discussed in a hurry (even though it
looks like an obvious thing).

Thanks,
S.

чт, 1 июл. 2021 г. в 16:55, Ivan Daschinsky :

> I suppose, that all normal users should not suffer from this restrictions.
> Nobody will pass password using jvm options. It is absolutely insane,
> normal users pass passwords using environment variables.
>
> At least, we could just hide params that match specific pattern
>
> Ilya, go ahead, file ticket and prepare a PR.
>
> чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин :
>
> > Hello,
> >
> > Unfortunately, the user can pass its own system properties via JVM
> options
> > as follows: -DMY_SECRET_PASSWORD=123
> > It does not seem, this approach is the best one. However, the user should
> > have a "kostyl" in order to hide these properties and values in the log
> > file, IMHO.
> >
> > Thanks,
> > S.
> >
> > ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :
> >
> > > Hi Igniters,
> > >
> > > This feature [1, 2] prevents logging of the VM arguments when
> > > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now,
> > method
> > > IgniteKernal#ackVmArguments remains mostly the same [3].
> > >
> > > Is this behaviour actual now? Often, we should be able to get from logs
> > the
> > > actual VM options used at startup even if output of sensitive data is
> > > restricted.
> > >
> > > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > > 2.
> > >
> > >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > > 3.
> > >
> > >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

> No it isn't, I have carefully read code and IEP, in your code you write
> schema id in each tuple.

There is no code for batch operations yet.

Here is the description of TUPLE_GET_ALL:
- UUID: table ID
- int: schema ID
- arr of arr: array of rows with values for all columns in given schema
(nil when value is missing for a column)

As you can see, schema ID is written once for all rows.
A row is just a set of values according to the schema.


> Also, my biggest concern -- extra serde step. I suppose we should pass
> bytearray to internal api, and use msgpack throughout all wire protocols,
> as tarantool does.

I agree. But this was decided before in IEP-54, and is out of scope for
current IEP.
Would you like to start a separate thread to discuss this? Or I can do this
a bit later.

On Thu, Jul 1, 2021 at 4:41 PM Ivan Daschinsky  wrote:

> > This is described in all operations that include multiple tuples.
> No it isn't, I have carefully read code and IEP, in your code you write
> schema id in each tuple.
>
> Also, my biggest concern -- extra serde step. I suppose we should pass
> bytearray to internal api, and use msgpack throughout all wire protocols,
> as tarantool does.
>
> чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :
>
> > Ivan,
> >
> > >  that there is not neccesary to write schema versions in each row
> > > in collectionof tuples
> >
> > This is described in all operations that include multiple tuples.
> >
> >
> > > it is not clear from your code (probably
> > > mistake?) how differ key tuples and value tuples from each other
> >
> > Key tuples include only key columns. Key columns come first in the
> schema.
> > Value tuples include all columns, key and value. Added "Key tuples"
> > section.
> >
> >
> > > As for me, these excercises with schema's doesn't worth a lot
> >
> > I'll add a benchmark and we'll see.
> >
> >
> > On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
> > wrote:
> >
> > > I suppose, that there is not neccesary to write schema versions in each
> > row
> > > in collectionof tuples. Also it is not clear from your code (probably
> > > mistake?) how differ key tuples and value tuples from each other. In
> > > readTuple you always read full schema and check for full length. As for
> > me,
> > > these excercises with schema's doesn't worth a lot. I.e. postgres just
> > > writes field names and then simpy rows with data. Saving few bytes
> > doesn't
> > > make much deal. Btw, msgpack has special types for short strings (i.e.
> > > str8). It is much easier use it and write field name as is.
> > >
> > > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> > >
> > > > Ivan, tuple serialization section added to the IEP, let me know if it
> > is
> > > > clear enough.
> > > >
> > > > Thanks!
> > > >
> > > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > I can't find any description of tuple serialization in IEP, only in
> > > code
> > > > >
> > > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > > > >
> > > > > > Ivan,
> > > > > >
> > > > > > 0. The IEP is not in progress, it is ready for review and
> > discussion.
> > > > > > 1. Tuple serialization is described in the IEP and demonstrated
> in
> > > the
> > > > > PoC
> > > > > > (see ClientMessageHandler#readTuple), let me know if more details
> > are
> > > > > > required
> > > > > > 2. Tuple schema serialization is described in SCHEMAS_GET
> section.
> > > > Table
> > > > > > schema (configuration) needs more details, you are right - I'll
> add
> > > > them.
> > > > > > 3. This IEP is about tables (tuple-based) API only, since it is
> the
> > > > only
> > > > > > API that we have right now, as noted in Risks and Assumptions.
> > > > > >
> > > > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky <
> > ivanda...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Also, is there any clear information about KV api? Is there any
> > > plan
> > > > to
> > > > > > > implement it? Or is there any proposal about it?
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky  >:
> > > > > > >
> > > > > > > > Pavel, but IEP is in progress, isn't it?
> > > > > > > >
> > > > > > > > 1. There is not any information about tuple serialization.
> And
> > > > there
> > > > > > > isn't
> > > > > > > > a clear consensus about it.
> > > > > > > > 2. There is not any information about schrma serialization
> > > format.
> > > > > And
> > > > > > > > AFAIK, there isn't a clear consensus also.
> > > > > > > >
> > > > > > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn <
> ptupit...@apache.org
> > >:
> > > > > > > >
> > > > > > > >> Igniters,
> > > > > > > >>
> > > > > > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > > > > > >> PoC is in progress [2]
> > > > > > > >>
> > > > > > > >> [1]
> > > > > > > >>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > > > > > >>
> > > > > > > >> [2] 

Re: Re[2]: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Atri Sharma]

AFAIK, this allows users to pass in custom VM options which may be undesirable.

On Thu, Jul 1, 2021 at 12:07 PM Zhenya Stanilovsky
 wrote:
>
>
> +1 for reverting, can anybody (possibly ticket starter?) explain how jvm 
> settings will rise some security problems ?
>
>
>
> >I suppose, that we should revert this particular line. I don't understand
> >who ever considers vm options as sensitive info.
> >
> >ср, 30 июн. 2021 г., 22:52 Shishkov Ilya < shishkovi...@gmail.com >:
> >
> >> Hi Igniters,
> >>
> >> This feature [1, 2] prevents logging of the VM arguments when
> >> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method
> >> IgniteKernal#ackVmArguments remains mostly the same [3].
> >>
> >> Is this behaviour actual now? Often, we should be able to get from logs the
> >> actual VM options used at startup even if output of sensitive data is
> >> restricted.
> >>
> >> 1.  https://issues.apache.org/jira/browse/IGNITE-4991
> >> 2.
> >>
> >>  
> >> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> >> 3.
> >>
> >>  
> >> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> >>
>
>
>
>

-- 
Regards,

Atri
Apache Concerted

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

I suppose, that all normal users should not suffer from this restrictions.
Nobody will pass password using jvm options. It is absolutely insane,
normal users pass passwords using environment variables.

At least, we could just hide params that match specific pattern

Ilya, go ahead, file ticket and prepare a PR.

чт, 1 июл. 2021 г., 16:45 Вячеслав Коптилин :

> Hello,
>
> Unfortunately, the user can pass its own system properties via JVM options
> as follows: -DMY_SECRET_PASSWORD=123
> It does not seem, this approach is the best one. However, the user should
> have a "kostyl" in order to hide these properties and values in the log
> file, IMHO.
>
> Thanks,
> S.
>
> ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :
>
> > Hi Igniters,
> >
> > This feature [1, 2] prevents logging of the VM arguments when
> > IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now,
> method
> > IgniteKernal#ackVmArguments remains mostly the same [3].
> >
> > Is this behaviour actual now? Often, we should be able to get from logs
> the
> > actual VM options used at startup even if output of sensitive data is
> > restricted.
> >
> > 1. https://issues.apache.org/jira/browse/IGNITE-4991
> > 2.
> >
> >
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> > 3.
> >
> >
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
> >
>

Re: Apache Ignite 2.11

[Nikita Safonov]

Ivan, thank you for the links!

Well, I can still find the docs on Python Thin Client under the "Thin
Сlients"->"Python Thin Client" in 2.10:
https://ignite.apache.org/docs/2.10.0/thin-clients/python-thin-client

So, I can replace the contents of  "Thin Сlients"->"Python Thin Client"
page with a link to the
https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/ and
a short note as follows:
"To see the documentation on Python Thin Client, please click here[link to
the readthedocs.io
].
+
And I can also change the link here

(after
"Ignite provides the following thin clients:", so that it would lead to the
readthedocs.io

 too).

Alex, my bad, I misread the first question and thought that all the docs on
thin clients have been migrated to the readthedocs.io
.
The articles are absolutely OK :)

Regards,
Nikita

чт, 1 июл. 2021 г. в 16:44, Alex Plehanov :

> Nikita,
>
> What's wrong with platform-specific thin-client articles? These articles
> describe how to configure and use specific thin client implementation, why
> should we remove them?
>
> чт, 1 июл. 2021 г. в 16:06, Ivan Daschinsky :
>
>> Binary prorocol description should stand and improved (there are some
>> mistakes in this article)
>>
>> чт, 1 июл. 2021 г., 16:05 Ivan Daschinsky :
>>
>> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
>> >
>> > AFAIK, thin clients have been already removed since 2.10
>> >
>> > чт, 1 июл. 2021 г., 15:57 Nikita Safonov :
>> >
>> >> Hi guys!
>> >>
>> >> Sorry, totally missed this email.
>> >>
>> >> I can do this right away.
>> >>
>> >> Just a couple of questions first:
>> >>
>> >> [1] On the website, under the Thin Clients tab, we have the following
>> >> articles:
>> >>
>> >> Thin Clients
>> >> Thin Clients Overview
>> >> Java Thin Client
>> >> .NET Thin Client
>> >> C++ Thin Client
>> >> Python Thin Client
>> >> PHP Thin Client
>> >> Node.js Thin Client
>> >> Binary Client Protocol
>> >>
>> >> I do believe that the introduction ( "Thin Clients" - the first one)
>> >> should
>> >> remain, while we can remove the rest (by the way, what about Binary
>> Client
>> >> Protocol? Should it remain too?).
>> >> What do you think about this?
>> >>
>> >> [2] Could you please indicate the place in the docs where the link to
>> the
>> >> actual readthedocs.io documentation is? I didn't manage to find it.
>> >> If we don't have it, I would place it at the beginning of the Thin
>> >> clients'
>> >> introduction article.
>> >>
>> >> Regards,
>> >> Nikita
>> >>
>> >>
>> >> ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :
>> >>
>> >> > Hi, Nikita!
>> >> >
>> >> > Your help with documentation is really needed. Can you describe how
>> we
>> >> can
>> >> > start with it?
>> >> >
>> >> > On 2021/06/09 10:06:09, Nikita Safonov 
>> >> wrote:
>> >> > > Hello Igniters!
>> >> > >
>> >> > > I guess that AI 2.11 will include documentation work.
>> >> > > I'll be glad to help with this.
>> >> > > And I'll be happy to know the scope of features that we need to
>> >> document.
>> >> > >
>> >> > > Regards,
>> >> > > Nikita
>> >> > >
>> >> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
>> >> > >
>> >> > > > hello,
>> >> > > >
>> >> > > > can someone can help to merge the following patch into master?
>> >> > > >
>> >> > > > https://issues.apache.org/jira/browse/IGNITE-12852
>> >> > > >
>> >> > > > This patch has been completed for a long time, and I think it is
>> a
>> >> > > > valuable one.
>> >> > > >
>> >> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
>> >> > > > > Sure, thanks for your reply
>> >> > > > >
>> >> > > > > Folks, could you please advice how to setup JIRA account
>> >> integration
>> >> > for
>> >> > > > non-committers?
>> >> > > > >
>> >> > > > > For the page
>> >> > > > >
>> >> >
>> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
>> >> > > > > I can see all issues, but Alexey's login seems to be not
>> >> sufficient.
>> >> > Can
>> >> > > > wiki admins set up smth? Or Alexey should do some setup on his
>> own?
>> >> > > > >
>> >> > > > > On 2021/06/03 14:55:25, Maksim Timonin <
>> timonin.ma...@gmail.com>
>> >> > wrote:
>> >> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear
>> >> there
>> >> > are
>> >> > > > >> chances for 2.11, but I guess we can postpone it.
>> >> > > > >>
>> >> > > > >>
>> >> > > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov <
>> dpav...@apache.org
>> >> >
>> >> > > > wrote:
>> >> > > > >>
>> >> > > > >>> ok, Maksim, keep us posted.
>> >> > > > >>>
>> >> > > > >>> We're in the middle of paperwork rigth now, so there is a
>> >> chance ))
>> >> > > > >>>
>> >> > > > >>> On 2021/06/03 14:30:55, Maksim Timonin <
>> 

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Вячеслав Коптилин]

Hello,

Unfortunately, the user can pass its own system properties via JVM options
as follows: -DMY_SECRET_PASSWORD=123
It does not seem, this approach is the best one. However, the user should
have a "kostyl" in order to hide these properties and values in the log
file, IMHO.

Thanks,
S.

ср, 30 июн. 2021 г. в 22:52, Shishkov Ilya :

> Hi Igniters,
>
> This feature [1, 2] prevents logging of the VM arguments when
> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method
> IgniteKernal#ackVmArguments remains mostly the same [3].
>
> Is this behaviour actual now? Often, we should be able to get from logs the
> actual VM options used at startup even if output of sensitive data is
> restricted.
>
> 1. https://issues.apache.org/jira/browse/IGNITE-4991
> 2.
>
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> 3.
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
>

Re: Apache Ignite 2.11

[Alex Plehanov]

Nikita,

What's wrong with platform-specific thin-client articles? These articles
describe how to configure and use specific thin client implementation, why
should we remove them?

чт, 1 июл. 2021 г. в 16:06, Ivan Daschinsky :

> Binary prorocol description should stand and improved (there are some
> mistakes in this article)
>
> чт, 1 июл. 2021 г., 16:05 Ivan Daschinsky :
>
> > https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
> >
> > AFAIK, thin clients have been already removed since 2.10
> >
> > чт, 1 июл. 2021 г., 15:57 Nikita Safonov :
> >
> >> Hi guys!
> >>
> >> Sorry, totally missed this email.
> >>
> >> I can do this right away.
> >>
> >> Just a couple of questions first:
> >>
> >> [1] On the website, under the Thin Clients tab, we have the following
> >> articles:
> >>
> >> Thin Clients
> >> Thin Clients Overview
> >> Java Thin Client
> >> .NET Thin Client
> >> C++ Thin Client
> >> Python Thin Client
> >> PHP Thin Client
> >> Node.js Thin Client
> >> Binary Client Protocol
> >>
> >> I do believe that the introduction ( "Thin Clients" - the first one)
> >> should
> >> remain, while we can remove the rest (by the way, what about Binary
> Client
> >> Protocol? Should it remain too?).
> >> What do you think about this?
> >>
> >> [2] Could you please indicate the place in the docs where the link to
> the
> >> actual readthedocs.io documentation is? I didn't manage to find it.
> >> If we don't have it, I would place it at the beginning of the Thin
> >> clients'
> >> introduction article.
> >>
> >> Regards,
> >> Nikita
> >>
> >>
> >> ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :
> >>
> >> > Hi, Nikita!
> >> >
> >> > Your help with documentation is really needed. Can you describe how we
> >> can
> >> > start with it?
> >> >
> >> > On 2021/06/09 10:06:09, Nikita Safonov 
> >> wrote:
> >> > > Hello Igniters!
> >> > >
> >> > > I guess that AI 2.11 will include documentation work.
> >> > > I'll be glad to help with this.
> >> > > And I'll be happy to know the scope of features that we need to
> >> document.
> >> > >
> >> > > Regards,
> >> > > Nikita
> >> > >
> >> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> >> > >
> >> > > > hello,
> >> > > >
> >> > > > can someone can help to merge the following patch into master?
> >> > > >
> >> > > > https://issues.apache.org/jira/browse/IGNITE-12852
> >> > > >
> >> > > > This patch has been completed for a long time, and I think it is a
> >> > > > valuable one.
> >> > > >
> >> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> >> > > > > Sure, thanks for your reply
> >> > > > >
> >> > > > > Folks, could you please advice how to setup JIRA account
> >> integration
> >> > for
> >> > > > non-committers?
> >> > > > >
> >> > > > > For the page
> >> > > > >
> >> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> >> > > > > I can see all issues, but Alexey's login seems to be not
> >> sufficient.
> >> > Can
> >> > > > wiki admins set up smth? Or Alexey should do some setup on his
> own?
> >> > > > >
> >> > > > > On 2021/06/03 14:55:25, Maksim Timonin  >
> >> > wrote:
> >> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear
> >> there
> >> > are
> >> > > > >> chances for 2.11, but I guess we can postpone it.
> >> > > > >>
> >> > > > >>
> >> > > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov <
> dpav...@apache.org
> >> >
> >> > > > wrote:
> >> > > > >>
> >> > > > >>> ok, Maksim, keep us posted.
> >> > > > >>>
> >> > > > >>> We're in the middle of paperwork rigth now, so there is a
> >> chance ))
> >> > > > >>>
> >> > > > >>> On 2021/06/03 14:30:55, Maksim Timonin <
> timonin.ma...@gmail.com
> >> >
> >> > > > wrote:
> >> > > > > Regarding non-SQL index API it really depends on readiness.
> I
> >> > guess
> >> > > > if
> >> > > > >>> it
> >> > > >  is not in the master, we can skip and release it next time.
> >> > > > 
> >> > > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release
> >> it
> >> > > > within
> >> > > > >>> the
> >> > > >  next release.
> >> > > > 
> >> > > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov <
> >> dpav...@apache.org>
> >> > > > wrote:
> >> > > > 
> >> > > > > Hi Alexey,
> >> > > > >
> >> > > > > Releasing master as-is makes sence for me.
> >> > > > >
> >> > > > > Regarding non-SQL index API it really depends on readiness.
> I
> >> > guess
> >> > > > if
> >> > > > >>> it
> >> > > > > is not in the master, we can skip and release it next time.
> >> > > > >
> >> > > > > We've a bit overdue initial scope freeze date. It's not a
> big
> >> > deal,
> >> > > > but
> >> > > > > still makes sense to start discussion, create and share wiki
> >> > page and
> >> > > > > create branch.
> >> > > > >
> >> > > > > Since committer privileges required to create branch, could
> >> you
> >> > > > please
> >> > > > > notify me in advance.
> >> > > > >
> >> > > > > Sincerely,
> >> > > > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

> This is described in all operations that include multiple tuples.
No it isn't, I have carefully read code and IEP, in your code you write
schema id in each tuple.

Also, my biggest concern -- extra serde step. I suppose we should pass
bytearray to internal api, and use msgpack throughout all wire protocols,
as tarantool does.

чт, 1 июл. 2021 г., 16:15 Pavel Tupitsyn :

> Ivan,
>
> >  that there is not neccesary to write schema versions in each row
> > in collectionof tuples
>
> This is described in all operations that include multiple tuples.
>
>
> > it is not clear from your code (probably
> > mistake?) how differ key tuples and value tuples from each other
>
> Key tuples include only key columns. Key columns come first in the schema.
> Value tuples include all columns, key and value. Added "Key tuples"
> section.
>
>
> > As for me, these excercises with schema's doesn't worth a lot
>
> I'll add a benchmark and we'll see.
>
>
> On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky 
> wrote:
>
> > I suppose, that there is not neccesary to write schema versions in each
> row
> > in collectionof tuples. Also it is not clear from your code (probably
> > mistake?) how differ key tuples and value tuples from each other. In
> > readTuple you always read full schema and check for full length. As for
> me,
> > these excercises with schema's doesn't worth a lot. I.e. postgres just
> > writes field names and then simpy rows with data. Saving few bytes
> doesn't
> > make much deal. Btw, msgpack has special types for short strings (i.e.
> > str8). It is much easier use it and write field name as is.
> >
> > чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
> >
> > > Ivan, tuple serialization section added to the IEP, let me know if it
> is
> > > clear enough.
> > >
> > > Thanks!
> > >
> > > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > I can't find any description of tuple serialization in IEP, only in
> > code
> > > >
> > > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > > >
> > > > > Ivan,
> > > > >
> > > > > 0. The IEP is not in progress, it is ready for review and
> discussion.
> > > > > 1. Tuple serialization is described in the IEP and demonstrated in
> > the
> > > > PoC
> > > > > (see ClientMessageHandler#readTuple), let me know if more details
> are
> > > > > required
> > > > > 2. Tuple schema serialization is described in SCHEMAS_GET section.
> > > Table
> > > > > schema (configuration) needs more details, you are right - I'll add
> > > them.
> > > > > 3. This IEP is about tables (tuple-based) API only, since it is the
> > > only
> > > > > API that we have right now, as noted in Risks and Assumptions.
> > > > >
> > > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky <
> ivanda...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Also, is there any clear information about KV api? Is there any
> > plan
> > > to
> > > > > > implement it? Or is there any proposal about it?
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > > > > >
> > > > > > > Pavel, but IEP is in progress, isn't it?
> > > > > > >
> > > > > > > 1. There is not any information about tuple serialization. And
> > > there
> > > > > > isn't
> > > > > > > a clear consensus about it.
> > > > > > > 2. There is not any information about schrma serialization
> > format.
> > > > And
> > > > > > > AFAIK, there isn't a clear consensus also.
> > > > > > >
> > > > > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn  >:
> > > > > > >
> > > > > > >> Igniters,
> > > > > > >>
> > > > > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > > > > >> PoC is in progress [2]
> > > > > > >>
> > > > > > >> [1]
> > > > > > >>
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > > > > >>
> > > > > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan,

>  that there is not neccesary to write schema versions in each row
> in collectionof tuples

This is described in all operations that include multiple tuples.


> it is not clear from your code (probably
> mistake?) how differ key tuples and value tuples from each other

Key tuples include only key columns. Key columns come first in the schema.
Value tuples include all columns, key and value. Added "Key tuples" section.


> As for me, these excercises with schema's doesn't worth a lot

I'll add a benchmark and we'll see.


On Thu, Jul 1, 2021 at 3:17 PM Ivan Daschinsky  wrote:

> I suppose, that there is not neccesary to write schema versions in each row
> in collectionof tuples. Also it is not clear from your code (probably
> mistake?) how differ key tuples and value tuples from each other. In
> readTuple you always read full schema and check for full length. As for me,
> these excercises with schema's doesn't worth a lot. I.e. postgres just
> writes field names and then simpy rows with data. Saving few bytes doesn't
> make much deal. Btw, msgpack has special types for short strings (i.e.
> str8). It is much easier use it and write field name as is.
>
> чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :
>
> > Ivan, tuple serialization section added to the IEP, let me know if it is
> > clear enough.
> >
> > Thanks!
> >
> > On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> > wrote:
> >
> > > I can't find any description of tuple serialization in IEP, only in
> code
> > >
> > > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> > >
> > > > Ivan,
> > > >
> > > > 0. The IEP is not in progress, it is ready for review and discussion.
> > > > 1. Tuple serialization is described in the IEP and demonstrated in
> the
> > > PoC
> > > > (see ClientMessageHandler#readTuple), let me know if more details are
> > > > required
> > > > 2. Tuple schema serialization is described in SCHEMAS_GET section.
> > Table
> > > > schema (configuration) needs more details, you are right - I'll add
> > them.
> > > > 3. This IEP is about tables (tuple-based) API only, since it is the
> > only
> > > > API that we have right now, as noted in Risks and Assumptions.
> > > >
> > > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> > > > wrote:
> > > >
> > > > > Also, is there any clear information about KV api? Is there any
> plan
> > to
> > > > > implement it? Or is there any proposal about it?
> > > > >
> > > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > > > >
> > > > > > Pavel, but IEP is in progress, isn't it?
> > > > > >
> > > > > > 1. There is not any information about tuple serialization. And
> > there
> > > > > isn't
> > > > > > a clear consensus about it.
> > > > > > 2. There is not any information about schrma serialization
> format.
> > > And
> > > > > > AFAIK, there isn't a clear consensus also.
> > > > > >
> > > > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > > > >
> > > > > >> Igniters,
> > > > > >>
> > > > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > > > >> PoC is in progress [2]
> > > > > >>
> > > > > >> [1]
> > > > > >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > > > >>
> > > > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Apache Ignite 2.11

[Ivan Daschinsky]

Binary prorocol description should stand and improved (there are some
mistakes in this article)

чт, 1 июл. 2021 г., 16:05 Ivan Daschinsky :

> https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/
>
> AFAIK, thin clients have been already removed since 2.10
>
> чт, 1 июл. 2021 г., 15:57 Nikita Safonov :
>
>> Hi guys!
>>
>> Sorry, totally missed this email.
>>
>> I can do this right away.
>>
>> Just a couple of questions first:
>>
>> [1] On the website, under the Thin Clients tab, we have the following
>> articles:
>>
>> Thin Clients
>> Thin Clients Overview
>> Java Thin Client
>> .NET Thin Client
>> C++ Thin Client
>> Python Thin Client
>> PHP Thin Client
>> Node.js Thin Client
>> Binary Client Protocol
>>
>> I do believe that the introduction ( "Thin Clients" - the first one)
>> should
>> remain, while we can remove the rest (by the way, what about Binary Client
>> Protocol? Should it remain too?).
>> What do you think about this?
>>
>> [2] Could you please indicate the place in the docs where the link to the
>> actual readthedocs.io documentation is? I didn't manage to find it.
>> If we don't have it, I would place it at the beginning of the Thin
>> clients'
>> introduction article.
>>
>> Regards,
>> Nikita
>>
>>
>> ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :
>>
>> > Hi, Nikita!
>> >
>> > Your help with documentation is really needed. Can you describe how we
>> can
>> > start with it?
>> >
>> > On 2021/06/09 10:06:09, Nikita Safonov 
>> wrote:
>> > > Hello Igniters!
>> > >
>> > > I guess that AI 2.11 will include documentation work.
>> > > I'll be glad to help with this.
>> > > And I'll be happy to know the scope of features that we need to
>> document.
>> > >
>> > > Regards,
>> > > Nikita
>> > >
>> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
>> > >
>> > > > hello,
>> > > >
>> > > > can someone can help to merge the following patch into master?
>> > > >
>> > > > https://issues.apache.org/jira/browse/IGNITE-12852
>> > > >
>> > > > This patch has been completed for a long time, and I think it is a
>> > > > valuable one.
>> > > >
>> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
>> > > > > Sure, thanks for your reply
>> > > > >
>> > > > > Folks, could you please advice how to setup JIRA account
>> integration
>> > for
>> > > > non-committers?
>> > > > >
>> > > > > For the page
>> > > > >
>> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
>> > > > > I can see all issues, but Alexey's login seems to be not
>> sufficient.
>> > Can
>> > > > wiki admins set up smth? Or Alexey should do some setup on his own?
>> > > > >
>> > > > > On 2021/06/03 14:55:25, Maksim Timonin 
>> > wrote:
>> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear
>> there
>> > are
>> > > > >> chances for 2.11, but I guess we can postpone it.
>> > > > >>
>> > > > >>
>> > > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov > >
>> > > > wrote:
>> > > > >>
>> > > > >>> ok, Maksim, keep us posted.
>> > > > >>>
>> > > > >>> We're in the middle of paperwork rigth now, so there is a
>> chance ))
>> > > > >>>
>> > > > >>> On 2021/06/03 14:30:55, Maksim Timonin > >
>> > > > wrote:
>> > > > > Regarding non-SQL index API it really depends on readiness. I
>> > guess
>> > > > if
>> > > > >>> it
>> > > >  is not in the master, we can skip and release it next time.
>> > > > 
>> > > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release
>> it
>> > > > within
>> > > > >>> the
>> > > >  next release.
>> > > > 
>> > > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov <
>> dpav...@apache.org>
>> > > > wrote:
>> > > > 
>> > > > > Hi Alexey,
>> > > > >
>> > > > > Releasing master as-is makes sence for me.
>> > > > >
>> > > > > Regarding non-SQL index API it really depends on readiness. I
>> > guess
>> > > > if
>> > > > >>> it
>> > > > > is not in the master, we can skip and release it next time.
>> > > > >
>> > > > > We've a bit overdue initial scope freeze date. It's not a big
>> > deal,
>> > > > but
>> > > > > still makes sense to start discussion, create and share wiki
>> > page and
>> > > > > create branch.
>> > > > >
>> > > > > Since committer privileges required to create branch, could
>> you
>> > > > please
>> > > > > notify me in advance.
>> > > > >
>> > > > > Sincerely,
>> > > > > Dmitriy Pavlov
>> > > > >
>> > > > > On 2021/06/03 14:08:34, ��  <
>> > > > > olive.c...@gmail.com> wrote:
>> > > > >> :scope:
>> > > > >>
>> > > > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite
>> > 2.11
>> > > > > release
>> > > > >> Let's start discussion
>> > > > >>
>> > > > >> [1]
>> > > > >>>
>> > > >
>> >
>> 

Re: Apache Ignite 2.11

[Ivan Daschinsky]

https://apache-ignite-binary-protocol-client.readthedocs.io/en/stable/

AFAIK, thin clients have been already removed since 2.10

чт, 1 июл. 2021 г., 15:57 Nikita Safonov :

> Hi guys!
>
> Sorry, totally missed this email.
>
> I can do this right away.
>
> Just a couple of questions first:
>
> [1] On the website, under the Thin Clients tab, we have the following
> articles:
>
> Thin Clients
> Thin Clients Overview
> Java Thin Client
> .NET Thin Client
> C++ Thin Client
> Python Thin Client
> PHP Thin Client
> Node.js Thin Client
> Binary Client Protocol
>
> I do believe that the introduction ( "Thin Clients" - the first one) should
> remain, while we can remove the rest (by the way, what about Binary Client
> Protocol? Should it remain too?).
> What do you think about this?
>
> [2] Could you please indicate the place in the docs where the link to the
> actual readthedocs.io documentation is? I didn't manage to find it.
> If we don't have it, I would place it at the beginning of the Thin clients'
> introduction article.
>
> Regards,
> Nikita
>
>
> ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :
>
> > Hi, Nikita!
> >
> > Your help with documentation is really needed. Can you describe how we
> can
> > start with it?
> >
> > On 2021/06/09 10:06:09, Nikita Safonov 
> wrote:
> > > Hello Igniters!
> > >
> > > I guess that AI 2.11 will include documentation work.
> > > I'll be glad to help with this.
> > > And I'll be happy to know the scope of features that we need to
> document.
> > >
> > > Regards,
> > > Nikita
> > >
> > > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> > >
> > > > hello,
> > > >
> > > > can someone can help to merge the following patch into master?
> > > >
> > > > https://issues.apache.org/jira/browse/IGNITE-12852
> > > >
> > > > This patch has been completed for a long time, and I think it is a
> > > > valuable one.
> > > >
> > > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> > > > > Sure, thanks for your reply
> > > > >
> > > > > Folks, could you please advice how to setup JIRA account
> integration
> > for
> > > > non-committers?
> > > > >
> > > > > For the page
> > > > >
> > https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> > > > > I can see all issues, but Alexey's login seems to be not
> sufficient.
> > Can
> > > > wiki admins set up smth? Or Alexey should do some setup on his own?
> > > > >
> > > > > On 2021/06/03 14:55:25, Maksim Timonin 
> > wrote:
> > > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear there
> > are
> > > > >> chances for 2.11, but I guess we can postpone it.
> > > > >>
> > > > >>
> > > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov 
> > > > wrote:
> > > > >>
> > > > >>> ok, Maksim, keep us posted.
> > > > >>>
> > > > >>> We're in the middle of paperwork rigth now, so there is a chance
> ))
> > > > >>>
> > > > >>> On 2021/06/03 14:30:55, Maksim Timonin 
> > > > wrote:
> > > > > Regarding non-SQL index API it really depends on readiness. I
> > guess
> > > > if
> > > > >>> it
> > > >  is not in the master, we can skip and release it next time.
> > > > 
> > > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release it
> > > > within
> > > > >>> the
> > > >  next release.
> > > > 
> > > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov <
> dpav...@apache.org>
> > > > wrote:
> > > > 
> > > > > Hi Alexey,
> > > > >
> > > > > Releasing master as-is makes sence for me.
> > > > >
> > > > > Regarding non-SQL index API it really depends on readiness. I
> > guess
> > > > if
> > > > >>> it
> > > > > is not in the master, we can skip and release it next time.
> > > > >
> > > > > We've a bit overdue initial scope freeze date. It's not a big
> > deal,
> > > > but
> > > > > still makes sense to start discussion, create and share wiki
> > page and
> > > > > create branch.
> > > > >
> > > > > Since committer privileges required to create branch, could you
> > > > please
> > > > > notify me in advance.
> > > > >
> > > > > Sincerely,
> > > > > Dmitriy Pavlov
> > > > >
> > > > > On 2021/06/03 14:08:34, ��  <
> > > > > olive.c...@gmail.com> wrote:
> > > > >> :scope:
> > > > >>
> > > > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite
> > 2.11
> > > > > release
> > > > >> Let's start discussion
> > > > >>
> > > > >> [1]
> > > > >>>
> > > >
> >
> https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > >> On 2021/05/13 14:27:33, Dmitry Pavlov 
> > wrote:
> > > > >>> Hi Alexey,
> > > > >>>
> > > > >>> :pmc to sign distribution:
> > > > >>> Formally, only PMC member can be a release manager, so part
> of
> > > > > activities should be picked up by committer and PMC member. If
> > noone
> > 

Re: Apache Ignite 2.11

[Nikita Safonov]

Hi guys!

Sorry, totally missed this email.

I can do this right away.

Just a couple of questions first:

[1] On the website, under the Thin Clients tab, we have the following
articles:

Thin Clients
Thin Clients Overview
Java Thin Client
.NET Thin Client
C++ Thin Client
Python Thin Client
PHP Thin Client
Node.js Thin Client
Binary Client Protocol

I do believe that the introduction ( "Thin Clients" - the first one) should
remain, while we can remove the rest (by the way, what about Binary Client
Protocol? Should it remain too?).
What do you think about this?

[2] Could you please indicate the place in the docs where the link to the
actual readthedocs.io documentation is? I didn't manage to find it.
If we don't have it, I would place it at the beginning of the Thin clients'
introduction article.

Regards,
Nikita


ср, 30 июн. 2021 г. в 09:36, Alexey Gidaspov :

> Hi, Nikita!
>
> Your help with documentation is really needed. Can you describe how we can
> start with it?
>
> On 2021/06/09 10:06:09, Nikita Safonov  wrote:
> > Hello Igniters!
> >
> > I guess that AI 2.11 will include documentation work.
> > I'll be glad to help with this.
> > And I'll be happy to know the scope of features that we need to document.
> >
> > Regards,
> > Nikita
> >
> > пт, 4 июн. 2021 г. в 03:50, 18624049226 <18624049...@163.com>:
> >
> > > hello,
> > >
> > > can someone can help to merge the following patch into master?
> > >
> > > https://issues.apache.org/jira/browse/IGNITE-12852
> > >
> > > This patch has been completed for a long time, and I think it is a
> > > valuable one.
> > >
> > > 在 2021/6/3 下午11:12, Dmitry Pavlov 写道:
> > > > Sure, thanks for your reply
> > > >
> > > > Folks, could you please advice how to setup JIRA account integration
> for
> > > non-committers?
> > > >
> > > > For the page
> > > >
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.11
> > > > I can see all issues, but Alexey's login seems to be not sufficient.
> Can
> > > wiki admins set up smth? Or Alexey should do some setup on his own?
> > > >
> > > > On 2021/06/03 14:55:25, Maksim Timonin 
> wrote:
> > > >> I mean I'm OK to release it with 2.12. So, I am glad to hear there
> are
> > > >> chances for 2.11, but I guess we can postpone it.
> > > >>
> > > >>
> > > >> On Thu, Jun 3, 2021 at 5:39 PM Dmitry Pavlov 
> > > wrote:
> > > >>
> > > >>> ok, Maksim, keep us posted.
> > > >>>
> > > >>> We're in the middle of paperwork rigth now, so there is a chance ))
> > > >>>
> > > >>> On 2021/06/03 14:30:55, Maksim Timonin 
> > > wrote:
> > > > Regarding non-SQL index API it really depends on readiness. I
> guess
> > > if
> > > >>> it
> > >  is not in the master, we can skip and release it next time.
> > > 
> > >  Hi, Dmitry! Yes, currently it is on review. I'm OK to release it
> > > within
> > > >>> the
> > >  next release.
> > > 
> > >  On Thu, Jun 3, 2021 at 5:26 PM Dmitry Pavlov 
> > > wrote:
> > > 
> > > > Hi Alexey,
> > > >
> > > > Releasing master as-is makes sence for me.
> > > >
> > > > Regarding non-SQL index API it really depends on readiness. I
> guess
> > > if
> > > >>> it
> > > > is not in the master, we can skip and release it next time.
> > > >
> > > > We've a bit overdue initial scope freeze date. It's not a big
> deal,
> > > but
> > > > still makes sense to start discussion, create and share wiki
> page and
> > > > create branch.
> > > >
> > > > Since committer privileges required to create branch, could you
> > > please
> > > > notify me in advance.
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > On 2021/06/03 14:08:34, ��  <
> > > > olive.c...@gmail.com> wrote:
> > > >> :scope:
> > > >>
> > > >> Here is a list [1] of 195 tickets, i picked for Apache Ignite
> 2.11
> > > > release
> > > >> Let's start discussion
> > > >>
> > > >> [1]
> > > >>>
> > >
> https://issues.apache.org/jira/browse/IGNITE-14781?jql=project%20%3D%20IGNITE%20%20AND%20status%20%20%3D%20Resolved%20and%20fixVersion%20%3D%202.11%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > >> On 2021/05/13 14:27:33, Dmitry Pavlov 
> wrote:
> > > >>> Hi Alexey,
> > > >>>
> > > >>> :pmc to sign distribution:
> > > >>> Formally, only PMC member can be a release manager, so part of
> > > > activities should be picked up by committer and PMC member. If
> noone
> > > >>> else
> > > > want to help you here, I would.
> > > >>> :timeline:
> > > >>> And, could you estimate all phases of the release.
> > > >>>
> > > >>> :scope:
> > > >>> I guess we can just pick up master current state and release
> it.
> > > >>> But
> > > > if someone has some ideas if we should wait for particular
> feature to
> > > >>> be
> > > > completed before branch divergence/release candidate build,
> please
> > > let
> > > >>> know.
> > > >>> Sincerely,

Re: IEP-61 Technical discussion

[Alexei Scherbakov]

Hi.

We have made some progress on the topic.

The JRaft fork is merged to Ignite 3 master, now it's integrated with other
ready components.

The design of transactional protocol in the first iteration is published on
the master [1]

[1] https://github.com/apache/ignite-3/tree/main/modules/transactions


сб, 20 мар. 2021 г. в 21:00, Alexei Scherbakov :

> Folks,
>
> I want to share some information about progress in implementing the raft
> protocol in ignite 3, which is a prerequisite for metastorage.
>
> The implementation will consist of client and server modules. The client
> is responsible for interoperability between raft server node and any other
> remote/local java process
>
> I have recently finished a raft client API. The public API part is
> available here [1] for review. The entry point is RaftGroupService
> interface. The service implementation has not been finished yet and can be
> skipped for now.
>
> As for the server part, currently we are investigating two options. First
> is etcd [2] implementation ported to Java. The drawback here is the amount
> of work required to make it working. Second option is the adoption of
> jraft [3] implementation. It is a full featured implementation already
> written in Java, but the code is not quite clean in my opinion and will
> require some refactoring.
>
> The next step is to make a raft client working with server
> implementations. At least one is required for the next alpha. It is planned
> to have the same client for both server implementations. As soon as both
> will be ready, we will compare them by running consistency tests and
> benchmarks and drop the worst. I will give the next update when we will
> have a working client and at least one server implementation ready.
>
> [1] https://github.com/apache/ignite-3/pull/59/files
> [2] https://github.com/etcd-io/etcd/tree/master/raft
> [3] https://github.com/sofastack/sofa-jraft
>
> пт, 27 нояб. 2020 г. в 20:26, Alexey Goncharuk  >:
>
>> Folks, thanks to everyone who joined the call. Summary:
>>
>>- We agree that it may be beneficial to separate metastorage and group
>>membership services, however, the abstractions should be clean enough
>> so
>>that we could implement group membership via metastorage
>>- Production cluster setup will involve an administrator 'init' command
>>that will initialize the metastorage raft group. Once the metastorage
>> is
>>initialized, all nodes may be restarted arbitrarily
>>- HA cluster must contain at least 3 nodes. 2-node cluster will stop
>>progress when one of the nodes fails (due to metastorage requirements)
>>- We will provide a 'developer' cluster mode which will allow a 1-node
>>setup and auto-initialization without the 'init' command
>>- We are targeting centralized affinity calculation that will be stored
>>to the metastorage. Metastorage downtime does not necessarily mean
>> cluster
>>availability (subject to the partition replication protocol choice). It
>>would be good to maximally hide the partition object so that we could
>>support range partitioning in the future
>>
>> To discuss at the next meeting (do not hesitate to send questions here
>> before the meeting):
>>
>>- Raft implementation details (API model, porting, etc)
>>- Transactions interaction with replication protocol
>>- Weaker consistency options
>>
>> Please add more if I forgot something and let's choose a time for the next
>> meeting.
>>
>> --AG
>>
>> чт, 26 нояб. 2020 г. в 16:12, Kseniya Romanova > >:
>>
>> > Done
>> >
>> > чт, 26 нояб. 2020 г. в 13:18, Ivan Daschinsky :
>> >
>> > > Alexey, is it possible to manage call at 16:00 MSK?
>> > >
>> > > чт, 26 нояб. 2020 г. в 12:30, Alexey Goncharuk <
>> > alexey.goncha...@gmail.com
>> > > >:
>> > >
>> > > > Hi Ivan,
>> > > >
>> > > > Unfortunately, the earliest window available for us is 12:00 MSK (1
>> > hour
>> > > > slot), or after 14:30 MSK. Let me know what time works best for you.
>> > > >
>> > > > ср, 25 нояб. 2020 г. в 21:38, Ivan Daschinsky > >:
>> > > >
>> > > > > Alexey, I kindly ask you to move the meeting a little bit earlier,
>> > > ideal
>> > > > > variant -- in the morning.
>> > > > >
>> > > > > ср, 25 нояб. 2020 г. в 20:10, Alexey Goncharuk <
>> > > > alexey.goncha...@gmail.com
>> > > > > >:
>> > > > >
>> > > > > > Folks, let's have the call on Friday, Nov 27th at 18:00 MSK? We
>> can
>> > > use
>> > > > > the
>> > > > > > following waiting room link:
>> > > > > >
>> > https://zoom.us/j/99450012496?pwd=RWZmOGhCNWlRK0ZpamdOOTZsYTJ0dz09
>> > > > > >
>> > > > > > Let me know if this time works for everybody.
>> > > > > >
>> > > > > > ср, 25 нояб. 2020 г. в 16:42, Alexey Goncharuk <
>> > > > > alexey.goncha...@gmail.com
>> > > > > > >:
>> > > > > >
>> > > > > > > Folks,
>> > > > > > >
>> > > > > > > I've made some edits in IEP-61 [1] regarding the group
>> membership
>> > > > > service
>> > > > > > > and transaction protocol interaction with the replication
>> > 

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

I suppose, that there is not neccesary to write schema versions in each row
in collectionof tuples. Also it is not clear from your code (probably
mistake?) how differ key tuples and value tuples from each other. In
readTuple you always read full schema and check for full length. As for me,
these excercises with schema's doesn't worth a lot. I.e. postgres just
writes field names and then simpy rows with data. Saving few bytes doesn't
make much deal. Btw, msgpack has special types for short strings (i.e.
str8). It is much easier use it and write field name as is.

чт, 1 июл. 2021 г., 14:56 Pavel Tupitsyn :

> Ivan, tuple serialization section added to the IEP, let me know if it is
> clear enough.
>
> Thanks!
>
> On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky 
> wrote:
>
> > I can't find any description of tuple serialization in IEP, only in code
> >
> > чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
> >
> > > Ivan,
> > >
> > > 0. The IEP is not in progress, it is ready for review and discussion.
> > > 1. Tuple serialization is described in the IEP and demonstrated in the
> > PoC
> > > (see ClientMessageHandler#readTuple), let me know if more details are
> > > required
> > > 2. Tuple schema serialization is described in SCHEMAS_GET section.
> Table
> > > schema (configuration) needs more details, you are right - I'll add
> them.
> > > 3. This IEP is about tables (tuple-based) API only, since it is the
> only
> > > API that we have right now, as noted in Risks and Assumptions.
> > >
> > > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> > > wrote:
> > >
> > > > Also, is there any clear information about KV api? Is there any plan
> to
> > > > implement it? Or is there any proposal about it?
> > > >
> > > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > > >
> > > > > Pavel, but IEP is in progress, isn't it?
> > > > >
> > > > > 1. There is not any information about tuple serialization. And
> there
> > > > isn't
> > > > > a clear consensus about it.
> > > > > 2. There is not any information about schrma serialization format.
> > And
> > > > > AFAIK, there isn't a clear consensus also.
> > > > >
> > > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > > >
> > > > >> Igniters,
> > > > >>
> > > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > > >> PoC is in progress [2]
> > > > >>
> > > > >> [1]
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > > >>
> > > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan, tuple serialization section added to the IEP, let me know if it is
clear enough.

Thanks!

On Thu, Jul 1, 2021 at 2:06 PM Ivan Daschinsky  wrote:

> I can't find any description of tuple serialization in IEP, only in code
>
> чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :
>
> > Ivan,
> >
> > 0. The IEP is not in progress, it is ready for review and discussion.
> > 1. Tuple serialization is described in the IEP and demonstrated in the
> PoC
> > (see ClientMessageHandler#readTuple), let me know if more details are
> > required
> > 2. Tuple schema serialization is described in SCHEMAS_GET section. Table
> > schema (configuration) needs more details, you are right - I'll add them.
> > 3. This IEP is about tables (tuple-based) API only, since it is the only
> > API that we have right now, as noted in Risks and Assumptions.
> >
> > On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> > wrote:
> >
> > > Also, is there any clear information about KV api? Is there any plan to
> > > implement it? Or is there any proposal about it?
> > >
> > > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> > >
> > > > Pavel, but IEP is in progress, isn't it?
> > > >
> > > > 1. There is not any information about tuple serialization. And there
> > > isn't
> > > > a clear consensus about it.
> > > > 2. There is not any information about schrma serialization format.
> And
> > > > AFAIK, there isn't a clear consensus also.
> > > >
> > > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > > >
> > > >> Igniters,
> > > >>
> > > >> Please review the IEP for thin client protocol in 3.0 [1].
> > > >> PoC is in progress [2]
> > > >>
> > > >> [1]
> > > >>
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > > >>
> > > >> [2] https://github.com/apache/ignite-3/pull/191
> > > >>
> > > >
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

I can't find any description of tuple serialization in IEP, only in code

чт, 1 июл. 2021 г., 13:59 Pavel Tupitsyn :

> Ivan,
>
> 0. The IEP is not in progress, it is ready for review and discussion.
> 1. Tuple serialization is described in the IEP and demonstrated in the PoC
> (see ClientMessageHandler#readTuple), let me know if more details are
> required
> 2. Tuple schema serialization is described in SCHEMAS_GET section. Table
> schema (configuration) needs more details, you are right - I'll add them.
> 3. This IEP is about tables (tuple-based) API only, since it is the only
> API that we have right now, as noted in Risks and Assumptions.
>
> On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky 
> wrote:
>
> > Also, is there any clear information about KV api? Is there any plan to
> > implement it? Or is there any proposal about it?
> >
> > чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
> >
> > > Pavel, but IEP is in progress, isn't it?
> > >
> > > 1. There is not any information about tuple serialization. And there
> > isn't
> > > a clear consensus about it.
> > > 2. There is not any information about schrma serialization format. And
> > > AFAIK, there isn't a clear consensus also.
> > >
> > > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> > >
> > >> Igniters,
> > >>
> > >> Please review the IEP for thin client protocol in 3.0 [1].
> > >> PoC is in progress [2]
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> > >>
> > >> [2] https://github.com/apache/ignite-3/pull/191
> > >>
> > >
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Ivan,

0. The IEP is not in progress, it is ready for review and discussion.
1. Tuple serialization is described in the IEP and demonstrated in the PoC
(see ClientMessageHandler#readTuple), let me know if more details are
required
2. Tuple schema serialization is described in SCHEMAS_GET section. Table
schema (configuration) needs more details, you are right - I'll add them.
3. This IEP is about tables (tuple-based) API only, since it is the only
API that we have right now, as noted in Risks and Assumptions.

On Thu, Jul 1, 2021 at 1:53 PM Ivan Daschinsky  wrote:

> Also, is there any clear information about KV api? Is there any plan to
> implement it? Or is there any proposal about it?
>
> чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :
>
> > Pavel, but IEP is in progress, isn't it?
> >
> > 1. There is not any information about tuple serialization. And there
> isn't
> > a clear consensus about it.
> > 2. There is not any information about schrma serialization format. And
> > AFAIK, there isn't a clear consensus also.
> >
> > чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
> >
> >> Igniters,
> >>
> >> Please review the IEP for thin client protocol in 3.0 [1].
> >> PoC is in progress [2]
> >>
> >> [1]
> >>
> >>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
> >>
> >> [2] https://github.com/apache/ignite-3/pull/191
> >>
> >
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Also, is there any clear information about KV api? Is there any plan to
implement it? Or is there any proposal about it?

чт, 1 июл. 2021 г., 13:51 Ivan Daschinsky :

> Pavel, but IEP is in progress, isn't it?
>
> 1. There is not any information about tuple serialization. And there isn't
> a clear consensus about it.
> 2. There is not any information about schrma serialization format. And
> AFAIK, there isn't a clear consensus also.
>
> чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :
>
>> Igniters,
>>
>> Please review the IEP for thin client protocol in 3.0 [1].
>> PoC is in progress [2]
>>
>> [1]
>>
>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
>>
>> [2] https://github.com/apache/ignite-3/pull/191
>>
>

Re: IEP-76 Thin Client Protocol for Ignite 3.0

[Ivan Daschinsky]

Pavel, but IEP is in progress, isn't it?

1. There is not any information about tuple serialization. And there isn't
a clear consensus about it.
2. There is not any information about schrma serialization format. And
AFAIK, there isn't a clear consensus also.

чт, 1 июл. 2021 г., 13:26 Pavel Tupitsyn :

> Igniters,
>
> Please review the IEP for thin client protocol in 3.0 [1].
> PoC is in progress [2]
>
> [1]
>
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0
>
> [2] https://github.com/apache/ignite-3/pull/191
>

IEP-76 Thin Client Protocol for Ignite 3.0

[Pavel Tupitsyn]

Igniters,

Please review the IEP for thin client protocol in 3.0 [1].
PoC is in progress [2]

[1]
https://cwiki.apache.org/confluence/display/IGNITE/IEP-76+Thin+Client+Protocol+for+Ignite+3.0

[2] https://github.com/apache/ignite-3/pull/191

Re: [VOTE][EXTENSION] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC5

[Nikolay Izhikov]

+1 (binding)

> 1 июля 2021 г., в 09:29, Petrov Mikhail  написал(а):
> 
> +1
> 
> Checked on Ubuntu 20.04. Checked built-report and print-statistics scripts 
> for the custom Ignite application. Checked archive signs. Built from the 
> sources.
> 
> On 01.07.2021 00:24, Sergei Ryzhov wrote:
>> +1
>> Checked a report with an ignite-performance-statistics-example.
>> 

Re: [Announcement] Apache Ignite 2.11 Code Freeze started

[Pavel Pereslegin]

Hello, Alexey!

Is it possible to include a hotfix that corrects the command syntax
output in the control script? [1]

This bug can significantly complicate the use of the snapshot restore
function (one of the important features of 2.11). In addition, this
may raise a number of questions to the product support, which we can
prevent by adding this patch in 2.11.

This patch does not affect any functions other than the "help" output
of the control script.

[1] https://issues.apache.org/jira/browse/IGNITE-14989

чт, 1 июл. 2021 г. в 11:56, Alexey Gidaspov :
>
> Hi, Iilya!
>
> As I can see, this feature highly improves debugging during incidents. So I 
> think we can call it blocker and cherry-pick to ignite-2.11 branch
>
> On 2021/06/30 20:26:43, Shishkov Ilya  wrote:
> > Hello, Alexey!
> > Is it possible to add system views for BaselineNode attributes [1] and
> > corresponding documentation [2] to 2.11?
> >
> > 1. https://issues.apache.org/jira/browse/IGNITE-15007
> > 2. https://issues.apache.org/jira/browse/IGNITE-15028
> >
> > ср, 30 июн. 2021 г. в 11:07, Nikita Amelchev :
> >
> > > Thanks! I have cherry-picked the commit to the 2.11 branch.
> > >
> > > ср, 30 июн. 2021 г. в 11:00, Alexey Gidaspov :
> > > >
> > > > Hi, Nikita!
> > > >
> > > > I think it's important fix and should be included in 2.11 release. I've
> > > tagged ticket by fixVersion 2.11. Can you cherry-pick it to ignite-2.11
> > > branch? And please fill release notes or delete flag.
> > > >
> > > > On 2021/06/30 07:55:04, Nikita Amelchev  wrote:
> > > > > Hello, Alexey.
> > > > >
> > > > > I suggest adding to the 2.11 scope the resolved issue [1]. It fixes
> > > > > incorrect values of cache, cache groups, data region metrics after
> > > > > cluster reactivation.
> > > > > WDYT?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-14990
> > > > >
> > > > > вт, 29 июн. 2021 г. в 15:09, Alexey Gidaspov :
> > > > > >
> > > > > > Ok, we can add this fix to release scope.
> > > > > >
> > > > > > On 2021/06/29 08:36:00, Сурков Александр
> > > Викторович wrote:
> > > > > > > Hi Alexey.
> > > > > > >
> > > > > > > I think need add ticket: JmxMetricExporter fails to export
> > > discovery metrics - https://issues.apache.org/jira/browse/IGNITE-14376
> > > > > > >
> > > > > > > On 2021/06/15 14:43:55, Alexey Gidaspov  wrote:
> > > > > > > > Apache Ignite 2.11 Code Freeze started now>
> > > > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Best wishes,
> > > > > Amelchev Nikita
> > > > >
> > >
> > >
> > >
> > > --
> > > Best wishes,
> > > Amelchev Nikita
> > >
> >

Re: Ignite 3.0 IgniteTables API Improvement Suggestion

[Pavel Tupitsyn]

Val, agreed.
Let's add length(), value(index), and Iterable to the Tuple interface.

I've updated the ticket: https://issues.apache.org/jira/browse/IGNITE-14342

On Wed, Jun 30, 2021 at 11:17 PM Valentin Kulichenko <
valentin.kuliche...@gmail.com> wrote:

> Pavel,
>
> Thanks for your response, makes sense.
>
> Regarding the values() method: I would instead add the required methods to
> the Tuple itself. E.g., it can implement Iterable, and additionally have
> the value(int index) method, plus anything else that we might need.
> I don't like returning a collection, because in Java it's a much wider API.
> We will end up returning an implementation that throws
> UnsupportedOperationException for most of the methods. I know this approach
> is not uncommon in the Java world, but this doesn't make it good.
> In .NET and other languages, we can use other abstractions, of course.
> Every platform has its own specifics and practices, so APIs don't have to
> be identical.
>
> -Val
>
> On Wed, Jun 30, 2021 at 7:44 AM Pavel Tupitsyn 
> wrote:
>
> > Hi Andrey,
> >
> > > This will force us to bother about interfaces/contracts and
> compatibility
> > > aspects in the future
> >
> > Schemas and versions are part of thin client wire protocol.
> > This protocol is a public API - we'll have to care about compatibility
> > anyway.
> >
> > Schema evolution is an important feature that users should understand.
> > I see no reason to hide it from the API.
> >
> >
> > > We already have a ticket [1]...
> > > Will 'idx -> column' mapping only be enough for your purposes?
> >
> > The ticket sounds reasonable, but there are no API details.
> > At the very least we need public access to column names, types, and
> > indexes.
> > I propose something like this:
> >
> > interface Tuple {
> > TupleSchema getSchema();
> > }
> >
> > class TupleSchema {
> > int getVersion();
> > List getColumns();
> > }
> >
> > class TupleSchemaColumn {
> > int index;
> > String name;
> > String typeName;
> > bool isKey;
> > }
> >
> > On Wed, Jun 30, 2021 at 11:05 AM Andrey Mashenkov <
> > andrey.mashen...@gmail.com> wrote:
> >
> > > Hi Pavel,
> > >
> > > 2. Schema and its version are internal things and shouldn't be exposed,
> > > otherwise, eventually, it will lead to the user will manage schemas
> > > manually on his side for some purposes.
> > > This will force us to bother about interfaces/contracts and
> compatibility
> > > aspects in the future with uncertain benefits for a user.
> > >
> > > As SchemaDescriptor is an internal API and the user expects a public
> API.
> > > I don't think we want to convert the descriptor back into public API
> > terms
> > > and it may be impossible for some data.
> > >
> > > We already have a ticket [1] to support accessing a column by an index
> > and
> > > exposing some metadata.
> > > Will 'idx -> column' mapping only be enough for your purposes?
> > >
> > > > int Tuple.columnIndex(String)
> > >
> > >
> > >
> > >  [1] https://issues.apache.org/jira/browse/IGNITE-14342
> > >
> > > On Wed, Jun 30, 2021 at 9:34 AM Pavel Tupitsyn 
> > > wrote:
> > >
> > > > Hi Val,
> > > >
> > > > Thanks for the comments, please see below:
> > > >
> > > >
> > > > > Users will identify tables using names, they will never use IDs
> > > >
> > > > Ok, let's keep it this way.
> > > >
> > > >
> > > > > Sounds like the Tuple should implement Iterable.
> > > >
> > > > I don't think Iterable is enough.
> > > > We should have a way to get values by column index: Tuple.value(int
> > > index),
> > > > where index is according to column order in schema.
> > > >
> > > > Combined with Tuple.schema(), it will provide an efficient way to
> > consume
> > > > data -
> > > > users will be able to read columns in any order, skip some of them,
> > etc.
> > > >
> > > > This is a common pattern in APIs like ODBC or System.Data [1],
> > > > which we'll need to implement on top of our thin clients later.
> > > >
> > > >
> > > > > However, whether a user might
> > > > > need to get a table schema for a particular version, I'm not sure.
> Do
> > > you
> > > > > have a use case in mind for this? If not, I would keep this
> internal
> > > >
> > > > Ok, we can keep the Table.schema(ver) method internal, as long as
> > > > Table.schemas() is public and includes schema versions.
> > > >
> > > >
> > > > > We already have async counterparts for all the methods where this
> is
> > > > applicable
> > > >
> > > > IgniteTables.createTable(), dropTable(), tables(), table() do not
> have
> > > > async counterparts,
> > > > while internally they perform blocking wait on some futures.
> > > >
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://docs.microsoft.com/en-us/dotnet/api/system.data.idatareader?view=net-5.0
> > > >
> > > > On Wed, Jun 30, 2021 at 4:51 AM Valentin Kulichenko <
> > > > valentin.kuliche...@gmail.com> wrote:
> > > >
> > > > > Hi Pavel,
> > > > >
> > > > > Please see my comments below.
> > > > >
> > > > > -Val
> > > > >
> > > > 

Re: [Announcement] Apache Ignite 2.11 Code Freeze started

[Alexey Gidaspov]

Hi, Iilya!

As I can see, this feature highly improves debugging during incidents. So I 
think we can call it blocker and cherry-pick to ignite-2.11 branch 

On 2021/06/30 20:26:43, Shishkov Ilya  wrote: 
> Hello, Alexey!
> Is it possible to add system views for BaselineNode attributes [1] and
> corresponding documentation [2] to 2.11?
> 
> 1. https://issues.apache.org/jira/browse/IGNITE-15007
> 2. https://issues.apache.org/jira/browse/IGNITE-15028
> 
> ср, 30 июн. 2021 г. в 11:07, Nikita Amelchev :
> 
> > Thanks! I have cherry-picked the commit to the 2.11 branch.
> >
> > ср, 30 июн. 2021 г. в 11:00, Alexey Gidaspov :
> > >
> > > Hi, Nikita!
> > >
> > > I think it's important fix and should be included in 2.11 release. I've
> > tagged ticket by fixVersion 2.11. Can you cherry-pick it to ignite-2.11
> > branch? And please fill release notes or delete flag.
> > >
> > > On 2021/06/30 07:55:04, Nikita Amelchev  wrote:
> > > > Hello, Alexey.
> > > >
> > > > I suggest adding to the 2.11 scope the resolved issue [1]. It fixes
> > > > incorrect values of cache, cache groups, data region metrics after
> > > > cluster reactivation.
> > > > WDYT?
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-14990
> > > >
> > > > вт, 29 июн. 2021 г. в 15:09, Alexey Gidaspov :
> > > > >
> > > > > Ok, we can add this fix to release scope.
> > > > >
> > > > > On 2021/06/29 08:36:00, Сурков Александр
> > Викторович wrote:
> > > > > > Hi Alexey.
> > > > > >
> > > > > > I think need add ticket: JmxMetricExporter fails to export
> > discovery metrics - https://issues.apache.org/jira/browse/IGNITE-14376
> > > > > >
> > > > > > On 2021/06/15 14:43:55, Alexey Gidaspov  wrote:
> > > > > > > Apache Ignite 2.11 Code Freeze started now>
> > > > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Best wishes,
> > > > Amelchev Nikita
> > > >
> >
> >
> >
> > --
> > Best wishes,
> > Amelchev Nikita
> >
> 

Re: Collision SPI Not Adhering to Specification

[Atri Sharma]

I have opened a JIRA for the same:

https://issues.apache.org/jira/browse/IGNITE-15043

Unless objections, I plan on sketching an implementation plan.

On Thu, Jul 1, 2021 at 1:07 AM Atri Sharma  wrote:
>
> Hi All,
>
> I have been playing around with Collision SPI and specifically used
> FifoQueueCollisionSPI and noticed that it is not really adhering to
> the specified task of restricting the number of concurrent tasks that
> can be run.
>
> Specifically, if there is only one slot available and N tasks
> concurrently land onto the node, all nodes will take the current
> active count, compare it with maximum jobs count and proceed.
>
> Essentially, we have no concurrency safety there.
>
> I propose refactoring FifoQueueCollisionSPI to use semaphores and/or
> atomic variables to ensure that all jobs get a realistic view of the
> active count.
>
> Please share your thoughts,
>
> Regards,
>
> Atri
>
> --
> Regards,
>
> Atri
> Apache Concerted



-- 
Regards,

Atri
Apache Concerted

Re[2]: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Zhenya Stanilovsky]

+1 for reverting, can anybody (possibly ticket starter?) explain how jvm 
settings will rise some security problems ?


 
>I suppose, that we should revert this particular line. I don't understand
>who ever considers vm options as sensitive info.
>
>ср, 30 июн. 2021 г., 22:52 Shishkov Ilya < shishkovi...@gmail.com >:
> 
>> Hi Igniters,
>>
>> This feature [1, 2] prevents logging of the VM arguments when
>> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method
>> IgniteKernal#ackVmArguments remains mostly the same [3].
>>
>> Is this behaviour actual now? Often, we should be able to get from logs the
>> actual VM options used at startup even if output of sensitive data is
>> restricted.
>>
>> 1.  https://issues.apache.org/jira/browse/IGNITE-4991
>> 2.
>>
>>  
>> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
>> 3.
>>
>>  
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
>> 
 
 
 
 

Re: [VOTE][EXTENSION] Release Apache Ignite performance-statistics-ext extension 1.0.0 RC5

[Petrov Mikhail]

+1

Checked on Ubuntu 20.04. Checked built-report and print-statistics 
scripts for the custom Ignite application. Checked archive signs. Built 
from the sources.


On 01.07.2021 00:24, Sergei Ryzhov wrote:

+1
Checked a report with an ignite-performance-statistics-example.

Re: Setting IGNITE_TO_STRING_INCLUDE_SENSITIVE=false prevents VM Arguments output

[Ivan Daschinsky]

I suppose, that we should revert this particular line. I don't understand
who ever considers vm options as sensitive info.

ср, 30 июн. 2021 г., 22:52 Shishkov Ilya :

> Hi Igniters,
>
> This feature [1, 2] prevents logging of the VM arguments when
> IGNITE_TO_STRING_INCLUDE_SENSITIVE option is set to false. Till now, method
> IgniteKernal#ackVmArguments remains mostly the same [3].
>
> Is this behaviour actual now? Often, we should be able to get from logs the
> actual VM options used at startup even if output of sensitive data is
> restricted.
>
> 1. https://issues.apache.org/jira/browse/IGNITE-4991
> 2.
>
> https://github.com/apache/ignite/pull/2428/commits/4f90b6fd77bd23fa818620f0757b792ba388ef93
> 3.
>
> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/internal/IgniteKernal.java#L3002
>