Re: Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

We discussed this with Pavel and Anton just a moment ago. Summary follows.

- New byte "flag" is to be added (ENCODED_STRING)
- 'Encoding' property is to be added at
  -- global level (BinaryConfiguration)
  -- per-class level (BinaryTypeConfiguration)
  -- per-field level (BinaryTypeConfiguration)

2017-07-28 14:15 GMT+03:00 Vladimir Ozerov [via Apache Ignite Developers] <
ml+s2346864n20159...@n4.nabble.com>:

> As Pavel mentioned, Marshaller should not be tied to cache, BinaryObject
> should be self-explanatory, i.e. containing all information necessary for
> unmarshalling. This is an absolute requirement.
>
> We will have one extra byte for in serialized form, meaning that advantage
> of custom encoding will become evident for all strings with length >= 1,
> which is perfectly fine. I do not quite understand what are we arguing
> about.
>
> As far as configuration, we can do it as follows:
>
> 1) Add global encoding, UTF8 by default.
> 2) Add per-cache encoding.
> 3) Add encoding to JDBC and ODBC driver properties.
>
> This should be enough.
>
>
-- 
Best regards,
  Andrey Kuznetsov.




--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Non-UTF-8-string-encoding-support-in-BinaryMarshaller-IGNITE-5655-tp20024p20161.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

Currently, marshaller determines the type of field (BYTE, INT, STRING etc.)
only by the Class of data being serialized. It seems rather non-trivial to
manage marshaling parameters at cache creation point. Alternatively, there
exists simple and flexible way: just to introduce new Java type, say,
StringWithEncoding, but it looks ugly to my mind.

2017-07-28 14:45 GMT+03:00 Vladimir Ozerov :

> String encoding is a concept similar to "collation" in RDBMS. You can
> define it either globally, or on per-table basis. The same should be done
> for Ignite. We do not define behavior of a type. We define behavior of a
> *storage*.
>
> Two cases when proposed approach with per-type and per-type-field approach
> doesn't work:
> 1) I have a class Person with field "name". I have two caches/tables - one
> for US persons, where name is in Latin, another for RU persons with
> Cyrillic names. How can achieve optimal encoding formats for both tables?
> 2) I have an empty grid. Now I want to create a cache/table with custom
> encoding. How can I do that without cluster restart? Nohow, because
> BinaryTypeConfiguration configured statically, while caches/tables can be
> created in runtime.
>

[GitHub] ignite pull request #2391: Ignite 5655

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2391

Ignite 5655

First implementation, uses global level configuration option (in 
BinaryConfiguration)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-5655

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2391.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2391


commit c682ed622c6896343dd6c46025ee6667c788cbcd
Author: andrey-kuznetsov <stku...@gmail.com>
Date:   2017-07-28T15:07:36Z

IGNITE-5655: Draft ENCODED_STRING type support

commit ae939fb26df34c9e9ac074c9b9414e05354053df
Author: Andrey Kuznetsov <stku...@mail.ru>
Date:   2017-08-01T18:21:58Z

IGNITE-5655: Added ENCODED_STRING to BinaryWriteMode as well

commit 9abd99153e154c77faa2e25d809446def83f1378
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-02T16:18:17Z

IGNITE-5655: Repaired BinaryMarshallerSelfTest for lossless encodings.

commit 2d25a76ca6751da027c9e955c2d68fedbcf68d7c
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-03T15:17:28Z

IGNITE-5655: Removed 'default' encoding to preserve compatibility.

commit f540a84b1f03c90ff128a702b1ae73f70facfc85
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-03T15:20:55Z

IGNITE-5655: String binary marshalling tests for non-utf-8 encodings

commit 305a25052a2fc4b2cfe0848107cd226ef2c0daec
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-03T15:41:51Z

Merge branch 'master' into ignite-5655

commit ac83e9695a4240d2d2c504713f913167319b275d
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-03T15:52:25Z

IGNITE-5655: Satisfying @NotNull contract

commit 4cafde333816e4bb9858801283878d4e8c577c46
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-03T16:12:44Z

IGNITE-5655: Removed redundant TODOs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

IgniteSemaphore methods semantics

[Andrey Kuznetsov]

Hi, igniters!



As IgniteSemaphore's javadoc states,



"Distributed semaphore provides functionality similar to {@code
java.util.concurrent.Semaphore}."



At the same time method semantics of current implementation is inverted,
i.e. acquire() decrements internal semaphore count and release() increments
count. Then newlyCreatedSemaphore.acquire() call blocks until some other
thread calls release(), and it looks confusing.Also, availablePermits()
returns permits acquired so far, that is, semaphore count.



Another difference is unbounded nature of IgniteSemaphore implementation,
while java.util.concurrent.Semaphore is bounded.



I think we are to do one of the following:



- Document uncommon IgniteSemaphore semantics properly



or



- Change its semantics to conform java.util.concurrent counterpart.



--

Best regards,

  Andrey Kuznetsov.

[GitHub] ignite pull request #2434: IGNITE-6009: Fixed IgniteSemaphore test.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2434

IGNITE-6009: Fixed IgniteSemaphore test.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6009

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2434.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2434


commit 0aa7cc63a58ae2699c225d99b5b147ee50c7
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-11T12:51:31Z

IGNITE-6009: Fixed IgniteSemaphore test.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

Hi Igniters,I'd like to discuss future changes related to  IGNITE-5655
<https://issues.apache.org/jira/browse/IGNITE-5655>  . Is it really good
idea to introduce new flag (ENCODED_STRING) for existing String datatype?
It's possible to use existing STRING flag at negligible performance cost.
Currently, utf-8-encoded string looks like 
byteFlag nonNegativeIntStrLen bytes
This format can be backward compatibly extended to
byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
Next, I suggest to add new BinaryConfiguration property for encoding to use
instead of using global property. It seems to be more convenient for
user.I'll appreciate your feedback.



-
Best regards,
  Andrey Kuznetsov.
--
View this message in context: 
http://apache-ignite-developers.2346864.n4.nabble.com/Non-UTF-8-string-encoding-support-in-BinaryMarshaller-IGNITE-5655-tp20024.html
Sent from the Apache Ignite Developers mailing list archive at Nabble.com.

Re: Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

I apologize for damaged formatting. Below is my message as it should be.


Hi Igniters,

I'd like to discuss future changes related to https://issues.apache.org/
jira/browse/IGNITE-5655.

Is it really good idea to introduce new flag (ENCODED_STRING) for existing
String datatype? It's possible to use existing STRING flag at negligible
performance cost.

Currently, utf-8-encoded string looks like

byteFlag nonNegativeIntStrLen bytes

This format can be backward compatibly extended to

byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes

Next, I suggest to add new BinaryConfiguration property for encoding to use
instead of using global property. It seems to be more convenient for user.

I'll appreciate your feedback.

2017-07-25 16:13 GMT+03:00 Andrey Kuznetsov <stku...@gmail.com>:

> Hi Igniters,I'd like to discuss future changes related to  IGNITE-5655
> <https://issues.apache.org/jira/browse/IGNITE-5655>  . Is it really good
> idea to introduce new flag (ENCODED_STRING) for existing String datatype?
> It's possible to use existing STRING flag at negligible performance cost.
> Currently, utf-8-encoded string looks like
> byteFlag nonNegativeIntStrLen bytes
> This format can be backward compatibly extended to
> byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
> Next, I suggest to add new BinaryConfiguration property for encoding to use
> instead of using global property. It seems to be more convenient for
> user.I'll appreciate your feedback.
>
>
>
> -
> Best regards,
>   Andrey Kuznetsov.
> --
> View this message in context: http://apache-ignite-
> developers.2346864.n4.nabble.com/Non-UTF-8-string-encoding-
> support-in-BinaryMarshaller-IGNITE-5655-tp20024.html
> Sent from the Apache Ignite Developers mailing list archive at Nabble.com.




-- 
Best regards,
  Andrey Kuznetsov.

Re: Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

Vladimir,

Thanks for reply. In any case, we'll break compatibility by introducing new
feature in marshalling. But both approaches preserve backward
compatibility.

I deemed it's unusual to make two differerent type markers (flags) for
single datatype. I can't see the source right now, but I'm unsure whether
it's possible to map two flags to single type in marshaller implementation.

25 июля 2017 г. 20:13 пользователь "Vladimir Ozerov" <voze...@gridgain.com>
написал:

> Andrey,
>
> You cannot have optional part in the middle as it will break compatibility
> in dangerous way, probably leading to node crash. Also having INT (4
> bytes)
> looks too much for me.
>
> Instead, I would add new type "encoded string":
> 1 byte - type
> 1 byte - encoding code, map frequently used encodings to some byte value;
> also have a special value, meaning that encoding will be written as string
> afterwards, this way we will support any encoding out of the box
> [optional] encoding name
> 4 bytes - string length
> Finally - string bytes
>
> Vladimir.
>
> вт, 25 июля 2017 г. в 18:24, Andrey Kuznetsov <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20039=0>>:
>
> > I apologize for damaged formatting. Below is my message as it should be.
> >
> >
> > Hi Igniters,
> >
> > I'd like to discuss future changes related to https://issues.apache.org/
> > jira/browse/IGNITE-5655
> > <https://issues.apache.org/jira/browse/IGNITE-5655>.
> >
> > Is it really good idea to introduce new flag (ENCODED_STRING) for
> existing
> > String datatype? It's possible to use existing STRING flag at negligible
> > performance cost.
> >
> > Currently, utf-8-encoded string looks like
> >
> > byteFlag nonNegativeIntStrLen bytes
> >
> > This format can be backward compatibly extended to
> >
> > byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
> >
> > Next, I suggest to add new BinaryConfiguration property for encoding to
> use
> > instead of using global property. It seems to be more convenient for
> user.
> >
> > I'll appreciate your feedback.
> >
> > 2017-07-25 16:13 GMT+03:00 Andrey Kuznetsov <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20039=1>>:
> >
> > > Hi Igniters,I'd like to discuss future changes related to  IGNITE-5655
> > > <https://issues.apache.org/jira/browse/IGNITE-5655>  . Is it really
> good
> > > idea to introduce new flag (ENCODED_STRING) for existing String
> datatype?
> > > It's possible to use existing STRING flag at negligible performance
> cost.
> > > Currently, utf-8-encoded string looks like
> > > byteFlag nonNegativeIntStrLen bytes
> > > This format can be backward compatibly extended to
> > > byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
> > > Next, I suggest to add new BinaryConfiguration property for encoding
> to
> > use
> > > instead of using global property. It seems to be more convenient for
> > > user.I'll appreciate your feedback.
> > >
> > >
> > >
> > > -
> > > Best regards,
> > >   Andrey Kuznetsov.
> > > --
> > > View this message in context: http://apache-ignite-
> > > developers.2346864.n4.nabble.com/Non-UTF-8-string-encoding-
> > > support-in-BinaryMarshaller-IGNITE-5655-tp20024.html
> > > Sent from the Apache Ignite Developers mailing list archive at
> > Nabble.com.
> >
> >
> >
> >
> > --
> > Best regards,
> >   Andrey Kuznetsov.
> >
>
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/Non-UTF-8-string-encoding-support-in-BinaryMarshaller-
> IGNITE-5655-tp20024p20039.html
> To unsubscribe from Non-UTF-8 string encoding support in BinaryMarshaller
> (IGNITE-5655), click here
> <http://apache-ignite-developers.2346864.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code=20024=c3RrdXptYUBnbWFpbC5jb218MjAwMjR8LTUwMjc0NDk4NA==>
> .
> NAML
> <http://apache-ignite-developers.2346864.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer=instant_html%21nabble%3Aemail.naml=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>

Assertions as binary data validation checks in deserialization

[Andrey Kuznetsov]

Hi Igniters,

While examining BinaryObjectImpl code I found this curious line in typeId()
method:

  assert arr[off] == GridBinaryMarshaller.STRING : arr[off];

Is it OK to check external binary data with assertions?
I think it can lead to undefined behaviour on corrupt data from the wire.

--
Best regards,
  Andrey Kuznetsov.

Re: Assertions as binary data validation checks in deserialization

[Andrey Kuznetsov]

Valentin,

I meant the behaviour of this code when corrupted data from the network are
being deserialized. Assertion is no-op in production, and we silently
ignore binary format violation.

27 июля 2017 г. 21:09 пользователь "Valentin Kulichenko" <
valentin.kuliche...@gmail.com> написал:

> Andrey,
>
> How will it corrupt the data? Assertions only reads the array, not updates
> it, right?
>
> -Val
>
> On Thu, Jul 27, 2017 at 8:54 AM, Andrey Kuznetsov <stku...@gmail.com>
> wrote:
>
> > Hi Igniters,
> >
> > While examining BinaryObjectImpl code I found this curious line in
> typeId()
> > method:
> >
> >   assert arr[off] == GridBinaryMarshaller.STRING : arr[off];
> >
> > Is it OK to check external binary data with assertions?
> > I think it can lead to undefined behaviour on corrupt data from the wire.
> >
> > --
> > Best regards,
> >   Andrey Kuznetsov.
> >
>

Re: Assertions as binary data validation checks in deserialization

[Andrey Kuznetsov]

May I change this line while working on existing ticket, unless someone
else objects?

27 июля 2017 г. 23:37 пользователь "Valentin Kulichenko" <
valentin.kuliche...@gmail.com> написал:

Makes sense to me. Feel free to create a ticket unless someone else has any
objection. However, I think that we should revise other code for such
places then. Fixing only this one line doesn't change a lot.

-Val

Re: Non-UTF-8 string encoding support in BinaryMarshaller (IGNITE-5655)

[Andrey Kuznetsov]

Vladimir,

It's rather simple to support string encoding by setting it in
BinaryConfiguration. But I'm unsure whether it's a desired change. We need
to express our goal more precisely: should we control encoding at cache
level, field level, or binary configuration level? Currently,
BinaryMarshaller is controlled only by BinaryConfiguration and it's hard
for me to estimate changes to bring string encoding, say, to per-cache
basis.

2017-07-25 20:17 GMT+03:00 Vladimir Ozerov [via Apache Ignite Developers] <
ml+s2346864n20046...@n4.nabble.com>:

> Vyacheslav,
> When we finish varlen optimization for string lengths, I am afraid we
> could
> end up with very messy protocol, should we mix encoded length and
> encoding.
>
> Dima,
> Encoding must be set on per field basis. This will give us as most
> flexible
> solution at the cost of 1-byte overhead.
>
> вт, 25 июля 2017 г. в 20:23, Dmitriy Setrakyan <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20046=0>>:
>
> > I don't understand why this encoding is done on per-object and not on
> > per-cache level. Shouldn't the column-to-encoding mapping be defined at
> > cache level configuration?
> >
> > On Tue, Jul 25, 2017 at 12:13 PM, Vladimir Ozerov <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20046=1>>
> > wrote:
> >
> > > Andrey,
> > >
> > > You cannot have optional part in the middle as it will break
> > compatibility
> > > in dangerous way, probably leading to node crash. Also having INT (4
> > bytes)
> > > looks too much for me.
> > >
> > > Instead, I would add new type "encoded string":
> > > 1 byte - type
> > > 1 byte - encoding code, map frequently used encodings to some byte
> value;
> > > also have a special value, meaning that encoding will be written as
> > string
> > > afterwards, this way we will support any encoding out of the box
> > > [optional] encoding name
> > > 4 bytes - string length
> > > Finally - string bytes
> > >
> > > Vladimir.
> > >
> > > вт, 25 июля 2017 г. в 18:24, Andrey Kuznetsov <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20046=2>>:
> > >
> > > > I apologize for damaged formatting. Below is my message as it should
> > be.
> > > >
> > > >
> > > > Hi Igniters,
> > > >
> > > > I'd like to discuss future changes related to
> > https://issues.apache.org/
> > > > jira/browse/IGNITE-5655
> > > > <https://issues.apache.org/jira/browse/IGNITE-5655>.
> > > >
> > > > Is it really good idea to introduce new flag (ENCODED_STRING) for
> > > existing
> > > > String datatype? It's possible to use existing STRING flag at
> > negligible
> > > > performance cost.
> > > >
> > > > Currently, utf-8-encoded string looks like
> > > >
> > > > byteFlag nonNegativeIntStrLen bytes
> > > >
> > > > This format can be backward compatibly extended to
> > > >
> > > > byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
> > > >
> > > > Next, I suggest to add new BinaryConfiguration property for encoding
> to
> > > use
> > > > instead of using global property. It seems to be more convenient for
> > > user.
> > > >
> > > > I'll appreciate your feedback.
> > > >
> > > > 2017-07-25 16:13 GMT+03:00 Andrey Kuznetsov <[hidden email]
> <http:///user/SendEmail.jtp?type=node=20046=3>>:
> > > >
> > > > > Hi Igniters,I'd like to discuss future changes related to
> > IGNITE-5655
> > > > > <https://issues.apache.org/jira/browse/IGNITE-5655>  . Is it
> really
> > > good
> > > > > idea to introduce new flag (ENCODED_STRING) for existing String
> > > datatype?
> > > > > It's possible to use existing STRING flag at negligible
> performance
> > > cost.
> > > > > Currently, utf-8-encoded string looks like
> > > > > byteFlag nonNegativeIntStrLen bytes
> > > > > This format can be backward compatibly extended to
> > > > > byteFlag [negativeIntCharsetCode] nonNegativeIntStrLen bytes
> > > > > Next, I suggest to add new BinaryConfiguration property for
> encoding
> > to
> > > > use
> > > > > instead of using global property. It seems to be more convenient
> for
> > > > > user.I'll ap

[GitHub] ignite pull request #2339: Ignite 4181 public api

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2339

Ignite 4181 public api



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-4181-public-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2339.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2339


commit 2e5d1206abb6ab897b1f173bd673f3d2fea7f5c7
Author: andrey-kuznetsov <stku...@gmail.com>
Date:   2017-07-24T11:40:15Z

IGNITE-4181: Reproducing with public API

commit 4dc88d742e851f50b3cd276c280dd417843d0dfd
Author: andrey-kuznetsov <stku...@gmail.com>
Date:   2017-07-24T13:04:42Z

IGNITE-4181: Reproducer logic improvement




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ignite Enhancement Proposal process

[Andrey Kuznetsov]

Really cool idea!

It's not convenient to look for subtle details in devlist discussion thread.

But I'd prefer the word "improvement" rather than "enhancement": the stuff
being described is not always a sharp-cut functionality.

2017-09-19 14:56 GMT+03:00 ALEKSEY KUZNETSOV <alkuznetsov...@gmail.com>:

> great suggestion
>
>

-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #2869: IGNITE-1804: Test is not not flaky anymore

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2869

IGNITE-1804: Test is not not flaky anymore



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite master

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2869.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2869


commit ffc3d84245b7f40c6b7f91a5e799aa2cb31a9607
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-10-17T12:25:33Z

IGNITE-1804: Reenabled 
GridCachePartitionedQueueCreateMultiNodeSelfTest.testTx: does not fail since 
v.2.0.




---

IGNITE-5655: Mixing binary string encodings in Ignite cluster

[Andrey Kuznetsov]

Hi Igniters!

I met a couple of issues related to different binary string encoding
settings on different cluster nodes.

Let cluster has two nodes. Node0 uses win-1251 to marshal strings with
BinaryMarshaller and Node1 uses default utf-8 encoding. Let's create
replicated cache and add some entry to Node0:

node0.cache("myCache").put("k", "v");

Then

node1.cache("myCache").get("k")

returns null.

Let me describe the cause. First, string key comes to Node1 as binary
payload of DHT update request, it has win-1251 encoding. This
representation stays in offheap area of Node1. Then GetTask comes with the
same key, plain (Serializable) Java object; BinaryMarshaller encodes the
key using utf-8 (Node1 setting). Finally, B+Tree lookup fails for this
binary key due to different encodings.

When the key is just a string then this can be fixed by decoding binary
strings entirely on B+Tree lookups. But when the key is an arbitrary object
with some strings inside this way is too expensive.

The second issue relates to lossy string encodings. Mixed-encoding cluster
does not guarantee string data integrity when "lossless" node goes down for
a while.

Any ideas on addressing these issues?

-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #2749: IGNITE-6186: Removed redundant parameter.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2749

IGNITE-6186: Removed redundant parameter.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6186

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2749.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2749


commit 9646f442bb6327bb08c984c5e0caf0a9f7438640
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-09-26T12:55:13Z

IGNITE-6186: Removed redundant parameter.




---

[GitHub] ignite pull request #2781: IGNITE-6520: Using actual AffinityReadyFuture res...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2781


---

[GitHub] ignite pull request #2781: IGNITE-6520: Using actual AffinityReadyFuture res...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2781

IGNITE-6520: Using actual AffinityReadyFuture result.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6520

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2781.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2781


commit 568b002006b8d5ddf4b72a7cbebc6d394b3ac240
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-09-29T14:46:23Z

IGNITE-6520: Using actual AffinityReadyFuture result.




---

[GitHub] ignite pull request #2748: IGNITE-6015: Rollback on exception in commitIfLoc...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2748

IGNITE-6015: Rollback on exception in commitIfLocked.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6015

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2748.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2748


commit 854a0e393c171192998d77ce0238d5959f79e629
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-09-26T11:54:08Z

IGNITE-6015: Rollback on exception in commitIfLocked.




---

Re: Ignite Usability: Deadlocks and Starvation

[Andrey Kuznetsov]

It's not public API, just implementation detail.

GridFutureAdapter::unregisterWaiter is not thread state in general. It
won't work properly if the argument is not current thread. And also I
couldn't prove clearly it wouldn't drop some unrelated Node in concurrent
operation.

2017-08-23 22:45 GMT+03:00 Yakov Zhdanov <yzhda...@apache.org>:

> Andrey, do you have an example of such method in Ignite public or private
> API?
>
> If it is not Ignite API please provide an example.
>
> --
> Yakov Zhdanov
>



-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #2496: IGNITE-6137: CAS logic fix in async commit.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2496

IGNITE-6137: CAS logic fix in async commit.

Reproducing test + fix

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6137

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2496.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2496


commit 4411cc2e08661c52f399f354ba4837580628a54a
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-08-22T06:55:15Z

IGNITE-6137: CAS logic fix in async commit.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Ignite Usability: Deadlocks and Starvation

[Andrey Kuznetsov]

Yakov, it would be good to get rid of the parameter at all. Strictly
speaking, such tiny things are not bugs, but can lead to bugs, and I'm
curious about the way of improving the situation.


> Andrey, it seems unregisterWaiter() method is supposed to be called only
> with current thread as parameter.
>
> --Yakov
>



-- 
Best regards,
  Andrey Kuznetsov.

Re: Ignite Usability: Deadlocks and Starvation

[Andrey Kuznetsov]

Hi folks,

Strictly speaking, it's not about improving deadlock/starvation detection
mechanisms, but how do we handle fragile concurrent code that (potentially)
leads to this kind of issues? For example, what if the method is assumed to
be thread-safe, but its safety depends on arguments supplied?

2017-08-23 19:44 GMT+03:00 Yakov Zhdanov <yzhda...@apache.org>:

> Hello, guys!
>
> Currently Ignite clusters are very vulnerable to deadlocks on tx and java
> levels and also thread pool starvation.
>
> Unfortunately, Ignite currently offers very poor functionality to protect
> and resolve situations of the kind.
>
> Here is the page with the info on this -
> https://cwiki.apache.org/confluence/display/IGNITE/Deadlock+Detection+And+
> Cluster+Protection
>
> Please provide comments here on dev list or on wiki. Then we will file
> tickets to plan these changes.
>
> --Yakov
>



-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #2877: IGNITE-1804: Test is not flaky anymore

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/2877

IGNITE-1804: Test is not flaky anymore



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-1804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/2877.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2877


commit 1fb3142caa0b7b9056e51d7659e646bfec2924db
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-10-18T13:13:54Z

IGNITE-1804: Reenabled 
GridCachePartitionedQueueCreateMultiNodeSelfTest.testTx: does not fail since 
v.2.0.




---

[GitHub] ignite pull request #2869: IGNITE-1804: Test is not flaky anymore

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2869


---

[GitHub] ignite pull request #3072: IGNITE-6963: Made PhysicalMemoryPages equal to To...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3072


---

[GitHub] ignite pull request #3232: IGNITE-6731: Dropping unnecessary PerfCounter inv...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3232

IGNITE-6731: Dropping unnecessary PerfCounter invocations.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6731

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3232.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3232


commit 2f71b48c5e6ac54ebba560c27abad4d7d44b47da
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-14T14:35:56Z

IGNITE-6731: Dropping unnecessary PerfCounter invocations.




---

Switching Java language level to 8

[Andrey Kuznetsov]

Hi Igniters,

As far as I know, we planned to get the project compilable/runnable on Java
9 and then switch to Java 8 language features. But some tickets like [1]
require Java 8 to be fixed. It seems we are to switch source level of the
project to 1.8 first.

Thoughts?

[1] https://issues.apache.org/jira/browse/IGNITE-6734

-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3248: IGNITE-6736: Switched GridCacheMapEntry synchroni...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3248

IGNITE-6736: Switched GridCacheMapEntry synchronization to locks.

For test purpose

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6736

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3248.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3248


commit 009e8710779355feb375ce63a60a24f12900c761
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-18T15:36:01Z

IGNITE-6736: Switched GridCacheMapEntry synchronization to locks.




---

Re: IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

[Andrey Kuznetsov]

Hi Anton,

I doubt it was merge conflict. Couple of days ago 'Ignite Start Nodes'
suite succeeded 1 time after unrelated changes, and now it is failing again.

2017-12-12 16:25 GMT+03:00 Anton Vinogradov <avinogra...@gridgain.com>:

> Hi,
>
> Seems we have merge conflict.
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=Ignite20Tests_
> IgniteStartNodes=buildTypeHistoryList_
> Ignite20Tests=%3Cdefault%3E
>
>
> --
Best regards,
  Andrey Kuznetsov.

Separate code paths for Java 8 and Java 9

[Andrey Kuznetsov]

Igniters,

As [1] proposes, a couple of new modules with version-specific code should
be created. While working on issues from [2] umbrella I saw no significant
reasons to emerge separate modules; most requirements can be covered by
tiny conditional code pieces in static init blocks of existing utility
classes.

Could someone share the ideas on the content of potential version-specific
modules?

[1] https://issues.apache.org/jira/browse/IGNITE-6730
[2] https://issues.apache.org/jira/browse/IGNITE-6728

-- 
Best regards,
  Andrey Kuznetsov.

Re: Time and scope for Apache Ignite 2.4

[Andrey Kuznetsov]

Hi Denis,

As for memory metrics, I'm fixing minor bugs now, and then we'll need some
time for review.

2017-12-19 23:07 GMT+03:00 Denis Magda <dma...@apache.org>:

> Igniters,
>
> It’s time to sync up on the progress of IMPORTANT tasks for 2.4:
> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.4#
> ApacheIgnite2.4-RequiredTickets <https://cwiki.apache.org/
> confluence/display/IGNITE/Apache+Ignite+2.4#ApacheIgnite2.4-
> RequiredTickets>
>
> How good are we if to assume 5th of Jan as a code freeze date? Please
> comment:
> - Nikolay Izhikov: Data Frames.
> - Sergey Kalashnikov, Vladimir Ozerov: DDL and SQL features from the list.
> - Andrey Kuznetsov, Alexandr Fedotov: memory metrics tasks.
> - Yuri: Machine Learning + neural networks
> - Vladimir Ozerov: Java 9 and Java 7
> - Sergey Kozlov: SQL scope testing
> - etc.: please check the link with the features.
>
> —
> Denis
>
> --
Best regards,
  Andrey Kuznetsov.

Add emergency node closing handler to public Ignite API

[Andrey Kuznetsov]

Hi Igniters!

When some node detects critical error, e.g. OOME, deadlock, etc, it should
invoke some user-defined callback and then attempt to close itself
gracefully. In order to make this possible we need to enhance Ignite
interface by adding something like Ignite.onEmergencyClose(SomeClosure).

First, I'd like to get your feedback on this potential change. Then we can
refine SomeClosure structure.

-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3072: IGNITE-6963: Made PhysicalMemoryPages equal to To...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3072

IGNITE-6963: Made PhysicalMemoryPages equal to TotalAllocatedPages when PDS 
is off



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6963

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3072.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3072


commit e5d412392c2d7d3e8310f2a7772db8fc104efac0
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-11-20T17:35:14Z

IGNITE-6963: Made PhysicalMemoryPages equal to TotalAllocatedPages when PDS 
is off.




---

Re: Add emergency node closing handler to public Ignite API

[Andrey Kuznetsov]

Vladimir,

Could you please refine, what are local metrics? Should I extend Ignite
interface by adding something similar to dataRegionMetrics() or there is
some universal mechanism to handle metrics?

2017-11-15 8:30 GMT+03:00 Vladimir Ozerov :
>
> This information should be available through local metrics, so that it is
> accessible from Ignite instance.
>

Re: Add emergency node closing handler to public Ignite API

[Andrey Kuznetsov]

Vladimir, Ignite instance won't tell me whether deadlock occurred or some
critical thread has died.

14 нояб. 2017 г. 22:28 пользователь "Vladimir Ozerov" 
написал:

You can get this info from injected Ignite instance.

Re: Add emergency node closing handler to public Ignite API

[Andrey Kuznetsov]

Lifecycle beans are ok, but they does not accept any info on the Reason in
case of emergency node stop.

2017-11-14 21:16 GMT+03:00 Valentin Kulichenko <
valentin.kuliche...@gmail.com>:

> Anton,
>
> I agree with Vova - we already have lifecycle bean. Why do we need anything
> on top of that?
>
> -Val
>
>

[GitHub] ignite pull request #3181: Ignite 6902

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3181

Ignite 6902



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6902

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3181.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3181


commit 2dd026ed3032d2c289187c1f7a66517c1e652bae
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-07T15:45:18Z

IGNITE-6902: Added new metric for checkpoint buffer size.

commit 41efb94112f6c328e7939243af8243d119cc9276
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-08T11:32:34Z

IGNITE-6902: Data region memory usage metrics in bytes.




---

[GitHub] ignite pull request #3184: IGNITE-6729: Removed obsolete reflection-based wo...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3184

IGNITE-6729: Removed obsolete reflection-based workaround



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6729

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3184.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3184


commit c41c4ae310c9531a23a3414c446112cf6a2dfd53
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-08T14:54:16Z

IGNITE-6729: Removed obsolete reflection-based workaround in URLClassLoader 
closing code.




---

[GitHub] ignite pull request #3165: IGNITE-6711: TotalAllocatedPages metric fix.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3165

IGNITE-6711: TotalAllocatedPages metric fix.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6711

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3165.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3165


commit e6c75e81b87ec7e4e2489dd6a96e2f5a0b3d0c89
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-07T08:23:47Z

IGNITE-6711: TotalAllocatedPages metric fix.




---

[GitHub] ignite pull request #2877: IGNITE-1804: Test is not flaky anymore

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2877


---

[GitHub] ignite pull request #3197: Ignite 6734

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3197

Ignite 6734



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6734

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3197


commit aff7c61f66ed0ba8de051f8ab037aa8f3076f167
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-11T12:04:45Z

IGNITE-6734: Switched language level to 8 for entire project.

commit cac8bcc74fc2075a75a74c445757ccce9f86c4f2
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-11T12:08:34Z

IGNITE-6734: Replaced sun.misc.BASE64Encoder with standard Java 8 encoder.




---

[GitHub] ignite pull request #2434: IGNITE-6009: Fixed IgniteSemaphore test.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2434


---

[GitHub] ignite pull request #3199: IGNITE-6740: Replaced direct sun.nio.ch.DirectBuf...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3199

IGNITE-6740: Replaced direct sun.nio.ch.DirectBuffer.address() call with 
reflective one



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6740

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3199


commit 14f5e8ed3ada76464740e5a7d9def469a60e04db
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-11T17:47:50Z

IGNITE-6740: Replaced direct sun.nio.ch.DirectBuffer.address() call with 
reflective one.




---

IgniteProjectionStartStopRestartSelfTest is failing for about a week on TC

[Andrey Kuznetsov]

Hi Igniters,

IgniteProjectionStartStopRestartSelfTest fails with timeout on TeamCity,
maybe due to some infrastructure issues. Can anybody take a look?

-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3206: IGNITE-6743: Cleaning direct buffer through refle...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3206

IGNITE-6743: Cleaning direct buffer through reflection.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6743

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3206


commit b37392cbbdd3fd04d7e4640563a12af9e5a6f09d
Author: Andrey Kuznetsov <stku...@gmail.com>
Date:   2017-12-12T13:11:17Z

IGNITE-6743: Cleaning direct buffer through reflection.




---

IgniteSet implementation: changes required

[Andrey Kuznetsov]

Hi, Igniters!

Current implementation of IgniteSet is fragile with respect to cluster
recovery from a checkpoint. We have an issue (IGNITE-5553) that addresses
set's size() behavior, but the problem is slightly broader. The text below
is my comment from Jira issue. I encourage you to discuss it.

We can put current set size into set header cache entry. This will fix
size(), but we have broken iterator() implementation as well.

Currently, set implementation maintains plain Java sets on every node, see
CacheDataStructuresManager.setDataMap. These sets duplicate backing-cache
entries, both primary and backup. size() and iterator() calls issue
distributed queries to collect/filter data from all setDataMap's. And
setDataMaps remain empty after cluster is recovered from checkpoint.

Now I see the following options to fix the issue.

#1 - Naive. Iterate over all datastructure-backing caches entries during
recover from checkpoint procedure, filter set-related entries and refill
setDataMap's.
Pros: easy to implement
Cons: inpredictable time/memory overhead.

#2 - More realistic. Avoid node-local copies of cache data. Maintain linked
list in datastructure-backing cache: key is set item, value is next set
item. List head is stored in set header cache entry (this set item is
youngest one). Iterators build on top of this structure are fail-fast.
Pros: less memory overhead, no need to maintain node-local mirrors of cache
data
Cons: iterators are not fail-safe.

#3 - Option #2 modified. We can store reference counter and 'removed' flag
along with next item reference. This allows to make iterators fail safe.
Pros: iterators are fail-safe
Cons: slightly more complicated implementation, may affect performance,
also I see no way to handle active iterators on remote nodes failures.


Best regards,

Andrey.

Re: Transaction classes naming

[Andrey Kuznetsov]

Alexey,

And what do you think about 'Adapters' in transaction heirarchy? This term
looks really obfuscating for new contributor, since in fact they are not
adapters for something but abstract superclasses.

2017-12-20 18:18 GMT+03:00 Alexey Goncharuk <alexey.goncha...@gmail.com>:

> Igniters,
>
> As I review transaction-related PRs and communicate with other fellow
> Igniters, I realized that our transaction classes naming became
> inconsistent and almost impossible to explain to a new contributor. A few
> examples:
>
> 1) We have a GridNearTxLocal class, but Near in the name does not
> necessarily mean that this transaction has to do with the near cache. It
> may or may not, depending on the cache configuration. Near in this case
> only stands for _originating_ node transaction. This is also reflected in
> many messages, such as Near*Request, where near stands for a message sent
> from an originating node to the primary node.
> 2) We have GridNearTxRemote and GridDhtTxRemote classes. First, near here
> always means near cache, as near remote transaction will only be created
> for near-enabled caches. Second, Remote in the class names stands for
> backups (affinity backups or temporarily near-reader backups). On the other
> hand, requests sent from primary to backups are named Dht*Request.
>
> All in all, the naming is extremely confusing. For a person who hasn't seen
> the evolution of transaction classes over time, it is impossible to infer
> or even assume what these classes stand for.
>
> I would like to kick off a discussion on new transaction classes and
> messages naming (especially as we started developing MVCC which most likely
> will introduce additional TX classes).
>
> How about:
> OriginatingTx for transaction created on originating node
> PrimaryTx for transaction created on primary nodes, but not the originating
> node
> BackupTx for transaction created on backup nodes
> NearTx for transaction created to update near cache
>
> Thoughts?
>



-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3269: IGNITE-6732: Preventing Hadoop module from start ...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3269

IGNITE-6732: Preventing Hadoop module from start on Java >= 9.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6732

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3269.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3269


commit 03e5f470866b252c17984d2f408ee7dcea43f498
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-21T12:12:57Z

IGNITE-6732: Preventing Hadoop module from start on Java >= 9.




---

Re: Ticket review checklist

[Andrey Kuznetsov]

Anton,

I agree, *MUST* for exception reasons and *SHOULD* for ways of resolution
sound clearer.

2018-05-08 12:56 GMT+03:00 Anton Vinogradov <a...@apache.org>:

> Andrey,
>
> How about
> 1.6) All exceptions thrown to a user *MUST* have explanation of workaround
> and contain original error.
> All exceptions thrown to a user *SHOULD* have explanation how to resolve if
> possible.
> ?
>
> вт, 8 мая 2018 г. в 12:26, Andrey Kuznetsov <stku...@gmail.com>:
>
> > Vladimir, checklist looks pleasant enough for me.
> >
> > I'd like to suggest one minor change. In 1.6 *MUST* seems to be too
> strict,
> > *SHOULD* would be enough. It can be frustrating for API user if I explain
> > how to fix NPEs in a trivial way, for example.
> >
> > 2018-05-08 11:34 GMT+03:00 Anton Vinogradov <a...@apache.org>:
> >
> > > Alex,
> > >
> > > It is not sounds like that, obviously.
> > >
> > > Tests should cover all negative and positive cases.
> > > You should add enough tests to cover all cases.
> > >
> > > Sometimes one test can cover more than one case, so two tests *CAN*
> > > partially check same things.
> > > In case some cases already covered you should not create duplicates.
> > >
> > > вт, 8 мая 2018 г. в 10:19, Александр Меньшиков <sharple...@gmail.com>:
> > >
> > > > Vladimir, the 3.1 is a bit unclear for me. Which code coverage is
> > > > acceptable? Now it sounds like two tests are enough (one for positive
> > and
> > > > one for negative cases).
> > > >
> > > > 2018-05-07 23:09 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org
> >:
> > > >
> > > > > Is this list on the Wiki?
> > > > >
> > > > > On Mon, May 7, 2018 at 7:26 AM, Vladimir Ozerov <
> > voze...@gridgain.com>
> > > > > wrote:
> > > > >
> > > > > > Igniters,
> > > > > >
> > > > > > This is the checklist I have at the moment. Please let me know if
> > you
> > > > > have
> > > > > > any comments on existing items, or want to add or remove
> anything.
> > It
> > > > > looks
> > > > > > like we may have not only strict rules, but *nice to have* points
> > > here
> > > > as
> > > > > > well with help of *MUST*, *SHOULD* and *MAY* words as per RFC2119
> > > [1].
> > > > So
> > > > > > please feel free to suggest optional items as well.
> > > > > >
> > > > > > 1) API
> > > > > > 1.1) API compatibility *MUST* be maintained between minor
> releases.
> > > Do
> > > > > not
> > > > > > remove existing methods or change their signatures, deprecate
> them
> > > > > instead
> > > > > > 1.2) Default behaviour "SHOULD NOT* be changed between minor
> > > releases,
> > > > > > unless absolutely needed. If change is made, it *MUST* be
> described
> > > in
> > > > > > "Migration Guide"
> > > > > > 1.3) New operation *MUST* be well-documented in code (javadoc,
> > > > > dotnetdoc):
> > > > > > documentation must contain method's purpose, description of
> > > parameters
> > > > > and
> > > > > > how their values affect the outcome, description of return value
> > and
> > > > it's
> > > > > > default, behavior in negative cases, interaction with other
> > > operations
> > > > > and
> > > > > > components
> > > > > > 1.4) API parity between Java and .NET platforms *SHOULD* be
> > > maintained
> > > > > when
> > > > > > operation makes sense on both platforms. If method cannot be
> > > > implemented
> > > > > in
> > > > > > a platform immediately, new JIRA ticket *MUST* be created and
> > linked
> > > to
> > > > > > current ticket
> > > > > > 1.5) API parity between thin clients (Java, .NET) *SHOULD* be
> > > > maintained
> > > > > > when operation makes sense on several clients. If method cannot
> be
> > > > > > implemented in a client immediately, new JIRA ticket *MUST* be
> > > created
> > > > > and
> > > > > > linked to current ticket
> > > > > >

Re: Ticket review checklist

[Andrey Kuznetsov]

oding Guidelines
> > > >
> > > > Vladimir.
> > > >
> > > > [1] https://www.ietf.org/rfc/rfc2119.txt
> > > >
> > > > On Fri, May 4, 2018 at 4:33 PM, Vladimir Ozerov <
> voze...@gridgain.com>
> > > > wrote:
> > > >
> > > > > Hi Dmitry,
> > > > >
> > > > > Yes, I'll do that in the nearest days.
> > > > >
> > > > > On Wed, Apr 25, 2018 at 8:24 PM, Dmitry Pavlov <
> > dpavlov@gmail.com>
> > > > > wrote:
> > > > >
> > > > >> Igniters, the idea was related to small refactorings co-located
> with
> > > > main
> > > > >> change.
> > > > >>
> > > > >> Main change itself indicates that existing code did not meet the
> > > > criteria
> > > > >> of practice. Approving of standalone refactorings instead
> > contradicts
> > > > with
> > > > >> principle don't touch if it works. So I still like idea of
> > co-located
> > > > >> changes improving code, javadocs, style, etc.
> > > > >>
> > > > >> But let's not argue about this point now, let's summarize the
> > > undisputed
> > > > >> points and add it to the wiki. Vladimir, would you please do it?
> > > > >>
> > > > >>
> > > > >> ср, 25 апр. 2018 г. в 16:42, Nikolay Izhikov <nizhi...@apache.org
> >:
> > > > >>
> > > > >> > Igniters,
> > > > >> >
> > > > >> > I agree with Vova.
> > > > >> >
> > > > >> > Don't fix if it works!
> > > > >> >
> > > > >> > If you 100% sure then it a useful addition to the product - just
> > > make
> > > > a
> > > > >> > separate ticket.
> > > > >> >
> > > > >> > В Ср, 25/04/2018 в 11:44 +0300, Vladimir Ozerov пишет:
> > > > >> > > Guys,
> > > > >> > >
> > > > >> > > The problem with in-place refactorings is that you increase
> > > affected
> > > > >> > scope.
> > > > >> > > It is not uncommon to break compatibility or public contracts
> > with
> > > > >> even
> > > > >> > > minor things. E.g. recently we decided drop org.jsr166 package
> > in
> > > > >> favor
> > > > >> > of
> > > > >> > > Java 8 classes. Innocent change. Result - broken storage.
> > Another
> > > > >> problem
> > > > >> > > is conflicts. It is not uncommon to have long-lived branches
> > which
> > > > we
> > > > >> > need
> > > > >> > > to merge with master over and over again. And a lot of
> > > refactorings
> > > > >> cause
> > > > >> > > conflicts. It is much easier to resolve them if you know that
> > > logic
> > > > >> was
> > > > >> > not
> > > > >> > > affected as opposed to cases when you need to resolve both
> > renames
> > > > and
> > > > >> > > method extractions along with business-logic changes.
> > > > >> > >
> > > > >> > > I'd like to repeat - if you have a time for refactoring then
> you
> > > > >> > definitely
> > > > >> > > have a time to extract these changes to separate PR and
> submit a
> > > > >> separate
> > > > >> > > ticket. I am quite understand what "low priority" do you mean
> if
> > > you
> > > > >> do
> > > > >> > > refactorings on your own.
> > > > >> > >
> > > > >> > > On Tue, Apr 24, 2018 at 10:52 PM, Andrey Kuznetsov <
> > > > stku...@gmail.com
> > > > >> >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > +1.
> > > > >> > > >
> > > > >> > > > Once again, I beg for "small refactoring permission" in a
> > > > checklist.
> > > > >> > As of
> > > > >> > > > today, separate tickets for sm

[GitHub] ignite pull request #4015: Ignite 6587

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4015

Ignite 6587

For test purposes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6587

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4015.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4015


commit 3b25cb934838feb3300c7088db4b3840ec7f9ab5
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-07T12:43:26Z

IGNITE-6587: Refactored ServerImpl.TcpServer to use GridWorker.

commit 665ba863ceb95b643a805d6920f3f384275dae20
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-07T13:06:22Z

IGNITE-6587: Refactored ServerImpl.RingMessageWorker to use GridWorker.

commit b10bcd350efb55de95792b369996ba266b3612a9
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-07T15:59:56Z

IGNITE-6587: Refactored WAL manager critical threads to use GridWorker.

commit 3154112c8bab70d1ff4fe20639648e8a9c7da373
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-07T18:55:25Z

IGNITE-6587: Minor improvements.

commit a44fc8064a9ba8f42e4faacde018ac9bc1193f01
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-08T10:52:01Z

IGNITE-6587: Refactored CommunicationWorker to use GridWorker.

commit 0efc5354f33666ac705c738d75955258084091de
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-08T15:25:56Z

IGNITE-6587: (WIP) switching to timed waits/polls, updating heartbeats.

commit 48c889441a119ef2192c1ff61485c20f80659e82
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-14T13:38:47Z

IGNITE-6587: Heartbeat updates in critical WAL manager workers.

commit 69059bda2071f7d6af0cef30cf8d77b160fc56be
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-14T15:55:51Z

IGNITE-6587: (WiP) More heartbeat updates in critical workers.

commit 6fb2edf1fef15bb187481f89eddb3710d5daadba
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-15T13:09:16Z

IGNITE-6587: Heartbeat updates in checkpointer.

commit 9cd8cba1fdc8c21d999df4c334daaef7e226aa0f
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-15T13:23:17Z

IGNITE-6587: More heartbeat updates in critical workers.

commit 5245abc0c72812c4746127b25638a28dac7af11a
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-16T20:14:31Z

IGNITE-6587: Extended StripedExecutor.Stripe from GridWorker.

commit 2eb448f6a7ff7a6be423742336480a7b98793c2d
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-17T11:37:08Z

IGNITE-6587: Heartbeat updates in StripedExecutor workers.

commit 8304fa92879fe293f112d1c7ba515b5bd30cc271
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-17T12:33:23Z

Merge branch 'master' into ignite-6587

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java

commit 5051aa4d44e78b5362dd0094bc5a57566eea4855
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-17T12:36:23Z

IGNITE-6587: Addition to merge commit.

commit 732d2c25013d94d67188f42b2f96c786eeaf6346
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-05-17T12:42:05Z

IGNITE-6587: Updated WAL FileDecompressors to conform uniform poll style.




---

Re: Avoid JIRA comments deletion

[Andrey Kuznetsov]

Folks,

Let me disagree with you.

As you've mentioned, there is a history of changes for an issue, so nothing
can be lost. Comment deletions can be useful to maintain clear path of
issue evolution. For me, it's ok to delete misleading comments, but this
should be done with care, of course.

2018-05-16 12:51 GMT+03:00 Dmitriy Govorukhin <dmitriy.govoruk...@gmail.com>
:

> +1 Dmitriy,
>
> I do not see any reason for deletion comments.
> Maybe only edit operation must be allowed.
>
>

-- 
Best regards,
  Andrey Kuznetsov.

Re: supporting different configuration format json,yaml...

[Andrey Kuznetsov]

gt;> > And we can do ignite more friendly for them, support
> > familiar
> > > > > > formats
> > > > > > > for
> > > > > > > >> > configuration.
> > > > > > > >> >
> > > > > > > >> > On Tue, May 15, 2018 at 12:13 PM, Dmitry Pavlov <
> > > > > > > dpavlov@gmail.com>
> > > > > > > >> > wrote:
> > > > > > > >> >
> > > > > > > >> > > Hi Igniters,
> > > > > > > >> > >
> > > > > > > >> > > In general I aggree with adding new format, e.g. JSON is
> > > more
> > > > > > > popular
> > > > > > > >> > than
> > > > > > > >> > > XML for new applications.
> > > > > > > >> > >
> > > > > > > >> > > In the same time I've never heard that user asked this
> in
> > > the
> > > > > user
> > > > > > > >> list.
> > > > > > > >> > Or
> > > > > > > >> > > did I missed such topics?
> > > > > > > >> > >
> > > > > > > >> > > Sincerely,
> > > > > > > >> > > Dmitriy Pavlov
> > > > > > > >> > >
> > > > > > > >> > > вт, 15 мая 2018 г. в 9:31, Pavel Tupitsyn <
> > > > ptupit...@apache.org
> > > > > >:
> > > > > > > >> > >
> > > > > > > >> > > > Dmitriy,
> > > > > > > >> > > >
> > > > > > > >> > > > We don't need to support different config formats on
> > > server
> > > > in
> > > > > > > order
> > > > > > > >> to
> > > > > > > >> > > add
> > > > > > > >> > > > that to thin clients.
> > > > > > > >> > > >
> > > > > > > >> > > > Thin client protocol provides a way to create a cache
> > with
> > > > > > custom
> > > > > > > >> > config
> > > > > > > >> > > > [1].
> > > > > > > >> > > > It is up to thin client library authors to use any
> > config
> > > > > format
> > > > > > > they
> > > > > > > >> > > like
> > > > > > > >> > > > and then convert it into protocol-defined format.
> > > > > > > >> > > >
> > > > > > > >> > > > C# thin client uses custom format, for example, not
> > > Spring.
> > > > > > > >> > > >
> > > > > > > >> > > > [1]
> > > > > > > >> > > >
> > > > > > > >> > > > https://apacheignite.readme.io/docs/binary-client-
> > > > > > > >> > > protocol-cache-configuration-
> operations#section-op_cache_
> > > > > > > >> > > create_with_configuration
> > > > > > > >> > > >
> > > > > > > >> > > > On Mon, May 14, 2018 at 7:54 PM, Ivan Rakov <
> > > > > > > ivan.glu...@gmail.com>
> > > > > > > >> > > wrote:
> > > > > > > >> > > >
> > > > > > > >> > > > > Dmitry,
> > > > > > > >> > > > >
> > > > > > > >> > > > > We rely on Spring Framework when we start Ignite
> node
> > > from
> > > > > XML
> > > > > > > >> > > > > configuration. Spring doesn't easily support another
> > > > formats
> > > > > > of
> > > > > > > >> > > > > configuration files. I think, the main reason for
> this
> > > is
> > > > > > > built-in
> > > > > > > >> > > > ability
> > > > > > > >> > > > > to validate configuration via XML Schema. We can
> > surely
> > > > hack
> > > > > > > this
> > > > > > > >> > > around
> > > > > > > >> > > > (I
> > > > > > > >> > > > > bet there are existing libraries for configuring
> > Spring
> > > > with
> > > > > > > JSON),
> > > > > > > >> > > but I
> > > > > > > >> > > > > don't think that anyone suffered from inability to
> > > > > statically
> > > > > > > >> > configure
> > > > > > > >> > > > > Ignite with json/yaml.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Regarding thin clients: makes sense. I suppose
> > necessary
> > > > > > > mappings
> > > > > > > >> > will
> > > > > > > >> > > be
> > > > > > > >> > > > > implemented as a part of thin client.
> > > > > > > >> > > > >
> > > > > > > >> > > > > Best Regards,
> > > > > > > >> > > > > Ivan Rakov
> > > > > > > >> > > > >
> > > > > > > >> > > > >
> > > > > > > >> > > > > On 14.05.2018 18:58, Dmitriy Govorukhin wrote:
> > > > > > > >> > > > >
> > > > > > > >> > > > >> Hi, Igniters!
> > > > > > > >> > > > >>
> > > > > > > >> > > > >> As far as I know, many people work on a thin client
> > for
> > > > > > > different
> > > > > > > >> > > > language
> > > > > > > >> > > > >> (go,js,php...).
> > > > > > > >> > > > >> Are there any reasons why ignite does not support
> > yaml
> > > or
> > > > > > json
> > > > > > > >> > format
> > > > > > > >> > > > for
> > > > > > > >> > > > >> configuration? or some other popular format?
> > > > > > > >> > > > >> In future, it can help to integrate with thin
> > clients,
> > > > for
> > > > > > > >> example,
> > > > > > > >> > js
> > > > > > > >> > > > >> client may want to dynamic cache start, he passes
> > cache
> > > > > > > >> > configuration
> > > > > > > >> > > > (in
> > > > > > > >> > > > >> native format, for js it will json) through TCP,
> > Ignite
> > > > > node
> > > > > > > >> unwrap
> > > > > > > >> > > and
> > > > > > > >> > > > >> remap to java representation and dynamic start
> cache.
> > > > > > > >> > > > >>
> > > > > > > >> > > > >>
> > > > > > > >> > > > >
> > > > > > > >> > > >
> > > > > > > >> > >
> > > > > > > >> >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Best regards,
  Andrey Kuznetsov.

Unreliable checks in tests for string presence in GridStringLogger contents

[Andrey Kuznetsov]

Folks,

While testing my PR on TeamCity I found a floating test failure. Sometimes
we use {{GridStringLogger}} test facility to capture Ignite node logs and
then check whether some marker strings exist in log output. Since logger's
buffer is limited (32K chars by default) unrelated changes that add more
logging can damage a test.

I propose to add some stuff to {{GridStringLogger}} in order to catch
preset marker strings during logging, then we will not depend on buffer
size.

Thoughts?

-- 
Best regards,
  Andrey Kuznetsov.

Re: Unreliable checks in tests for string presence in GridStringLogger contents

[Andrey Kuznetsov]

Thanks, Vyacheslav.

Created the issue [1] based on your idea.

[1]  https://issues.apache.org/jira/browse/IGNITE-8570


2018-05-23 12:41 GMT+03:00 Vyacheslav Daradur <daradu...@gmail.com>:

> Hi, Andrey, I have faced this problem too.
>
> I'd suggest introducing new logger for tests instead of extending API
> of *GridStringLogger*.
>
> The new logger should be some kind of *listened*, for example with the
> folowing API:
>
> void addListener(String pattern, CountDownLatch latch);
> void addListener(IgniteInClosure lsnr);
>
> This approach reduces memory load in comparison with *GridStringLogger*.
>
> Just for example these should demonstrate my idea, *listened logger* -
> [1], *listener* - [2]:
>
> [1] https://github.com/apache/ignite/blob/master/modules/
> compatibility/src/test/java/org/apache/ignite/compatibility/testframework/
> junits/logger/ListenedGridTestLog4jLogger.java
> [2] https://github.com/apache/ignite/blob/master/modules/
> compatibility/src/test/java/org/apache/ignite/compatibility/testframework/
> junits/IgniteCompatibilityAbstractTest.java#L304
>
>
>
-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #4091: IGNITE-8642: Added thread dumping to FailureProce...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/4091


---

[GitHub] ignite pull request #4173: IGNITE-8562: As single large commit.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4173

IGNITE-8562: As single large commit.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8562-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4173.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4173


commit 1d8b7ce77fb2b5adb1f7a5e8e0ec85ec9c9c9e13
Author: Andrey Kuznetsov 
Date:   2018-06-09T15:34:48Z

IGNITE-8562: As single large commit.




---

[GitHub] ignite pull request #4121: IGNITE-8562: Squashed partial commits.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/4121


---

[GitHub] ignite pull request #3861: IGNITE-8303: Avoided failure handler invocation w...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3861


---

[GitHub] ignite pull request #3652: IGNITE-7770 A bunch of changes to fix tx timeout ...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3652


---

[GitHub] ignite pull request #3843: Ignite 7770 2

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3843


---

[GitHub] ignite pull request #3870: IGNITE-8312: Simplified the choice of exception t...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3870


---

[GitHub] ignite pull request #4173: IGNITE-8562: As single large commit.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/4173


---

[GitHub] ignite pull request #4091: IGNITE-8642: Added thread dumping to FailureProce...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4091

IGNITE-8642: Added thread dumping to FailureProcessor.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8642

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4091.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4091


commit 75353527233477582856b6889b6ea6f36f13984c
Author: Andrey Kuznetsov 
Date:   2018-05-29T18:59:15Z

IGNITE-8642: Added thread dumping to FailureProcessor.




---

[GitHub] ignite pull request #4115: IGNITE-8290: Trying to reproduce on TC.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4115

IGNITE-8290: Trying to reproduce on TC.

Debug through CI.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8290

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4115


commit b8f4a20cee48220e47efaba8433820ed1c150153
Author: Andrey Kuznetsov 
Date:   2018-06-04T11:49:30Z

IGNITE-8290: Trying to reproduce on TC.




---

[GitHub] ignite pull request #4121: IGNITE-8562: Squashed partial commits.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4121

IGNITE-8562: Squashed partial commits.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8562

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4121.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4121


commit cf466344de49c24f249e34e718ba4188beb606b6
Author: Andrey Kuznetsov 
Date:   2018-06-04T13:10:18Z

IGNITE-8562: Squashed partial commits.




---

[GitHub] ignite pull request #4089: Ignite 6587 true

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4089

Ignite 6587 true

For TC tests now

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6587-true

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4089.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4089


commit 3b25cb934838feb3300c7088db4b3840ec7f9ab5
Author: Andrey Kuznetsov 
Date:   2018-05-07T12:43:26Z

IGNITE-6587: Refactored ServerImpl.TcpServer to use GridWorker.

commit 665ba863ceb95b643a805d6920f3f384275dae20
Author: Andrey Kuznetsov 
Date:   2018-05-07T13:06:22Z

IGNITE-6587: Refactored ServerImpl.RingMessageWorker to use GridWorker.

commit b10bcd350efb55de95792b369996ba266b3612a9
Author: Andrey Kuznetsov 
Date:   2018-05-07T15:59:56Z

IGNITE-6587: Refactored WAL manager critical threads to use GridWorker.

commit 3154112c8bab70d1ff4fe20639648e8a9c7da373
Author: Andrey Kuznetsov 
Date:   2018-05-07T18:55:25Z

IGNITE-6587: Minor improvements.

commit a44fc8064a9ba8f42e4faacde018ac9bc1193f01
Author: Andrey Kuznetsov 
Date:   2018-05-08T10:52:01Z

IGNITE-6587: Refactored CommunicationWorker to use GridWorker.

commit 0efc5354f33666ac705c738d75955258084091de
Author: Andrey Kuznetsov 
Date:   2018-05-08T15:25:56Z

IGNITE-6587: (WIP) switching to timed waits/polls, updating heartbeats.

commit 48c889441a119ef2192c1ff61485c20f80659e82
Author: Andrey Kuznetsov 
Date:   2018-05-14T13:38:47Z

IGNITE-6587: Heartbeat updates in critical WAL manager workers.

commit 69059bda2071f7d6af0cef30cf8d77b160fc56be
Author: Andrey Kuznetsov 
Date:   2018-05-14T15:55:51Z

IGNITE-6587: (WiP) More heartbeat updates in critical workers.

commit 6fb2edf1fef15bb187481f89eddb3710d5daadba
Author: Andrey Kuznetsov 
Date:   2018-05-15T13:09:16Z

IGNITE-6587: Heartbeat updates in checkpointer.

commit 9cd8cba1fdc8c21d999df4c334daaef7e226aa0f
Author: Andrey Kuznetsov 
Date:   2018-05-15T13:23:17Z

IGNITE-6587: More heartbeat updates in critical workers.

commit 5245abc0c72812c4746127b25638a28dac7af11a
Author: Andrey Kuznetsov 
Date:   2018-05-16T20:14:31Z

IGNITE-6587: Extended StripedExecutor.Stripe from GridWorker.

commit 2eb448f6a7ff7a6be423742336480a7b98793c2d
Author: Andrey Kuznetsov 
Date:   2018-05-17T11:37:08Z

IGNITE-6587: Heartbeat updates in StripedExecutor workers.

commit 8304fa92879fe293f112d1c7ba515b5bd30cc271
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:33:23Z

Merge branch 'master' into ignite-6587

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java

commit 5051aa4d44e78b5362dd0094bc5a57566eea4855
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:36:23Z

IGNITE-6587: Addition to merge commit.

commit 732d2c25013d94d67188f42b2f96c786eeaf6346
Author: Andrey Kuznetsov 
Date:   2018-05-17T12:42:05Z

IGNITE-6587: Updated WAL FileDecompressors to conform uniform poll style.

commit 0c6f7762f1bc709c1444e545ff6475edc3fd6152
Author: Andrey Kuznetsov 
Date:   2018-05-18T09:54:05Z

IGNITE-6587: Added Checkpointer to worker registry.

commit d64e2f8e97e9d8a339695087ceea1443f8f75141
Author: Andrey Kuznetsov 
Date:   2018-05-18T14:43:22Z

IGNITE-6587: Added Stripe workers to worker registry.

commit fd142e80f80e73d420c0a03923ba3a4783dc7f34
Author: Andrey Kuznetsov 
Date:   2018-05-21T11:40:46Z

IGNITE-6587: Optional reference to worker registry in GridNioServer.

commit de98f8782f1728cc7e4a2852fbd0931cb21d3af9
Author: Andrey Kuznetsov 
Date:   2018-05-21T11:42:11Z

IGNITE-6587: Added worker registry to communication SPI GridNioServer.

commit f24f93abe8b8a3833f556b204c9995d28cd9cf44
Author: Andrey Kuznetsov 
Date:   2018-05-21T13:47:11Z

Merge branch 'master' into ignite-6587

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FileWriteAheadLogManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java

commit e3a7b9aaed79d45b205715b6b281db702e7be95d
Author: Andrey Kuznetsov 
Date:   2018-05-22T12:30:30Z

IGNITE-6587: Recovering after incorrect merge commit.

commit 45c64a1ebaf5dabcfdf64e35a14542c0bddaa4c2
Author: Andrey Kuznetsov 
Date:   2018-05-22T12:56:27Z

IGNITE-6587: Moved WAL timeout constants to proper place.

commit 95f1ce0a9c3002a72fa47c616f928521570d2d40
Author: Andrey Kuznetsov 
Date:   2018-05-22T13:12:36Z

IGNITE-6587: Made WAL FileDecompressors not critical again.

commit

[GitHub] ignite pull request #4266: IGNITE-8885 Failing test method explicitly.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4266

IGNITE-8885 Failing test method explicitly.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8885

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4266.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4266


commit 9071533f192c0847427fb0854b8163bed01d61d3
Author: Andrey Kuznetsov 
Date:   2018-06-27T12:33:26Z

IGNITE-8885 Failing test method explicitly.




---

[GitHub] ignite pull request #4265: IGNITE-8887 Failing test method explicitly.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/4265

IGNITE-8887 Failing test method explicitly.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-8887

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4265.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4265


commit 56b2ffdafbea0856f09fd483062d78bb588ed211
Author: Andrey Kuznetsov 
Date:   2018-06-27T12:27:24Z

IGNITE-8887 Failing test method explicitly.




---

[GitHub] ignite pull request #3302: IGNITE-7312: Made use of java.util.Base64 for bas...

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3302

IGNITE-7312: Made use of java.util.Base64 for base64 encoding.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-7312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3302.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3302


commit 870c6e6b6b76e9b3a15bb8ea006361b18b537edf
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-27T14:54:01Z

IGNITE-7312: Made use of java.util.Base64 for base64 encoding.




---

Re: Java 9 support

[Andrey Kuznetsov]

Hi Petr!

Could you please clarify what is wrong with fix proposed in IGNITE-6736,
and what is supposed to be a replacement for monitorEnter/monitorExit now?

2018-01-09 19:08 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>:

> Hi all.
>
>
> After some thorough research and with help of fellow igniters, I’ve
> managed to prepare more or less stable Java 9 build configuration of Apache
> Ignite.
>
> Here are changes to make it work:
>  - Java 8 profiles and build process revision, made in IGNITE-7203;
>  - Java 9 maven profile prepared in IGNITE-7144 (will be moved to
> IGNITE-6730 as subtask);
>  - specific maven-compiler-plugin configuration with JVM args for Java 9
> profile (as was proposed by Vladimir Ozerov);
>  - maven-bundle-plugin version is updated to 3.5.0;
>  - maven-compiler-plugin version synchronised to 3.7.0 (in Cassandra
> modules);
>  - scala version updated to 2.12.4;
>  - disabled scalar-2.10, spark-2.10 and visor-console-2.10 modules (due to
> dependency in scala 2.10 which is unsupported by Java 9);
>  - sun.misc.JavaNioAccess import changed to jdk.internal.misc.JavaNioAccess
> in GridUnsafe.java and PageMemoryImpl.java;
>  - sun.misc.SharedSecrets import changed to jdk.internal.misc.SharedSecrets
> in GridUnsafe.java and PageMemoryImpl.java;
>  - methods monitorEnter and monitorExit bodies commented out (fix from
> IGNITE-6736 did not work).
>
> I’d like to put these changes into ignite-6730 to have working compiling
> under Java 9 branch — so that we can continue work on improving Apache
> Ignite’s Java 9 support.




-- 
Best regards,
  Andrey Kuznetsov.

Re: Java 9 support

[Andrey Kuznetsov]

Thanks, Petr.

I heard of some activity related to performance consequences of
ReentrantLocks in IGNITE-6736 fix. So, I'd like to get a reviewer feedback
first.

Andrey G., Vladimir O., is it possible to merge the fix to master?

2018-01-10 9:56 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>:

> Andrey — double checked your solution and it works now. Guess there was
> some merge error first time.
> Sorry for misleading.
>
> So, there is working solution for Java 9 build and I’d like to save this
> configuration in ignite-6730 (making IGNITE-7144 and IGNITE-6736 to become
> subtasks in the process).
> What do you think?
>
>
> > On 9 Jan 2018, at 20:49, Andrey Kuznetsov <stku...@gmail.com> wrote:
> >
> > Hi Petr!
> >
> > Could you please clarify what is wrong with fix proposed in IGNITE-6736,
> > and what is supposed to be a replacement for monitorEnter/monitorExit
> now?
> >
> > 2018-01-09 19:08 GMT+03:00 Petr Ivanov <mr.wei...@gmail.com>:
> >
> >> Hi all.
> >>
> >>
> >> After some thorough research and with help of fellow igniters, I’ve
> >> managed to prepare more or less stable Java 9 build configuration of
> Apache
> >> Ignite.
> >>
> >> Here are changes to make it work:
> >> - Java 8 profiles and build process revision, made in IGNITE-7203;
> >> - Java 9 maven profile prepared in IGNITE-7144 (will be moved to
> >> IGNITE-6730 as subtask);
> >> - specific maven-compiler-plugin configuration with JVM args for Java 9
> >> profile (as was proposed by Vladimir Ozerov);
> >> - maven-bundle-plugin version is updated to 3.5.0;
> >> - maven-compiler-plugin version synchronised to 3.7.0 (in Cassandra
> >> modules);
> >> - scala version updated to 2.12.4;
> >> - disabled scalar-2.10, spark-2.10 and visor-console-2.10 modules (due
> to
> >> dependency in scala 2.10 which is unsupported by Java 9);
> >> - sun.misc.JavaNioAccess import changed to jdk.internal.misc.
> JavaNioAccess
> >> in GridUnsafe.java and PageMemoryImpl.java;
> >> - sun.misc.SharedSecrets import changed to jdk.internal.misc.
> SharedSecrets
> >> in GridUnsafe.java and PageMemoryImpl.java;
> >> - methods monitorEnter and monitorExit bodies commented out (fix from
> >> IGNITE-6736 did not work).
> >>
> >> I’d like to put these changes into ignite-6730 to have working compiling
> >> under Java 9 branch — so that we can continue work on improving Apache
> >> Ignite’s Java 9 support.
> >
> >
> >
> >
> > --
> > Best regards,
> >  Andrey Kuznetsov.
>
>


-- 
Best regards,
  Andrey Kuznetsov.

Re: Java 9 support

[Andrey Kuznetsov]

No. We got rid of monitorEnter/monitorExit there. And ReentrantLock is
supposed to be a replacement.

2018-01-10 13:49 GMT+03:00 Антон Чураев <churaev...@gmail.com>:

> Andrey, do You mean https://issues.apache.org/jira/browse/IGNITE-4908?
>
>
> --
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3377: IGNITE-7386: Got rid of ThreadLocalRandom8.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3377

IGNITE-7386: Got rid of ThreadLocalRandom8.

For test purposes.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-7386

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3377.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3377


commit bfa217419c72623e3ed4a9f4c0ff2f9bcb7db962
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-01-15T14:16:24Z

IGNITE-7386: Got rid of ThreadLocalRandom8.




---

[GitHub] ignite pull request #3181: Ignite 6902

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3181


---

Eliminating org.jsr166 from Ignite codebase

[Andrey Kuznetsov]

Hi Igniters!

I'm revising org.jsr166 package contents now. Ideally, we should get rid of
it completely. As far as I see some classes can be transparently replaced
by Java8 equivalents, some other can reuse standard Java8 versions, and
some can't be removed at all. I'd like to discuss this.

1. LongAdder8, ThreadLocalAdder8, ConcurrentHashMap8 [1], [2]:
These can be simply replaced by their j.u.c equivalents.

2. ConcurrentLinkedHashMap [3]:
Originally this was a mix of two JSR166 classes: ConcurrentHashMapV8 and
ConcurrentHashMap, as they looked like at some time between late 2011 and
mid-2012, but before CHMV8 was merged into CHM in August, 2012. Then
ConcurrentLinkedDeque8 was added to stitch the structure with a list. Later
there were made some significant performance improvements to Segment
(partial hash table) inner class, BTW Segments were removed from CHM
implementation upon CHMV8-to-CHM merge. Taking this story into account I'd
like to leave this extremely customized class unchanged.

3. ConcurrentLinkedDeque8 [4]:
This one is very similar to standard ConcurrentLinkedDeque, but has a
couple of differences. First, LongAdder was attached to provide fast size()
method. Second, queue Node was exposed as public declaration. Latter
feature is used extensively in EvictionPolicy implementations. As for me,
the only benefit of this is CAS logic in unlinkx() method. And this logic
relates to concrete data item being stored in a queue, but not to queue
itself. So one can hide Node back and add AtomicBoolean (as one of the
possible options) to queue item along with EvictableEntry. Ultimately, we
can extend standard ConcurrentLinkedDeque by adding faster size() and get
rid of unlinkx() and other ***x() stuff. Yet another consumer of unlinkx()
method is MemoryEventStorageSpi, one can fix it the same way as eviction
policies, however only logging is affected there, so I'm unsure if we need
to fix it at all.

I'll appreciate your thoughts on the subject.


[1] https://issues.apache.org/jira/browse/IGNITE-7518

[2] https://issues.apache.org/jira/browse/IGNITE-7513

[3] https://issues.apache.org/jira/browse/IGNITE-7516

[4] https://issues.apache.org/jira/browse/IGNITE-7517


--
Best regards,
  Andrey Kuznetsov.

Re: IgniteSet implementation: changes required

[Andrey Kuznetsov]

Hi all,

Current set implementation has significant flaw: all set data are
duplicated in onheap maps on _every_ node in order to make iterator() and
size(). For me it looks like simple yet ineffective implementation.
Currently, these maps are damaged by checkpointing/recovery, and we could
patch them somehow. Another future change to Ignite caches can damage them
again. This looks fragile when datastructure is not entirely backed by
caches. Pavel's proposal seems to be a reliable solution for non-collocated
sets.

9 февр. 2018 г. 22:46 пользователь "Valentin Kulichenko" <
valentin.kuliche...@gmail.com> написал:

Pavel,

I'm a bit confused. In my understanding, issue exists because we have local
in-memory maps which are used as the main source of truth about which
structures currently exist. During restart, we lose all this data even if
data structures cache(s) are persisted. Once we fix this, issue goes away,
regardless of weather we store data structure per cache or everything in
single cache. Am I missing something?

I also agree with Dmitry. While cache per set approach can make sense for
non-collocated sets, for collocated ones it definitely doesn't. So I would
fix the original issue first, and then change the architecture if it's
really needed.

-Val

Re: IgniteSet implementation: changes required

[Andrey Kuznetsov]

Indeed, all sets, regardless of whether they collocated or not, share
single cache, and also use onheap data structures irresistable to
checkpointing/recovery.

2018-02-13 2:14 GMT+03:00 Dmitriy Setrakyan <dsetrak...@apache.org>:

> On Fri, Feb 9, 2018 at 6:26 PM, Andrey Kuznetsov <stku...@gmail.com>
> wrote:
>
> > Hi all,
> >
> > Current set implementation has significant flaw: all set data are
> > duplicated in onheap maps on _every_ node in order to make iterator() and
> > size(). For me it looks like simple yet ineffective implementation.
> > Currently, these maps are damaged by checkpointing/recovery, and we could
> > patch them somehow. Another future change to Ignite caches can damage
> them
> > again. This looks fragile when datastructure is not entirely backed by
> > caches. Pavel's proposal seems to be a reliable solution for
> non-collocated
> > sets.
> >
>
> I would agree, but I was under an impression that non-collocated sets are
> already implemented this way. Am I wrong?
>



-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3525: Ignite 7517

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3525

Ignite 7517



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-7517

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3525.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3525


commit d970629c7f527e958ffb4c2e46b3917579c668db
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-02-12T16:05:37Z

IGNITE-7517: WIP.

commit dfde3d9c89f1969701f86233a1d794b379db2cc1
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-02-14T08:58:57Z

IGNITE-7517: WIP.

commit e3b5cf0c3a64a17193f1750e57c32eca1e0eccf1
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-02-14T16:09:34Z

IGNITE-7517: Removed ConcurrentLinkedDeque8.

commit 4d956e89489515776d535b5ae5c055fe002c45a0
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-02-14T16:41:37Z

Merge branch 'master' into ignite-7517

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/igfs/common/IgfsLogger.java
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/communication/GridIoManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/deployment/GridDeploymentLocalStore.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridCacheMvccManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/GridDeferredAckMessageSender.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/distributed/dht/GridDhtLocalPartition.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/transactions/IgniteTxManager.java
#   
modules/core/src/main/java/org/apache/ignite/spi/collision/jobstealing/JobStealingCollisionSpi.java
#   
modules/hadoop/src/main/java/org/apache/ignite/internal/processors/hadoop/taskexecutor/external/HadoopExternalTaskExecutor.java




---

EvictableEntry.isCached() state meaning

[Andrey Kuznetsov]

Hi Igniters!

I can't comprehend the meaning of the following note in javadoc for
EvictableEntry.isCached() method.

"If entry is not in cache (e.g. has been removed) {@code false} is
returned. In this case all operations on this entry will cause creation of
a new entry in cache."

That is, if I call getKey() or getValue() on removed entry it will be
resurrected in cache? This sounds too magically. Could someone explain the
real message of the phrase, please?

--
Best regards,
  Andrey Kuznetsov.

Re: Stop nodes after test by default - IGNITE-6842

[Andrey Kuznetsov]

Hi Maxim,

Regarding your first question, the use of afterTestsStopped is not enough
to stop all nodes, since each individual test (method) can start custom set
of notes during its operation, and this very test should stop all those
nodes. If you delegate closing to afterTestsStopped this will affect only
last test (method). Nodes that survive between tests can affect successive
tests.

2018-02-07 1:10 GMT+03:00 Maxim Muzafarov <maxmu...@gmail.com>:

> Hello,
>
> I've made some research about this problem and i think that in general we
> should move stopAllGrids method in GridAbstractTest class to
> afterTestsStopped method with some changes. Am I right?
>
> Also, I have a question about stopAllGrids(boolean cancel) this "cancel"
> argument. Why in some cases we should interrupt ComputeJob and in some
> cases shouldn't? For example here:
> IgniteBaselineAffinityTopologyActivationTest#afterTest
> we call method stopAllGrids(false) this way. Why not "true" argument
> instead?
>
>
> --
Best regards,
  Andrey Kuznetsov.

Re: Abandoned Patch Available JIRA tickets

[Andrey Kuznetsov]

Periodic punches can frustrate committers, and also it's unpleasant for
contributors. Most IP->PA transitions are commented with something like
"John Doe, this awesome feature is ready and needs your review." Maybe it's
better to adopt following rule of thumb? If the change is clean and
straightforward it should be reviewed is a day, otherwise committer puts a
comment in Jira issue about planned review time.

Is this acceptable?

2018-02-07 0:14 GMT+03:00 Denis Magda <dma...@apache.org>:

> I guess it’s all about discipline.
>
> Committers need to walk-through a list of the pull-request regularly while
> contributors have to remind of a pending pull-request periodically. So both
> parts have to be proactive.
>
> Another approach is to find a volunteer from the community who will keep
> an eye on the contributions and spread them out among committers.
>
> Not sure I like the latter approach and would rather go for the one when
> both the committers and contributors are proactive and disciplined. But
> guess what, if you want to make the contributors proactive then the
> committers have to be an example.
>
> —
> Denis
>
>
>


-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3165: IGNITE-6711: TotalAllocatedPages metric fix.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3165


---

[GitHub] ignite pull request #3428: Ignite 7513

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3428


---

[GitHub] ignite pull request #3377: IGNITE-7386: Got rid of ThreadLocalRandom8.

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3377


---

[GitHub] ignite pull request #2339: Ignite 4181 public api

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2339


---

[GitHub] ignite pull request #3197: Ignite 6734

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3197


---

[GitHub] ignite pull request #3248: IGNITE-6736: Switched GridCacheMapEntry synchroni...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3248


---

[GitHub] ignite pull request #3302: IGNITE-7312: Made use of java.util.Base64 for bas...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3302


---

[GitHub] ignite pull request #3531: IGNITE-7685: Fixed allocation rate.

[andrey-kuznetsov]

GitHub user andrey-kuznetsov opened a pull request:

https://github.com/apache/ignite/pull/3531

IGNITE-7685: Fixed allocation rate.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-7685

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3531.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3531


commit 5c575b187ee2ac07dc34017a26ec37425306108c
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2018-02-15T17:37:31Z

IGNITE-7685: Fixed allocation rate.




---

[GitHub] ignite pull request #3197: Ignite 6734

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3197


---

[GitHub] ignite pull request #3206: IGNITE-6743: Cleaning direct buffer through refle...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3206


---

[GitHub] ignite pull request #3197: Ignite 6734

[andrey-kuznetsov]

GitHub user andrey-kuznetsov reopened a pull request:

https://github.com/apache/ignite/pull/3197

Ignite 6734



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/andrey-kuznetsov/ignite ignite-6734

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3197


commit aff7c61f66ed0ba8de051f8ab037aa8f3076f167
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-11T12:04:45Z

IGNITE-6734: Switched language level to 8 for entire project.

commit cac8bcc74fc2075a75a74c445757ccce9f86c4f2
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-11T12:08:34Z

IGNITE-6734: Replaced sun.misc.BASE64Encoder with standard Java 8 encoder.

commit 9e310825570314afda08cc4d2b2996cdd7aee15d
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-26T13:38:03Z

Merge branch 'master' into ignite-6734

commit 5c7913b685eaf659dccec68a20c704d7f2c81ac2
Author: Andrey Kuznetsov <stkuzma@...>
Date:   2017-12-26T14:08:09Z

IGNITE-6734: Changed to reflective Base64 encoding.




---

[GitHub] ignite pull request #2748: IGNITE-6015: Rollback on exception in commitIfLoc...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/2748


---

Re: Time and scope for Apache Ignite 2.4

[Andrey Kuznetsov]

Thanks, Denis!

I hope I'll finish the task on memory metrics in a couple of days.

2017-12-20 23:28 GMT+03:00 Denis Magda <dma...@apache.org>:

> Andrey K., excellent progress! Do you think you will finish the task by
> the end of the year?
>
> Furthermore, I see you are working on Java 9 specific tasks:
> https://issues.apache.org/jira/browse/IGNITE-6728 <
> https://issues.apache.org/jira/browse/IGNITE-6728>
>
> Is everything alright there? If you need a review try to ping a reviewer
> directly.
>
> —
> Denis
>
> > On Dec 20, 2017, at 3:29 AM, Andrey Kuznetsov <stku...@gmail.com> wrote:
> >
> > Hi Denis,
> >
> > As for memory metrics, I'm fixing minor bugs now, and then we'll need
> some
> > time for review.
> >
> > 2017-12-19 23:07 GMT+03:00 Denis Magda <dma...@apache.org>:
> >
> >> Igniters,
> >>
> >> It’s time to sync up on the progress of IMPORTANT tasks for 2.4:
> >> https://cwiki.apache.org/confluence/display/IGNITE/Apache+Ignite+2.4#
> >> ApacheIgnite2.4-RequiredTickets <https://cwiki.apache.org/
> >> confluence/display/IGNITE/Apache+Ignite+2.4#ApacheIgnite2.4-
> >> RequiredTickets>
> >>
> >> How good are we if to assume 5th of Jan as a code freeze date? Please
> >> comment:
> >> - Nikolay Izhikov: Data Frames.
> >> - Sergey Kalashnikov, Vladimir Ozerov: DDL and SQL features from the
> list.
> >> - Andrey Kuznetsov, Alexandr Fedotov: memory metrics tasks.
> >> - Yuri: Machine Learning + neural networks
> >> - Vladimir Ozerov: Java 9 and Java 7
> >> - Sergey Kozlov: SQL scope testing
> >> - etc.: please check the link with the features.
> >>
> >> —
> >> Denis
> >>
> >> --
> > Best regards,
> >  Andrey Kuznetsov.
>
>


-- 
Best regards,
  Andrey Kuznetsov.

[GitHub] ignite pull request #3199: IGNITE-6740: Replaced direct sun.nio.ch.DirectBuf...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3199


---

[GitHub] ignite pull request #3184: IGNITE-6729: Removed obsolete reflection-based wo...

[andrey-kuznetsov]

Github user andrey-kuznetsov closed the pull request at:

https://github.com/apache/ignite/pull/3184


---

Re: IEP-22: Direct Data Load proposal

[Andrey Kuznetsov]

Vladimir,

Great IEP, but I couldn't comprehend the beginning of the "Direct Data
Load" paragraph. Maybe, there are some typos?

ср, 20 июн. 2018 г. в 18:08, Vladimir Ozerov :

> Igniters,
>
> Initial data load is one of the most important use cases for our product.
> This is one the first things user try to do with Ignite. And if it takes
> too much time, it is very likely that user will look for other solutions.
>

Best regards,
  Andrey Kuznetsov.