Re: PHP/Python versions for Thin Clients

[Dmitriy Setrakyan]

Thanks Pavel, this makes sense now.

On Tue, May 15, 2018 at 5:30 AM, Pavel Petroshenko 
wrote:

> Hi Dmitriy,
>
> PHP 5.6 and 7.0 are going to be end-of-life shortly [1]. So the minimal
> version for the Thin Client is going to be either 7.1 or 7.2 (I would
> finalize this along with the PHP Thin Client API proposal).
>
> As for Python, there is still some legacy code on 2.7, the oldest active
> 2.x version. However the use of Python 2 is declining as it’s not actively
> developed, doesn’t get new features, and its maintenance is going to be
> stopped in 2020 [2]. Python 3 is a strong leader with 75% and Python 2 is
> used as the main interpreter by only 25% (rapidly declining) [3]. So I'm
> leaning towards supporting 3.4+ (the oldest active 3.x version). However, I
> would keep the 2.7 in mind for API design.
>
> I hope it makes sense.
>
> Thanks,
> p.
>
> [1] http://php.net/supported-versions.php
> [2] https://legacy.python.org/dev/peps/pep-0373/
> [3] https://www.jetbrains.com/research/python-developers-survey-2017/
>
>
> On Sat, May 12, 2018 at 6:21 AM, Dmitriy Setrakyan 
> wrote:
>
> > Pavel,
> >
> > Can you suggest what would be the advantages and disadvantages of
> > supporting different versions?
> >
> > D.
> >
> > On Sat, May 12, 2018 at 7:21 AM, Pavel Petroshenko <
> pa...@petroshenko.com>
> > wrote:
> >
> > > Igniters,
> > >
> > > Are there any strong opinions on which language versions should the
> Thin
> > > Clients written in Python and PHP support? Any objections to using PHP
> > 7.1+
> > > and Python 3.5+?
> > >
> > > Thanks,
> > >
> > > p.
> > >
> >
>

[jira] [Created] (IGNITE-8489) Web Console: Landing page do not show images under Retina display

[Alexey Kuznetsov (JIRA)]

Alexey Kuznetsov created IGNITE-8489:


 Summary: Web Console: Landing page do not show images under Retina 
display
 Key: IGNITE-8489
 URL: https://issues.apache.org/jira/browse/IGNITE-8489
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Reporter: Alexey Kuznetsov
Assignee: Andrey Novikov
 Fix For: 2.6






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8488) Web console: scroll bar doesn't work inside cluster selector component

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-8488:
--

 Summary: Web console: scroll bar doesn't work inside cluster 
selector component
 Key: IGNITE-8488
 URL: https://issues.apache.org/jira/browse/IGNITE-8488
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
Assignee: Ilya Borisov


List of clusters can be scrolled by mouse wheel but can't by dragging the 
scrollbar.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: PHP/Python versions for Thin Clients

[Pavel Petroshenko]

Hi Dmitriy,

PHP 5.6 and 7.0 are going to be end-of-life shortly [1]. So the minimal
version for the Thin Client is going to be either 7.1 or 7.2 (I would
finalize this along with the PHP Thin Client API proposal).

As for Python, there is still some legacy code on 2.7, the oldest active
2.x version. However the use of Python 2 is declining as it’s not actively
developed, doesn’t get new features, and its maintenance is going to be
stopped in 2020 [2]. Python 3 is a strong leader with 75% and Python 2 is
used as the main interpreter by only 25% (rapidly declining) [3]. So I'm
leaning towards supporting 3.4+ (the oldest active 3.x version). However, I
would keep the 2.7 in mind for API design.

I hope it makes sense.

Thanks,
p.

[1] http://php.net/supported-versions.php
[2] https://legacy.python.org/dev/peps/pep-0373/
[3] https://www.jetbrains.com/research/python-developers-survey-2017/


On Sat, May 12, 2018 at 6:21 AM, Dmitriy Setrakyan 
wrote:

> Pavel,
>
> Can you suggest what would be the advantages and disadvantages of
> supporting different versions?
>
> D.
>
> On Sat, May 12, 2018 at 7:21 AM, Pavel Petroshenko 
> wrote:
>
> > Igniters,
> >
> > Are there any strong opinions on which language versions should the Thin
> > Clients written in Python and PHP support? Any objections to using PHP
> 7.1+
> > and Python 3.5+?
> >
> > Thanks,
> >
> > p.
> >
>

Null check removal in IGNITE-5779

[Sunny Chan, CLSA]

Hello,


In Ignite-5779 patch, CassandraSessionImpl.java line 289 a null check for row 
has been removed (before the change: 
https://github.com/apache/ignite/blob/924b1faa64026107bf933ba441e743cf52cb94d1/modules/cassandra/store/src/main/java/org/apache/ignite/cache/store/cassandra/session/CassandraSessionImpl.java#L289
)

I cannot tell immediately why the null check is no longer needed, especially 
the row assignment in line 289 can definitely return a null. Is it a mistake or 
we are expecting BatchExecutionAssistant is able to deal with null?

Thanks.

Sunny Chan
Senior Lead Engineer, Executive Services
D  +852 2600 8907  |  M  +852 6386 1835  |  T  +852 2600 
5/F, One Island East, 18 Westlands Road, Island East, Hong Kong

[:1. Social Media Icons:CLSA_Social Media 
Icons_linkedin.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_twitter.png][:1. Social Media 
Icons:CLSA_Social Media 
Icons_youtube.png][:1.
 Social Media Icons:CLSA_Social Media 
Icons_facebook.png]

clsa.com
Insights. Liquidity. Capital.

[CLSA_RGB]

A CITIC Securities Company

The content of this communication is intended for the recipient and is subject 
to CLSA Legal and Regulatory Notices.
These can be viewed at https://www.clsa.com/disclaimer.html or sent to you upon 
request.
Please consider before printing. CLSA is ISO14001 certified and committed to 
reducing its impact on the environment.

Re: Thin clients features wiki page

[Denis Magda]

Igor, thanks for putting together the page.

As for the immediate improvements, I'm backing up Alexey's suggestions.

This page deserves to be moved to the readme.io. We'll do that once it's
sorted out how to group the thin clients' documentation.

--
Denis

On Mon, May 14, 2018 at 12:39 PM, Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Hi Igor,
>
> "Get Binary Cache" (if it is about getting a complex object in a "binary"
> form, w/o deserialization) - is supported by NodeJS:
> BinaryObject is returned for a complex object if an app has not specified
> another JavaScript Object for deserialization.
>
> Maybe worth to add type registration functionality:
> - complex object type registration (supported by NodeJS)
> - enum type registration (not supported by NodeJS)
> - type name registration (not supported by NodeJS)
>
> Also, maybe add a list of all type codes, supported for read and write.
> But not sure, as all of them are supported by all three clients.
>
> Thanks,
> -Alexey
>
> 14.05.2018 22:06, Igor Sapego пишет:
>
> Hello Igniters,
>>
>> I've created a new page in wiki, that summarizes a features
>> availability of thin clients. It should help us to plan development
>> of thin clients while also helping out users to understand a state
>> of clients.
>>
>> Check it out and tell me what you think. And if there are some
>> mistakes, don't hesitate telling me.
>>
>> [1] -
>> https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features
>>
>> Best Regards,
>> Igor
>>
>>

Re: Suggestion about Docker Swarm Services and Scaling Ignite automatically

[Denis Magda]

Jonathan,

Thanks for the extensive summary.

Personally, I like the manager node approach because it's built in Docker
Swarm and doesn't require 3rd party dependencies. Moreover, it reminds me
the way we supported Kubernetes where Ignite pods request IP addresses from
a Kubernetes master.

However, you mentioned that "it's still a limitation of docker". What is
your concern about that approach?

--
Denis

On Tue, May 8, 2018 at 2:05 AM, Jonathan Schoreels <
jonathan.schore...@gmail.com> wrote:

> Hi Denis,
>
> It seems like one workaround would be to configure the URL to one of the
> manager node, and then the Docker API allows to loop over all the nodes net
> interfaces :
> https://github.com/bitsofinfo/docker-discovery-swarm-service#status. The
> problem is it needs to know which node is a manager and which is a worker.
>
> However, that's still a limitation of docker. After some searching on
> "Docker Swarm Peer Discovery", I've found another implementation of the
> similar problem, but for rabbitmq, which also relies on a third party
> service discovery : Consul.
> Here : https://github.com/rabbitmq/rabbitmq-peer-discovery-consul
> One integration with ignite can be found here :
> https://github.com/andrea-zanetti/ignite-consul/blob/
> master/src/main/java/org/apache/ignite/spi/discovery/tcp/ipfinder/consul/
> TcpDiscoveryConsulIpFinder.java
>
> One other interesting possibility would be to use a dns lookup based on
> "dig
>  short" response. This implementation has been made for
> example
> here for zookeeper :
> https://github.com/itsaur/zookeeper-docker-swarm/blob/master/docker-swarm-
> entrypoint.sh.
> Indeed, Docker Swarm updates a DNS with all the containers loadbalanced
> behind. We could cycle through those information and get all the nodes IP.
>
> I've personnally tested it with scaling, and the new IP addresses are
> dynamically added :
>
> bash-4.4# dig tasks.zookeeper1 +short
> 10.0.0.12
>
> docker service scale stack_zookeeper1=2
> stack_zookeeper1 scaled to 2
> overall progress: 2 out of 2 tasks
> 1/2: running   [==>]
> 2/2: running   [==>]
> verify: Service converged
>
> bash-4.4# dig tasks.zookeeper1 +short
> 10.0.0.12
> 10.0.0.17
>
> Any thoughts ? Isn't it a bit too low level ?
>
> Jonathan
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

Re: tracking SQL queries in 2.5

[Dmitriy Setrakyan]

On Mon, May 14, 2018 at 3:55 PM, Vladimir Ozerov 
wrote:

> Hi Dima,
>
> This is not that simple unfortunately. We need to build the whole
> infrastructure for cache-less SQL queries. This is not very complex, but
> require considerable efforts. Let's do that in AI 2.6 scope.
>

Got it, thanks!

[GitHub] ignite pull request #3996: -

[daradurvs]

GitHub user daradurvs opened a pull request:

https://github.com/apache/ignite/pull/3996

-



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/daradurvs/ignite ignite-8381

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3996.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3996


commit fc63a85c3eca061cfdc836d380a0b886e494dc22
Author: Vyacheslav Daradur 
Date:   2018-05-14T21:19:08Z

ignite-8381: workaround

commit bdf1a78d45af48f2ac04917d02ce377b838b4a2c
Author: Vyacheslav Daradur 
Date:   2018-05-14T21:29:48Z

StubTestSuite to test




---

Re: stopAllGrids() used by default and further steps

[Nikolay Izhikov]

Yes.

В Пн, 14/05/2018 в 20:48 +0300, Dmitry Pavlov пишет:
> Hi Nikolay,
> 
> Would you have a minute to finalize this review?
> 
> Sincerely,
> Dmitriy Pavlov
> 
> пт, 4 мая 2018 г. в 16:05, Maxim Muzafarov :
> > Dmitry,
> > 
> > Task of migration to JUnit 4/5 sounds very interesting for me, but I'm not
> > sure that I will have time for it in the next few weeks. Anyway let's
> > create new task to it e.g. "providing design and analisys for migration to
> > JUnit 4/5". I'll try to help with it!
> > 
> > 
> > Test cases IgniteUidAsConsistentIdMigrationTest and
> > TxRollbackAsyncNearCacheTest
> > are not affected by my change.
> > Nevertheless, I've rerun Run::All for this PR.
> > 
> > 
> > All other preparations have already been done for this issue:
> > 
> > PR: https://github.com/apache/ignite/pull/3844
> > TC:
> > https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll=buildTypeStatusDiv_IgniteTests24Java8=pull%2F3844%2Fhead
> > Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-581
> > JIRA: https://issues.apache.org/jira/browse/IGNITE-8266
> > 
> > Will you or others have to to review it?
> > 
> > 
> > 
> > пт, 4 мая 2018 г. в 14:24, Dmitry Pavlov :
> > 
> > > Hi Maxim,
> > >
> > > I think next step can be creation of Junit4/5 IgniteAbstractTest and/or
> > > IgniteTestRunner. Would you like to contribute this prototype?
> > >
> > > Regarding TC run there is a number of suspicious tests
> > >   (e.g. IgnitePdsNativeIoTestSuite2:
> > >
> > > IgniteUidAsConsistentIdMigrationTest.testNewStyleAlwaysSmallestNodeIndexIsCreatedMultithreaded
> > > (fail rate 0,0%) & IgniteCacheTestSuite6:
> > > TxRollbackAsyncNearCacheTest.testSynchronousRollback (fail rate 0,0%)  )
> > > and a number of timeouts ocurred, so I've retriggered re-run for failed
> > > tests -
> > >
> > > https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8_IgniteTests24Java8=pull%2F3844%2Fhead
> > >
> > > For IGNITE-8266   could
> > > you please create CR?
> > >
> > > Sincerely,
> > > Dmitriy Pavlov
> > >
> > >
> > > пт, 4 мая 2018 г. в 12:27, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > > Recenly, we've pached Ingite testing framework to stop all started
> > > > instances after all test-cases completion by default.
> > > > Details of impelemtaion can be viewed here [1]. This change leads us to 
> > > > a
> > > > lot of boilerplate code.
> > > >
> > > >
> > > > 1) I've created issue [2] and prepared PR [3] which removes all this
> > > > boilerplate code. Most of these changes is about removing:
> > > > ```
> > > > @Override protected void afterTestsStopped() throws Exception {
> > > > super.afterTestsStopped();
> > > > stopAllGrids();
> > > > }
> > > > ```
> > > > All tests looks good here. I've double cheched all requiremets and now I
> > > > need your help with review and futher steps up to merge.
> > > >
> > > > Can anyone help me?
> > > >
> > > >
> > > > 2) I've created issue [4] and planning to clean rarely used methods
> > > related
> > > > to stopAllGrids().
> > > > E.g. stopAllClients and stopAllServers methods from GridAbstactTest used
> > > > only once in whole project but they locates in the root class.
> > > > From my point of view, this will simplify for futher migration Ignite
> > > > project to JUnit 4/5 framework.
> > > >
> > > > What else can be done here?
> > > > Please, share your thoughts.
> > > >
> > > >
> > > >
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-6842
> > > > [2] https://issues.apache.org/jira/browse/IGNITE-8266
> > > > [3] https://github.com/apache/ignite/pull/3844
> > > > [4] https://issues.apache.org/jira/browse/IGNITE-8157
> > > >
> > >

signature.asc
Description: This is a digitally signed message part

Thin clients features wiki page

[Igor Sapego]

Hello Igniters,

I've created a new page in wiki, that summarizes a features
availability of thin clients. It should help us to plan development
of thin clients while also helping out users to understand a state
of clients.

Check it out and tell me what you think. And if there are some
mistakes, don't hesitate telling me.

[1] -
https://cwiki.apache.org/confluence/display/IGNITE/Thin+clients+features

Best Regards,
Igor

Re: NodeJS thin client: full API

[Denis Magda]

Alexey,

If a lib is contributed to Ignite (accepted to its main sources like the
node.js thin client), then the documentation has to be in a single place
which is readme.io for now.
Having the docs both on readme.io and in sources is confusing and harder to
maintain.

If a lib doesn't not a part of Ignite distribution (like the
beforementioned Go or Python clients), then its doc is located in some
other place.

--
Denis


On Sat, May 12, 2018 at 2:15 AM, Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Denis,
>
> OK for the legacy docs.
>
> Agree for the core docs (specs, common getting started, etc.).
>
> But who/what does mandate that for totally all Ignite related docs?
>
> Why do not follow an approach which many technologies/products follow? -
> maintain a centralized list of references to different client libs/SDKs
> (sometimes even alternative libs for the same language/platform).
> It is possible to mention a "status" of every lib - is it "verified"
> (fully tested, fully documented, etc.) or just a prototype...
> And it is not mandated that every lib must be developed/released under
> Apache...
>
> The list could already have references to:
> - java client
> - .net client
> - node.js client (as early access)
> - Go client by Aleksandr S. (as prototype)
> - Python client by Sergey K. (as prototype)
>
> With this approach the docs naturally come together with a concrete
> client, but may be cross-referenced on readme.io for some clients as
> well...
>
> -Alexey
>
> 12.05.2018 6:37, Denis Magda пишет:
>
> Alexey,
>>
>> Presently, Ignite hosts all the docs in readme.io without exception. It
>> means that once your contribution is accepted by the community the Node.JS
>> docs should be placed on readme.io.
>>
>> You're right saying that we're planning to migrate from readme.io to
>> another documentation engine that would allow us storing doc sources in
>> Ignite repo. It might happen by 2.6 or might take longer.
>>
>> Thus, we need to host the Node.JS docs on readme.io and edit them there
>> once your pull-request is merged (it means there wouldn't be docs' copy
>> added to Ignite repo for now). It's easy to move the docs to readme.io
>> which understands the standard markdown. I'll ask Prachi to assist here.
>>
>> --
>> Denis
>>
>> On Fri, May 11, 2018 at 1:45 PM, Alexey Kosenchuk <
>> alexey.kosenc...@nobitlost.com> wrote:
>>
>> Denis,
>>>
>>> As for the docs, are you ready to bring them to readme.io? Just let me

>>> know
>>>
 and I'll be happy to arrange an account for you and discuss the

>>> structure.
>>>
>>> I remember some discussion regarding moving the docs from readme.io to
>>> GitHub pages in 2.6.
>>> No?
>>>
>>> In any case, in my opinion, a readme near the code is a right primary
>>> place for the docs for thin clients.
>>> Is there any script/automation to convert .md to readme.io?
>>> Or maybe just place a link from the readme.io to the repo readme?
>>> Manual support of the same docs in two places seems not an effective
>>> solution.
>>>
>>> The docs for NodeJS client is ready for review in the repo.
>>> The links and the installation procedure will have to be updated when the
>>> client is integrated into the apache repo and released on npmjs.
>>>
>>> -Alexey
>>>
>>>
>>

Re: Tests coverage report of Ignite 2.5 core module

[Александр Меньшиков]

Hi Dmitry,
You are right this is not a sorting, this is a ranking by two parameters
where both of them equal in priority.
For such task in math, we have an idea of "dominating subset".
It simply too understand by example:

  P1, P2
E1  2, 2
E2  1, 2
E3  2, 1
E4  1, 1

Here element E1 dominate all other elements. E2 and E3 dominate E4, and E4
is a loser.
So we can do a ranking like that: E1 has rank 1, E2 and E3 have rank 2, and
E4 has rank 3.

I have done the same with JoCoCo results.
Where less coverage is better, and greate complexity is better (because we
are looking for classes with the problems).
In my list, there are all classes with rank 1 (except well covered, small,
and with a strange result of 0% coverage).

Finally, I have sorted this list by one value "coverage/complexity" -- just
for better viewing.

I have written R script for this, and if you think about some other
parameters or filtering -- let me know.


2018-05-14 20:54 GMT+03:00 Dmitry Pavlov :

> Hi Alexander,
>
> I did not quite understand how this list of top tests was obtained. It does
> not look like sorted.
>
> Can you clarify?
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 14 мая 2018 г. в 20:48, Александр Меньшиков :
>
> > Vyacheslav,
> > I have made a simple ranking for your data for highlighting classes which
> > more than other needs testing.
> >
> > Please take a look at the list which I have done (36 classes):
> >
> > https://drive.google.com/file/d/1RJrtnN77MeEWLTShFzqaLInCdOSb9
> WdC/view?usp=sharing
> >
> >
> > Below you can read how did it in details:
> >
> > First I have removed all small classes (less than 200 lines), classes
> with
> > more than 80% coverage, and with 0% (too strange result).
> > After that, I have extracted subset which dominates other classes by two
> > parameters: coverage and complexity.
> > It means all other classes have not less coverage and not greater
> > complexity, and also better at least at one of these parameters.
> >
> > Finally, I have sorted this list by value "coverage/complexity".
> >
>

Re: Tests coverage report of Ignite 2.5 core module

[Dmitry Pavlov]

Hi Alexander,

I did not quite understand how this list of top tests was obtained. It does
not look like sorted.

Can you clarify?

Sincerely,
Dmitriy Pavlov

пн, 14 мая 2018 г. в 20:48, Александр Меньшиков :

> Vyacheslav,
> I have made a simple ranking for your data for highlighting classes which
> more than other needs testing.
>
> Please take a look at the list which I have done (36 classes):
>
> https://drive.google.com/file/d/1RJrtnN77MeEWLTShFzqaLInCdOSb9WdC/view?usp=sharing
>
>
> Below you can read how did it in details:
>
> First I have removed all small classes (less than 200 lines), classes with
> more than 80% coverage, and with 0% (too strange result).
> After that, I have extracted subset which dominates other classes by two
> parameters: coverage and complexity.
> It means all other classes have not less coverage and not greater
> complexity, and also better at least at one of these parameters.
>
> Finally, I have sorted this list by value "coverage/complexity".
>

Re: Tests coverage report of Ignite 2.5 core module

[Александр Меньшиков]

Vyacheslav,
I have made a simple ranking for your data for highlighting classes which
more than other needs testing.

Please take a look at the list which I have done (36 classes):
https://drive.google.com/file/d/1RJrtnN77MeEWLTShFzqaLInCdOSb9WdC/view?usp=sharing


Below you can read how did it in details:

First I have removed all small classes (less than 200 lines), classes with
more than 80% coverage, and with 0% (too strange result).
After that, I have extracted subset which dominates other classes by two
parameters: coverage and complexity.
It means all other classes have not less coverage and not greater
complexity, and also better at least at one of these parameters.

Finally, I have sorted this list by value "coverage/complexity".

Re: stopAllGrids() used by default and further steps

[Dmitry Pavlov]

Hi Nikolay,

Would you have a minute to finalize this review?

Sincerely,
Dmitriy Pavlov

пт, 4 мая 2018 г. в 16:05, Maxim Muzafarov :

> Dmitry,
>
> Task of migration to JUnit 4/5 sounds very interesting for me, but I'm not
> sure that I will have time for it in the next few weeks. Anyway let's
> create new task to it e.g. "providing design and analisys for migration to
> JUnit 4/5". I'll try to help with it!
>
>
> Test cases IgniteUidAsConsistentIdMigrationTest and
> TxRollbackAsyncNearCacheTest
> are not affected by my change.
> Nevertheless, I've rerun Run::All for this PR.
>
>
> All other preparations have already been done for this issue:
>
> PR: https://github.com/apache/ignite/pull/3844
> TC:
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_RunAll=buildTypeStatusDiv_IgniteTests24Java8=pull%2F3844%2Fhead
> Upsource: https://reviews.ignite.apache.org/ignite/review/IGNT-CR-581
> JIRA: https://issues.apache.org/jira/browse/IGNITE-8266
>
> Will you or others have to to review it?
>
>
>
> пт, 4 мая 2018 г. в 14:24, Dmitry Pavlov :
>
> > Hi Maxim,
> >
> > I think next step can be creation of Junit4/5 IgniteAbstractTest and/or
> > IgniteTestRunner. Would you like to contribute this prototype?
> >
> > Regarding TC run there is a number of suspicious tests
> >   (e.g. IgnitePdsNativeIoTestSuite2:
> >
> >
> IgniteUidAsConsistentIdMigrationTest.testNewStyleAlwaysSmallestNodeIndexIsCreatedMultithreaded
> > (fail rate 0,0%) & IgniteCacheTestSuite6:
> > TxRollbackAsyncNearCacheTest.testSynchronousRollback (fail rate 0,0%)  )
> > and a number of timeouts ocurred, so I've retriggered re-run for failed
> > tests -
> >
> >
> https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8_IgniteTests24Java8=pull%2F3844%2Fhead
> >
> > For IGNITE-8266 
> could
> > you please create CR?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> >
> > пт, 4 мая 2018 г. в 12:27, Maxim Muzafarov :
> >
> > > Igniters,
> > >
> > > Recenly, we've pached Ingite testing framework to stop all started
> > > instances after all test-cases completion by default.
> > > Details of impelemtaion can be viewed here [1]. This change leads us
> to a
> > > lot of boilerplate code.
> > >
> > >
> > > 1) I've created issue [2] and prepared PR [3] which removes all this
> > > boilerplate code. Most of these changes is about removing:
> > > ```
> > > @Override protected void afterTestsStopped() throws Exception {
> > > super.afterTestsStopped();
> > > stopAllGrids();
> > > }
> > > ```
> > > All tests looks good here. I've double cheched all requiremets and now
> I
> > > need your help with review and futher steps up to merge.
> > >
> > > Can anyone help me?
> > >
> > >
> > > 2) I've created issue [4] and planning to clean rarely used methods
> > related
> > > to stopAllGrids().
> > > E.g. stopAllClients and stopAllServers methods from GridAbstactTest
> used
> > > only once in whole project but they locates in the root class.
> > > From my point of view, this will simplify for futher migration Ignite
> > > project to JUnit 4/5 framework.
> > >
> > > What else can be done here?
> > > Please, share your thoughts.
> > >
> > >
> > >
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-6842
> > > [2] https://issues.apache.org/jira/browse/IGNITE-8266
> > > [3] https://github.com/apache/ignite/pull/3844
> > > [4] https://issues.apache.org/jira/browse/IGNITE-8157
> > >
> >
>

Re: method arguments code style

[Dmitry Pavlov]

Hi Vyacheslav,

plugin was published in AI wiki, feel free to create ticket to add method
code style check.

Actually I'm not sure it is easy to validate code style in plugin, but I
guess you know it better.

Sincerely,
Dmitriy Pavlov

вт, 8 мая 2018 г. в 21:47, Vyacheslav Daradur :

> Hi, Igniters!
>
> After the completion of publishing abbr-plugin [1][2] we will be able
> to automate checking of method arguments code style.
>
> It will be easy to check rules approved by the community during writing
> code.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-5698
> [2]
> http://apache-ignite-developers.2346864.n4.nabble.com/abbrevation-rules-plugin-td19356.html
>
> On Tue, May 8, 2018 at 8:34 PM, Dmitry Pavlov 
> wrote:
> > Folks, I've messed with another topic, where Vladimir was going to update
> > review check-list.
> >
> > Here I've updated Coding Guidelines:
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Coding+Guidelines#CodingGuidelines-MethodArguments
> >
> > Please review changes, so we can consider it is final.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > вт, 8 мая 2018 г. в 20:05, Dmitry Pavlov :
> >
> >> I thought that Vladimir will update.
> >>
> >> By the way, Denis M, I propose to grant access to the wiki to Dmitry G.
> >> WDYT?
> >>
> >> вт, 8 мая 2018 г. в 19:28, Dmitriy Govorukhin <
> >> dmitriy.govoruk...@gmail.com>:
> >>
> >>> Dmitriy,
> >>>
> >>> Сould you please update code style wiki page in accordance with the
> >>> results of the discussion?
> >>> On May 7 2018, at 11:00 am, Vladimir Ozerov 
> wrote:
> >>> >
> >>> > Dmitry,
> >>> > Agree, mixed style when some arguments share the same line and others
> >>> don't
> >>> > looks very bad. My proposal was to allow two styles - first when all
> >>> > arguments are on the same line splitted by 120 char limit, second
> when
> >>> all
> >>> > every arguments is on a separate line.
> >>> > Mixed style should be disallowed.
> >>> >
> >>> > On Mon, May 7, 2018 at 12:35 AM, Dmitriy Govorukhin <
> >>> > dmitriy.govoruk...@gmail.com> wrote:
> >>> >
> >>> > > Vladimir,
> >>> > > My eyes cry when I see this
> >>> > > public double getCost(Session ses, int[] masks, TableFilter[]
> filters,
> >>> > > int filter, SortOrder sortOrder,
> >>> > > HashSet cols) {
> >>> > > return SpatialTreeIndex.getCostRangeIndex(masks,table.
> >>> > > getRowCountApproximation(),
> >>> > > columns) / 10;
> >>> > > }
> >>> > >
> >>> > > Why did arguments split into 3 line?
> >>> > > It is much better readable for me
> >>> > > public double getCost(
> >>> > > Session ses,
> >>> > > int[] masks,
> >>> > > TableFilter[] filters,
> >>> > > int filter,
> >>> > > SortOrder sortOrder,
> >>> > > HashSet cols
> >>> > > ) {
> >>> > > return SpatialTreeIndex.getCostRangeIndex(masks,
> >>> > > able.getRowCountApproximation(),
> >>> > > columns) / 10;
> >>> > > }
> >>> > >
> >>> > > Do we have a serious reason to continue writing code as before?
> >>> > >
> >>> > >
> >>> > > On Thu, May 3, 2018 at 2:24 PM, Vladimir Ozerov <
> voze...@gridgain.com
> >>> >
> >>> > > wrote:
> >>> > >
> >>> > > > My opinion is that we should allow both styles and not enforce
> any
> >>> of
> >>> > > them.
> >>> > > > I hardly can say that this
> >>> > > >
> >>> > > > public double getCost(
> >>> > > > Session ses,
> >>> > > > int[] masks,
> >>> > > > TableFilter[] filters,
> >>> > > > int filter,
> >>> > > > SortOrder sortOrder,
> >>> > > > HashSet cols
> >>> > > > ) {
> >>> > > > return SpatialTreeIndex.getCostRangeIndex(masks,
> >>> > > > table.getRowCountApproximation(), columns) / 10;
> >>> > > > }
> >>> > > >
> >>> > > > is better than this
> >>> > > > public double getCost(Session ses, int[] masks, TableFilter[]
> >>> filters,
> >>> > > > int filter, SortOrder sortOrder,
> >>> > > > HashSet cols) {
> >>> > > > return SpatialTreeIndex.getCostRangeIndex(masks,
> >>> > > > table.getRowCountApproximation(), columns) / 10;
> >>> > > > }
> >>> > > >
> >>> > > >
> >>> > > > But this
> >>> > > > public CacheLockCandidates doneRemote(
> >>> > > > GridCacheVersion ver,
> >>> > > > Collection pending,
> >>> > > > Collection committed,
> >>> > > > Collection rolledback
> >>> > > > )
> >>> > > >
> >>> > > >
> >>> > > > looks better than this
> >>> > > > public CacheLockCandidates doneRemote(GridCacheVersion ver,
> >>> > > > Collection pending,
> >>> > > > Collection committed,
> >>> > > > Collection rolledback)
> >>> > > >
> >>> > > >
> >>> > > > The very problem is that our eyes feel comfortable when we either
> >>> move
> >>> > > them
> >>> > > > horizontally, or vertically (example 2). But with proposed code
> >>> style you
> >>> > > > have to do zigzag movements in general case because lines are not
> >>> aligned
> >>> > > > (example 1).
> >>> > > > Merge conflicts on multiliners are hardly of major concern for
> us.
> >>> > > >
> >>> > > > Vladimir.
> >>> > > > On Thu, May 3, 2018 at 1:46 PM, Eduard 

Re: IGNITE-6430 (Done) CacheGroupsMetricsRebalanceTest.testRebalanceEstimateFinishTime test fails periodically

[Dmitry Pavlov]

Hi Dmitriy G,

could you please pick up this review?

Sincerely,
Dmitriy Pavlov

чт, 10 мая 2018 г. в 15:50, Александр Меньшиков :

> Hi,
>
> I have fixed problem with the test
> CacheGroupsMetricsRebalanceTest.testRebalanceEstimateFinishTime.
>
> Please review and merge if okay.
>
> P.S.
> Aleksey Plekhanov already approved it (see Jira).
>
>
> JIRA: https://issues.apache.org/jira/browse/IGNITE-6430
> PR: https://github.com/apache/ignite/pull/3914
> TC (with 120 passes):
>
> https://ci.ignite.apache.org/viewLog.html?buildId=1246058=IgniteTests24Java8_Cache3=testsInfo
> TC:
>
> https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_BuildApacheIgnite_IgniteTests24Java8=pull%2F3914%2Fhead=buildTypeStatusDiv
>

Re: One more round for PR review

[Dmitry Pavlov]

Hi Alexey K,

could you please check that all proposals were applied?

Sincerely,
Dmitriy Pavlov

чт, 10 мая 2018 г. в 22:59, Igor Rudyak :

> Hi guys,
>
> I addressed all the comments and need one more round of review for the PR:
> https://github.com/apache/ignite/pull/3940
>
> Igor
>

[GitHub] ignite pull request #3995: IGNITE-8486: Updated ConcurrentLinkedDeque8 to ma...

[slukyano]

GitHub user slukyano opened a pull request:

https://github.com/apache/ignite/pull/3995

IGNITE-8486: Updated ConcurrentLinkedDeque8 to match the latest version in 
JSR 166 repo.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8486

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3995.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3995


commit e5704f9975de02fa08a8f9c3cee58bc1c83d6ddd
Author: Stanislav Lukyanov 
Date:   2018-05-14T16:06:48Z

IGNITE-8486: Updated ConcurrentLinkedDeque8 to match the latest version in 
JSR 166 repo.




---

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Dmitry Pavlov]

Hi Nikolay,

I don't mean we should release TDE without WAL, I mean we can consider
phase-1 as minmal mergeable chunk of functionality, which does not fail
tests and contains meaningfull set of changes for TDE.

Sincerely,
Dmitriy Pavlov

пн, 14 мая 2018 г. в 17:45, Nikolay Izhikov :

> Dmitry.
>
> From my point of view, WAL encryption should be done in Phase-1.
> We should provide only production ready features to the users, isn't it?
>
> Ticket for a phase-1 created -
> https://issues.apache.org/jira/browse/IGNITE-8485
>
> I'm starting working on it and expecting to implement it in a couple
> months.
>
> В Пн, 14/05/2018 в 17:33 +0300, Dmitry Pavlov пишет:
> > Hi Nickolay,
> >
> > Thank you for sharing results.
> >
> > I would suggest to make phase 1 as small as possible, for example,
> skipping
> > WAL encryption or something like that.
> >
> > It would not be full TDE implementation, but will allow us to move by
> small
> > steps, it also allows us to merge smaller changes to master.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 14 мая 2018 г. в 16:53, Nikolay Izhikov :
> >
> > > Hello, guys.
> > >
> > > We had private discussion about TDE with Dmitriy Pavlov, Vladimir
> Ozerov
> > > and Anton Vinogradov.
> > >
> > > Some decisions was made I want to approve with communtiy:
> > >
> > >   1. Current design of TDE is OK. We can start work on implementation.
> > >
> > >   2. We should split implementation to phases.
> > >  So we can focus on basic features and deliver it to users.
> > >  And after it, extend it for a full compliance with enterprise
> > > standarts such as PCI DSS and others.
> > >
> > > I propose following plan:
> > >
> > >   Phase-1. Basic TDE:
> > > 1. Usage of standard JKS, Java Security.
> > >
> > > 2. Persistent Data Encryption/Decryption.
> > >
> > >   Phase-2. Keys rotation:
> > > 1. Key regeneration.
> > >
> > > 2. CEK and MEK rotation.
> > >
> > > Other phases to be added.
> > >
> > > Thoughts or objections?
> > >
> > > В Пн, 07/05/2018 в 10:19 +, Dmitry Pavlov пишет:
> > > > Hi, just 2 remarks,
> > > >
> > > > 1) We should somehow separate issue with disc corruption and
> incorrect
> > >
> > > key.
> > > > For incorrect key I suggest to adopt Key Check Value (KCV) techique.
> It
> > >
> > > is
> > > > some heading bytes (e.g. 3 bytes) of encrypted 00...00 block using
> this
> > > > key. KCV allow us to check key decrypted correctly and check key
> before
> > > > processing storage.
> > > >
> > > > 2) I'm not sure about AES CBC, would it be applied only to one page,
> or
> > >
> > > to
> > > > whole store? We need random access to pages for example for page
> > > > replacement process, so we need to reset CBC init vector (IV) to some
> > >
> > > well
> > > > known value. If we set IV to 0 for each page dectyption, it means
> CBC is
> > > > applied only in scope of page. Continious IV accumulating CBC mode
> for
> > > > whole store seems to be not working. Same question about IV is
> applicable
> > > > to WAL records.
> > > >
> > > > Could we consider ECB? or limit CBC with zero or well known IV?
> > > >
> > > > Sincerely,
> > > > Dmitriy Pavlov
> > > >
> > > > сб, 5 мая 2018 г. в 17:49, Nikolay Izhikov :
> > > >
> > > > > Hello, Guys.
> > > > >
> > > > > Here are answers to the TDE design questions.
> > > > > I will create FAQ in IEP-18 with this answers, also.
> > > > >
> > > > > > 1. MEK, CEK rotation. Should we provide the way to
> change(regenerate)
> > > > >
> > > > > MEK, CEK from time to time?
> > > > >
> > > > > Yes. PCI DSS are require it. See 3.6.4, 3.6.5 sections.
> > > > >
> > > > > > 2. Does CEK(table keys) relate to user access permission?
> > > > >
> > > > > No. It is prohibited by PCI DSS. 3.4.1: "Decryption keys must not
> be
> > > > > associated with user accounts"
> > > > >
> > > > > > 3. WAL encryption. How will it be implemented?
> > > > >
> > > > > LogicalRecord – key+value for encrypted cache entries are
> encrypted.
> > > > > PhysicalRecord.FullPageSnaphot - regular page encryption algorithm
> > >
> > > used.
> > > > > PhysicalRecord.DeltaRecord - delta is encrypted.
> > > > >
> > > > > > 4. We should keep CEKs in MetaStore.
> > > > >
> > > > > Yes, we should.
> > > > > We can't store CEKs in KeyStore because of PCI DSS 3.5.3.c:
> > > > > "Key-encrypting keys are stored separately from data-encrypting
> keys."
> > > > >
> > > > > > 5. How should we handle following case
> > > > >
> > > > > I propose to not include the node to cluster with the appropriate
> > > > > exception message.
> > > > >
> > > > > > 6. Public API to deal with CEK should be provided.
> > > > >
> > > > > The first draft of public API for TDE -
> > > > > https://github.com/nizhikov/ignite/pull/12
> > > > >
> > > > > > 7. Can each node use own CEK? What are pros and cons for that?
> > > > >
> > > > > I propose to use one CEK per cache.
> > > > > CEK would be the same on every cluster node.
> > 

[GitHub] ignite pull request #3994: IGNITE-8487 tmpfs is employed by ZooKeeper tests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3994


---

Re: supporting different configuration format json,yaml...

[Ivan Rakov]

Dmitry,

We rely on Spring Framework when we start Ignite node from XML 
configuration. Spring doesn't easily support another formats of 
configuration files. I think, the main reason for this is built-in 
ability to validate configuration via XML Schema. We can surely hack 
this around (I bet there are existing libraries for configuring Spring 
with JSON), but I don't think that anyone suffered from inability to 
statically configure Ignite with json/yaml.


Regarding thin clients: makes sense. I suppose necessary mappings will 
be implemented as a part of thin client.


Best Regards,
Ivan Rakov

On 14.05.2018 18:58, Dmitriy Govorukhin wrote:

Hi, Igniters!

As far as I know, many people work on a thin client for different language
(go,js,php...).
Are there any reasons why ignite does not support yaml or json format for
configuration? or some other popular format?
In future, it can help to integrate with thin clients, for example, js
client may want to dynamic cache start, he passes cache configuration (in
native format, for js it will json) through TCP, Ignite node unwrap and
remap to java representation and dynamic start cache.

[GitHub] ignite pull request #3994: IGNITE-8487 tmpfs is employed by ZooKeeper tests

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/3994

IGNITE-8487 tmpfs is employed by ZooKeeper tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8487

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3994.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3994


commit 215c4e0658fb4da554db084158c7013e92a446d2
Author: Sergey Chugunov 
Date:   2018-05-14T16:45:54Z

IGNITE-8487 tmpfs is employed by ZooKeeper tests




---

Re: abbrevation rules plugin

[Dmitry Pavlov]

Anton, thank you for pointing to this.

I've updated wiki page with build and references:
https://cwiki.apache.org/confluence/display/IGNITE/Abbreviation+Rules

and attached jar file
https://cwiki.apache.org/confluence/pages/viewpageattachments.action?pageId=57901460


Thanks to Vyacheslav plugin is now published.

Sincerely,
Dmitriy Pavlov


пн, 14 мая 2018 г. в 19:14, Anton Vinogradov :

> Jar already attached to issue :)
>
> see https://issues.apache.org/jira/browse/IGNITE-5698
>
> пн, 14 мая 2018 г. в 19:05, Dmitry Pavlov :
>
> > Hi Folks,
> >
> > I suggest to keep it in https://github.com/dspavlov/ignite-abbrev-plugin
> > for
> > the beginning. Late we can consider Apache repo.
> >
> > But for wiki I need some distribution to attach first.
> >
> > Vyacheslav, could you remind me if we already have one?
> >
> > пн, 14 мая 2018 г. в 19:00, Anton Vinogradov :
> >
> > > Awesome!
> > >
> > > Let's mention it officially on wiki.
> > > Do we need special apache repo for this plugin? As for me, public
> github
> > > repo is enough.
> > >
> > > пн, 14 мая 2018 г. в 18:34, Dmitriy Pavlov :
> > >
> > > > Hi Igniters, Vyacheslav,
> > > >
> > > > I had discussion about this plugin. As GridGain representative I
> > approve
> > > > change of file headers and making this tool fully opensource.
> > > >
> > > > So please proceed with distributive build.
> > > >
> > > > Best Regards,
> > > > Dmitriy Palvov
> > > >
> > > >
> > > > On Mon, May 14, 2018 at 6:28 PM Dmitry Pavlov  >
> > > > wrote:
> > > >
> > > >>
> > > >>
> > > >> -- Forwarded message -
> > > >> From: Andrey Gura 
> > > >> Date: ср, 4 апр. 2018 г. в 12:05
> > > >> Subject: Re: abbrevation rules plugin
> > > >> To: 
> > > >>
> > > >>
> > > >> Both prefixes "Grid" and "Ignite" in class names are redundant for
> > most
> > > >> cases. So there is no any rules for this.
> > > >>
> > > >> вт, 3 апр. 2018 г., 19:28 Vyacheslav Daradur :
> > > >>
> > > >> > Hi, Igniters!
> > > >> >
> > > >> > I got into the task [1] in agreement with Dmitry.
> > > >> >
> > > >> > I’ve prepared the PR [2] with following changes:
> > > >> > - added Apache 2.0 license to the header of each file
> > > >> > - replaced prefix ‘Grid’ by ‘Ignite’ in class names
> > > >> > - added README.md with a simple description
> > > >> > - changed version 2.6.3 -> 3.0.0 because the project has own
> GitHub
> > > >> > releases tab and completely new versioning isn't preferable IMO
> > > >> >
> > > >> > Also, I’ve built new artifact and tested it, it works well.
> > > >> >
> > > >> > Am I understood the task correct?
> > > >> > If not, please share your notes I will fix them shortly.
> > > >> >
> > > >> > As far as I understand someone of GridGain employee should confirm
> > the
> > > >> > plugin's code's donation. It’d be great to confirm that by
> corporate
> > > >> > email.
> > > >> >
> > > >> >
> > > >> > [1] https://issues.apache.org/jira/browse/IGNITE-5698
> > > >> > [2] https://github.com/dspavlov/ignite-abbrev-plugin/pull/1
> > > >> >
> > > >> >
> > > >> > On Thu, Jul 6, 2017 at 12:30 PM, Dmitry Pavlov <
> > dpavlov@gmail.com
> > > >
> > > >> > wrote:
> > > >> > > Hi Dmitriy, Denis,
> > > >> > >
> > > >> > > Thank you for your effort to find out solution.
> > > >> > > It sound good for me, I have assigned IGNITE-5698 to myself.
> > > >> > >
> > > >> > > Best Regards,
> > > >> > > Dmitriy Pavlov
> > > >> > >
> > > >> > >
> > > >> > > чт, 6 июл. 2017 г. в 2:45, Denis Magda :
> > > >> > >
> > > >> > >> Dmitriy P.,
> > > >> > >>
> > > >> > >> Does it sound good to you? If yes, please make sure the plugin
> is
> > > >> > >> available via a public GitHub repo and refer to it from the
> doc.
> > > >> > >>
> > > >> > >> —
> > > >> > >> Denis
> > > >> > >>
> > > >> > >> > On Jul 5, 2017, at 4:43 PM, Dmitriy Setrakyan <
> > > >> dsetrak...@apache.org>
> > > >> > >> wrote:
> > > >> > >> >
> > > >> > >> > On Wed, Jul 5, 2017 at 4:41 PM, Denis Magda <
> dma...@apache.org
> > >
> > > >> > wrote:
> > > >> > >> >
> > > >> > >> >> Ok, what if we make sure the plugin is available on GitHub
> > (not
> > > >> > Ignite)
> > > >> > >> >> and give a link to it on the documentation page? This seems
> to
> > > be
> > > >> the
> > > >> > >> >> easiest way to handle the topic.
> > > >> > >> >
> > > >> > >> >
> > > >> > >> > I think this will work.
> > > >> > >>
> > > >> > >>
> > > >> >
> > > >> >
> > > >>
> > > >> >
> > > >> > --
> > > >> > Best Regards, Vyacheslav D.
> > > >> >
> > > >>
> > > >
> > >
> >
>

[jira] [Created] (IGNITE-8487) ZookeeperDiscoverySpi tests should use tmpfs on public TC

[Sergey Chugunov (JIRA)]

Sergey Chugunov created IGNITE-8487:
---

 Summary: ZookeeperDiscoverySpi tests should use tmpfs on public TC
 Key: IGNITE-8487
 URL: https://issues.apache.org/jira/browse/IGNITE-8487
 Project: Ignite
  Issue Type: Task
Reporter: Sergey Chugunov
Assignee: Sergey Chugunov


ZookeeperDiscoverySpi suites don't look very stable on TC these failures are 
flaky.

As ZooKeeper persists data to disk slow disks on TC agents may contribute to 
these flaky failures.

Zookeeper tests should configure internal ZooKeeper instances to use tmpfs 
available on some TC agents to avoid failures because of slow ZooKeeper cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3992: IGNITE-8469: add memory leak test case

[Mmuzaf]

Github user Mmuzaf closed the pull request at:

https://github.com/apache/ignite/pull/3992


---

Re: abbrevation rules plugin

[Anton Vinogradov]

Jar already attached to issue :)

see https://issues.apache.org/jira/browse/IGNITE-5698

пн, 14 мая 2018 г. в 19:05, Dmitry Pavlov :

> Hi Folks,
>
> I suggest to keep it in https://github.com/dspavlov/ignite-abbrev-plugin
> for
> the beginning. Late we can consider Apache repo.
>
> But for wiki I need some distribution to attach first.
>
> Vyacheslav, could you remind me if we already have one?
>
> пн, 14 мая 2018 г. в 19:00, Anton Vinogradov :
>
> > Awesome!
> >
> > Let's mention it officially on wiki.
> > Do we need special apache repo for this plugin? As for me, public github
> > repo is enough.
> >
> > пн, 14 мая 2018 г. в 18:34, Dmitriy Pavlov :
> >
> > > Hi Igniters, Vyacheslav,
> > >
> > > I had discussion about this plugin. As GridGain representative I
> approve
> > > change of file headers and making this tool fully opensource.
> > >
> > > So please proceed with distributive build.
> > >
> > > Best Regards,
> > > Dmitriy Palvov
> > >
> > >
> > > On Mon, May 14, 2018 at 6:28 PM Dmitry Pavlov 
> > > wrote:
> > >
> > >>
> > >>
> > >> -- Forwarded message -
> > >> From: Andrey Gura 
> > >> Date: ср, 4 апр. 2018 г. в 12:05
> > >> Subject: Re: abbrevation rules plugin
> > >> To: 
> > >>
> > >>
> > >> Both prefixes "Grid" and "Ignite" in class names are redundant for
> most
> > >> cases. So there is no any rules for this.
> > >>
> > >> вт, 3 апр. 2018 г., 19:28 Vyacheslav Daradur :
> > >>
> > >> > Hi, Igniters!
> > >> >
> > >> > I got into the task [1] in agreement with Dmitry.
> > >> >
> > >> > I’ve prepared the PR [2] with following changes:
> > >> > - added Apache 2.0 license to the header of each file
> > >> > - replaced prefix ‘Grid’ by ‘Ignite’ in class names
> > >> > - added README.md with a simple description
> > >> > - changed version 2.6.3 -> 3.0.0 because the project has own GitHub
> > >> > releases tab and completely new versioning isn't preferable IMO
> > >> >
> > >> > Also, I’ve built new artifact and tested it, it works well.
> > >> >
> > >> > Am I understood the task correct?
> > >> > If not, please share your notes I will fix them shortly.
> > >> >
> > >> > As far as I understand someone of GridGain employee should confirm
> the
> > >> > plugin's code's donation. It’d be great to confirm that by corporate
> > >> > email.
> > >> >
> > >> >
> > >> > [1] https://issues.apache.org/jira/browse/IGNITE-5698
> > >> > [2] https://github.com/dspavlov/ignite-abbrev-plugin/pull/1
> > >> >
> > >> >
> > >> > On Thu, Jul 6, 2017 at 12:30 PM, Dmitry Pavlov <
> dpavlov@gmail.com
> > >
> > >> > wrote:
> > >> > > Hi Dmitriy, Denis,
> > >> > >
> > >> > > Thank you for your effort to find out solution.
> > >> > > It sound good for me, I have assigned IGNITE-5698 to myself.
> > >> > >
> > >> > > Best Regards,
> > >> > > Dmitriy Pavlov
> > >> > >
> > >> > >
> > >> > > чт, 6 июл. 2017 г. в 2:45, Denis Magda :
> > >> > >
> > >> > >> Dmitriy P.,
> > >> > >>
> > >> > >> Does it sound good to you? If yes, please make sure the plugin is
> > >> > >> available via a public GitHub repo and refer to it from the doc.
> > >> > >>
> > >> > >> —
> > >> > >> Denis
> > >> > >>
> > >> > >> > On Jul 5, 2017, at 4:43 PM, Dmitriy Setrakyan <
> > >> dsetrak...@apache.org>
> > >> > >> wrote:
> > >> > >> >
> > >> > >> > On Wed, Jul 5, 2017 at 4:41 PM, Denis Magda  >
> > >> > wrote:
> > >> > >> >
> > >> > >> >> Ok, what if we make sure the plugin is available on GitHub
> (not
> > >> > Ignite)
> > >> > >> >> and give a link to it on the documentation page? This seems to
> > be
> > >> the
> > >> > >> >> easiest way to handle the topic.
> > >> > >> >
> > >> > >> >
> > >> > >> > I think this will work.
> > >> > >>
> > >> > >>
> > >> >
> > >> >
> > >>
> > >> >
> > >> > --
> > >> > Best Regards, Vyacheslav D.
> > >> >
> > >>
> > >
> >
>

Re: Disable WAL for several cache groups within one exchange

[Anton Vinogradov]

Ivan,

enableWal/disableWal will return false in case enabling/disabling was
caused not by this call.
For example it will return false in case wal already enabled/disabled.

Example:

boolean res1 = srv.cluster().enableWal(CACHE_NAME);
boolean res2 = srv.cluster().enableWal(CACHE_NAME);

assert res1;
assert !res2;


Vova,

Since you made final tuning for this solution, is there any special cases
when false can be returned?

пт, 11 мая 2018 г. в 18:39, Ivan Rakov :

> Agree about collections.
>
> Regarding return type: it's a tricky question. Maybe author of this
> feature may help.
> Anton V., in which case enableWal/disableWal can return false?
>
> Best Regards,
> Ivan Rakov
>
> On 11.05.2018 18:19, Vladimir Ozerov wrote:
> > Ivan,
> >
> > This proven to be too hard to understand.  It is better to have a lot
> small
> > methods with clear and compact semantics. Also arrays are harder to
> manage
> > than collections, users typically prefer the latest.
> > Also we need to think on what would be a result of this operation.
> Current
> > methods with a single cache return true/false based on whether they
> changed
> > something or not. Should we continue returning a single boolean for batch
> > operations as well?
> >
> > On Fri, May 11, 2018 at 6:13 PM, Ivan Rakov 
> wrote:
> >
> >> It would be six methods in total (3 for enabling, 3 for disabling).
> >> What about accepting null in *enableWAL(String... caches)* as wildcard?
> >>
> >> Best Regards,
> >> Ivan Rakov
> >>
> >>
> >> On 11.05.2018 17:52, Andrey Mashenkov wrote:
> >>
> >>> Ivan,
> >>>
> >>> Huge +1 for this improvement.
> >>>
> >>> I think we can have 2 overloaded method enableWal() with no args to
> enable
> >>> WAL for all caches
> >>> and enableWAL(String... caches) for one or multiple caches. (and same
> for
> >>> disable wal)
> >>>
> >>>
> >>>
> >>> On Fri, May 11, 2018 at 5:25 PM, Dmitriy Govorukhin <
> >>> dmitriy.govoruk...@gmail.com> wrote:
> >>>
> >>> Ivan,
>  Agree, if we have the batch method for cache create, we should have
> the
>  ability to enable/disable WAL in the batch too.
> 
>  On Fri, May 11, 2018 at 5:17 PM, Ivan Rakov 
>  wrote:
> 
>  Igniters,
> > API method for disabling WAL in IgniteCluster accepts only one cache
> >
>  name.
> 
> > Every call triggers exchange and checkpoints cluster-wide - it takes
> >
>  plenty
> 
> > of time to disable/enable WAL for multiple caches.
> > I think, we should add option to disable/enable WAL for several
> caches
> > with single command.
> >
> > Thoughts?
> >
> > --
> > Best Regards,
> > Ivan Rakov
> >
> >
> >
> >>>
>
>

[jira] [Created] (IGNITE-8486) Update ConcurrentLinkedDeque in Ignite's master repository to the latest JDK version

[Stanislav Lukyanov (JIRA)]

Stanislav Lukyanov created IGNITE-8486:
--

 Summary: Update ConcurrentLinkedDeque in Ignite's master 
repository to the latest JDK version
 Key: IGNITE-8486
 URL: https://issues.apache.org/jira/browse/IGNITE-8486
 Project: Ignite
  Issue Type: Improvement
Reporter: Stanislav Lukyanov
Assignee: Stanislav Lukyanov


Ignite still uses copies of several JSR 166 (j.u.concurrent) classes in it's 
sources. Those copies are now outdated compared to the latest versions used in 
JDK.
In particular, `ConcurrentLinkedDeque` has received a couple of correctness 
fixes recently (https://bugs.openjdk.java.net/browse/JDK-8188900, 
https://bugs.openjdk.java.net/browse/JDK-8189387). It would be good to have 
them in Ignite as well to protect ourselves from possible issues.

The task is to update Ignite's `ConcurrentLinkedDeque8` to the latest version 
of `ConcurrentLinkedDeque`, although keeping compatibility with earlier Java 
version (e.g. JDK's version now uses Java 9's VarHandles which we can't use 
yet).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: abbrevation rules plugin

[Dmitry Pavlov]

Hi Folks,

I suggest to keep it in https://github.com/dspavlov/ignite-abbrev-plugin for
the beginning. Late we can consider Apache repo.

But for wiki I need some distribution to attach first.

Vyacheslav, could you remind me if we already have one?

пн, 14 мая 2018 г. в 19:00, Anton Vinogradov :

> Awesome!
>
> Let's mention it officially on wiki.
> Do we need special apache repo for this plugin? As for me, public github
> repo is enough.
>
> пн, 14 мая 2018 г. в 18:34, Dmitriy Pavlov :
>
> > Hi Igniters, Vyacheslav,
> >
> > I had discussion about this plugin. As GridGain representative I approve
> > change of file headers and making this tool fully opensource.
> >
> > So please proceed with distributive build.
> >
> > Best Regards,
> > Dmitriy Palvov
> >
> >
> > On Mon, May 14, 2018 at 6:28 PM Dmitry Pavlov 
> > wrote:
> >
> >>
> >>
> >> -- Forwarded message -
> >> From: Andrey Gura 
> >> Date: ср, 4 апр. 2018 г. в 12:05
> >> Subject: Re: abbrevation rules plugin
> >> To: 
> >>
> >>
> >> Both prefixes "Grid" and "Ignite" in class names are redundant for most
> >> cases. So there is no any rules for this.
> >>
> >> вт, 3 апр. 2018 г., 19:28 Vyacheslav Daradur :
> >>
> >> > Hi, Igniters!
> >> >
> >> > I got into the task [1] in agreement with Dmitry.
> >> >
> >> > I’ve prepared the PR [2] with following changes:
> >> > - added Apache 2.0 license to the header of each file
> >> > - replaced prefix ‘Grid’ by ‘Ignite’ in class names
> >> > - added README.md with a simple description
> >> > - changed version 2.6.3 -> 3.0.0 because the project has own GitHub
> >> > releases tab and completely new versioning isn't preferable IMO
> >> >
> >> > Also, I’ve built new artifact and tested it, it works well.
> >> >
> >> > Am I understood the task correct?
> >> > If not, please share your notes I will fix them shortly.
> >> >
> >> > As far as I understand someone of GridGain employee should confirm the
> >> > plugin's code's donation. It’d be great to confirm that by corporate
> >> > email.
> >> >
> >> >
> >> > [1] https://issues.apache.org/jira/browse/IGNITE-5698
> >> > [2] https://github.com/dspavlov/ignite-abbrev-plugin/pull/1
> >> >
> >> >
> >> > On Thu, Jul 6, 2017 at 12:30 PM, Dmitry Pavlov  >
> >> > wrote:
> >> > > Hi Dmitriy, Denis,
> >> > >
> >> > > Thank you for your effort to find out solution.
> >> > > It sound good for me, I have assigned IGNITE-5698 to myself.
> >> > >
> >> > > Best Regards,
> >> > > Dmitriy Pavlov
> >> > >
> >> > >
> >> > > чт, 6 июл. 2017 г. в 2:45, Denis Magda :
> >> > >
> >> > >> Dmitriy P.,
> >> > >>
> >> > >> Does it sound good to you? If yes, please make sure the plugin is
> >> > >> available via a public GitHub repo and refer to it from the doc.
> >> > >>
> >> > >> —
> >> > >> Denis
> >> > >>
> >> > >> > On Jul 5, 2017, at 4:43 PM, Dmitriy Setrakyan <
> >> dsetrak...@apache.org>
> >> > >> wrote:
> >> > >> >
> >> > >> > On Wed, Jul 5, 2017 at 4:41 PM, Denis Magda 
> >> > wrote:
> >> > >> >
> >> > >> >> Ok, what if we make sure the plugin is available on GitHub (not
> >> > Ignite)
> >> > >> >> and give a link to it on the documentation page? This seems to
> be
> >> the
> >> > >> >> easiest way to handle the topic.
> >> > >> >
> >> > >> >
> >> > >> > I think this will work.
> >> > >>
> >> > >>
> >> >
> >> >
> >>
> >> >
> >> > --
> >> > Best Regards, Vyacheslav D.
> >> >
> >>
> >
>

Re: Tests coverage report of Ignite 2.5 core module

[Александр Меньшиков]

Hi Vyacheslav,
Great job!

I specifically happy to see jacoco.csv file into the archive.
I think I can range classes by some complex criterion, like
coverage/complexity.
That classes could be the first in the queue for testing.

2018-05-14 17:49 GMT+03:00 Vyacheslav Daradur :

> TestSuites weren't used.
>
> Set of tests classes was created as result of finding not abstract
> classes which names contain '*Test*' except '*TestCase*' in
> '/src/test/java/org/apache/ignite' as Maven does [1].
>
> Each tests class was executed and covered separately.
>
> You can see used tests classes in the file 'tests-classes.txt' which
> was included in the archive.
>
>
> [1] http://maven.apache.org/surefire/maven-surefire-
> plugin/examples/inclusion-exclusion.html
>
>
>
> On Mon, May 14, 2018 at 5:30 PM, Dmitry Pavlov 
> wrote:
> > Ok, thank you. I missed that I should download the archive first.
> >
> > I've found for example that method truncate in FSyncWalManager is not
> > covered.
> >
> > Which test suites were executed?
> >
> > пн, 14 мая 2018 г. в 17:16, Vyacheslav Daradur :
> >
> >> Hi, Dmitry,
> >>
> >> Download and unpack the archive first:
> >>
> >> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-
> core-module-tests-coverage-report-rev-d83f1ec.zip
> >>
> >>
> >> On Mon, May 14, 2018 at 5:12 PM, Dmitry Pavlov 
> >> wrote:
> >> > Hi Vyacheslav,
> >> >
> >> >
> >> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-
> core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0&
> file_subpath=%2Fcoverage-report%2Findex.html
> >> >
> >> > says error 400 for me.
> >> >
> >> > Sincerely,
> >> > Dmitriy Pavlov
> >> >
> >> > пн, 14 мая 2018 г. в 17:06, Vyacheslav Daradur :
> >> >
> >> >> Hi, Igniters!
> >> >>
> >> >> I’ve made tests coverage report of core module using Jacoco 0.8.1
> >> >>
> >> >> I’d like to share results with the community. Here is a link:
> >> >>
> >> >>
> >> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-
> core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0
> >> >>
> >> >> For viewing the report unpack the archive and open:
> >> >> coverage-report/index.html
> >> >>
> >> >> Also, there is ‘tests-classes.txt’ which includes names of tests
> >> >> classes which have been used.
> >> >>
> >> >> The report was generated from branch: ‘ignite-2-5’, commit:
> >> >>
> >> >>
> >> https://github.com/apache/ignite/tree/d83f1ec0a4d010e57fecc12ffd3d1f
> 8346ded61c
> >> >>
> >> >>
> >> >> --
> >> >> Best Regards, Vyacheslav D.
> >> >>
> >>
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav D.
> >>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: abbrevation rules plugin

[Dmitriy Pavlov]

Hi Igniters, Vyacheslav,

I had discussion about this plugin. As GridGain representative I approve
change of file headers and making this tool fully opensource.

So please proceed with distributive build.

Best Regards,
Dmitriy Palvov


On Mon, May 14, 2018 at 6:28 PM Dmitry Pavlov  wrote:

>
>
> -- Forwarded message -
> From: Andrey Gura 
> Date: ср, 4 апр. 2018 г. в 12:05
> Subject: Re: abbrevation rules plugin
> To: 
>
>
> Both prefixes "Grid" and "Ignite" in class names are redundant for most
> cases. So there is no any rules for this.
>
> вт, 3 апр. 2018 г., 19:28 Vyacheslav Daradur :
>
> > Hi, Igniters!
> >
> > I got into the task [1] in agreement with Dmitry.
> >
> > I’ve prepared the PR [2] with following changes:
> > - added Apache 2.0 license to the header of each file
> > - replaced prefix ‘Grid’ by ‘Ignite’ in class names
> > - added README.md with a simple description
> > - changed version 2.6.3 -> 3.0.0 because the project has own GitHub
> > releases tab and completely new versioning isn't preferable IMO
> >
> > Also, I’ve built new artifact and tested it, it works well.
> >
> > Am I understood the task correct?
> > If not, please share your notes I will fix them shortly.
> >
> > As far as I understand someone of GridGain employee should confirm the
> > plugin's code's donation. It’d be great to confirm that by corporate
> > email.
> >
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-5698
> > [2] https://github.com/dspavlov/ignite-abbrev-plugin/pull/1
> >
> >
> > On Thu, Jul 6, 2017 at 12:30 PM, Dmitry Pavlov 
> > wrote:
> > > Hi Dmitriy, Denis,
> > >
> > > Thank you for your effort to find out solution.
> > > It sound good for me, I have assigned IGNITE-5698 to myself.
> > >
> > > Best Regards,
> > > Dmitriy Pavlov
> > >
> > >
> > > чт, 6 июл. 2017 г. в 2:45, Denis Magda :
> > >
> > >> Dmitriy P.,
> > >>
> > >> Does it sound good to you? If yes, please make sure the plugin is
> > >> available via a public GitHub repo and refer to it from the doc.
> > >>
> > >> —
> > >> Denis
> > >>
> > >> > On Jul 5, 2017, at 4:43 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > >> wrote:
> > >> >
> > >> > On Wed, Jul 5, 2017 at 4:41 PM, Denis Magda 
> > wrote:
> > >> >
> > >> >> Ok, what if we make sure the plugin is available on GitHub (not
> > Ignite)
> > >> >> and give a link to it on the documentation page? This seems to be
> the
> > >> >> easiest way to handle the topic.
> > >> >
> > >> >
> > >> > I think this will work.
> > >>
> > >>
> >
> >
>
> >
> > --
> > Best Regards, Vyacheslav D.
> >
>

Re: async operation is not fair async

[Dmitriy Govorukhin]

Alexey,

Could you please add more description information for this task? [1]
Perhaps, base steps for implementation.

[1] https://issues.apache.org/jira/browse/IGNITE-8475

On Mon, May 14, 2018 at 4:58 PM, Alexey Goncharuk <
alexey.goncha...@gmail.com> wrote:

> Another +1 for the true asynchronous approach. I remember a while ago one
> of the Ignite users raised a similar question regarding the *async method
> being blocked on establishing a TCP connection.
>
> As far as deadlocks go, I have a counter-example. Currently, we check the
> thread-local chain only for a single cache, so if I run the following code:
> cache1.getAsync(k1);
> cache2.getAsync(k2);
> then the deadlock is still possible, and I did not see a single user
> complaining about unexpected deadlocks. Rather than implementing this
> cross-cache chain (which would probably add another overhead), I would make
> it consistent and allow operations to be run in parallel.
>
> There are many use-cases when having true async operations dramatically
> improve performance. Consider, for example, a streaming example when keys
> are being pushed by a client to a cluster. Currently, to run effective
> processing, the user will have to use a data streamer with custom keys
> receiver which may be a huge usability downside. Async operations can
> utilize the cluster resources very efficiently.
>
> Finally, if we want to be on the safe side, we can keep the operation chain
> inside a transaction. I see absolutely no point in maintaining this chain
> outside of transactions.
>
> --AG
>
> 2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin <
> dmitriy.govoruk...@gmail.com>
> :
>
> > Andrey,
> >
> > Do you prefer change behavior at runtime?
> > I guess will be better have different methods for getting cache instance
> > with fair and not fair sync.
> >
> > On Mon, May 14, 2018 at 3:39 PM, Andrey Gura  wrote:
> >
> > > +1 for fair async operations.
> > >
> > > But I don't like idea use withFairSync() method. We added xxxAsync()
> > > methods recently and withAsync() is deprecated.
> > >
> > > I think we should just make methods are async in nature and provide
> > > ability of switching to the old behaviour using flag or property.
> > >
> > > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> > >  wrote:
> > > > Vladimir,
> > > >
> > > > In general I agree, but I do get greatly *close-minded* (pun
> intended)
> > > > whenever users' code that worked for the past several years all of a
> > > sudden
> > > > gets deadlocked after an upgrade. Making this feature optional is
> even
> > > > worse and more confusing. In this case the best action is no action
> at
> > > all.
> > > >
> > > > BTW, would be interesting to find out how Oracle async driver behaves
> > in
> > > > this case.
> > > >
> > > > D.
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > > > wrote:
> > > >
> > > >> Guys,
> > > >>
> > > >> To build a great product we should be open minded and look to the
> > > future,
> > > >> not to the past.
> > > >>
> > > >> Dima raised very valid point - why async is not async? Current
> > > programming
> > > >> culture and demanding performance requirements pushes users towards
> > > >> reactive-style programming. I do not want my thread to ever be
> > blocked.
> > > >> Instead, I want to send a number of concurrent commands and
> optionally
> > > >> subscribe to final result. So trully async API makes total sense to
> > me.
> > > >>
> > > >> But personally, my primary interest in this area is SQL. Oracle is
> > > >> preparing new async driver. ADBA - async database access. It was
> > > presented
> > > >> on recent JavaOne [1]. It is under active development right now -
> juse
> > > >> weave through the mailing list [2]. Some prototypes are already
> there
> > > [3].
> > > >> PostgreSQL community even started adopted it [4]!
> > > >>
> > > >> I am not pushing for immediate actions, but at least we should
> > > understand
> > > >> which way the wind is blowing. As a mid-term goals I would propose
> to
> > > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> > for
> > > >> suspend/resume in different threads. And as a next step I would
> think
> > on
> > > >> adopting async cache and SQL APIs.
> > > >>
> > > >> Vladimir.
> > > >>
> > > >> [1]
> > > >> http://www.oracle.com/technetwork/database/
> > > application-development/jdbc/
> > > >> con1491-3961036.pdf
> > > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > > >> [3] https://github.com/oracle/oracle-db-examples/tree/
> master/java/AoJ
> > > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > > >>
> > > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > > dsetrak...@apache.org>
> > > >> wrote:
> > > >>
> > > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > > >> > dmitriy.govoruk...@gmail.com> wrote:
> > > >> >
> > > >> > > I will 

Re: Tests coverage report of Ignite 2.5 core module

[Vyacheslav Daradur]

TestSuites weren't used.

Set of tests classes was created as result of finding not abstract
classes which names contain '*Test*' except '*TestCase*' in
'/src/test/java/org/apache/ignite' as Maven does [1].

Each tests class was executed and covered separately.

You can see used tests classes in the file 'tests-classes.txt' which
was included in the archive.


[1] 
http://maven.apache.org/surefire/maven-surefire-plugin/examples/inclusion-exclusion.html



On Mon, May 14, 2018 at 5:30 PM, Dmitry Pavlov  wrote:
> Ok, thank you. I missed that I should download the archive first.
>
> I've found for example that method truncate in FSyncWalManager is not
> covered.
>
> Which test suites were executed?
>
> пн, 14 мая 2018 г. в 17:16, Vyacheslav Daradur :
>
>> Hi, Dmitry,
>>
>> Download and unpack the archive first:
>>
>> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip
>>
>>
>> On Mon, May 14, 2018 at 5:12 PM, Dmitry Pavlov 
>> wrote:
>> > Hi Vyacheslav,
>> >
>> >
>> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0_subpath=%2Fcoverage-report%2Findex.html
>> >
>> > says error 400 for me.
>> >
>> > Sincerely,
>> > Dmitriy Pavlov
>> >
>> > пн, 14 мая 2018 г. в 17:06, Vyacheslav Daradur :
>> >
>> >> Hi, Igniters!
>> >>
>> >> I’ve made tests coverage report of core module using Jacoco 0.8.1
>> >>
>> >> I’d like to share results with the community. Here is a link:
>> >>
>> >>
>> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0
>> >>
>> >> For viewing the report unpack the archive and open:
>> >> coverage-report/index.html
>> >>
>> >> Also, there is ‘tests-classes.txt’ which includes names of tests
>> >> classes which have been used.
>> >>
>> >> The report was generated from branch: ‘ignite-2-5’, commit:
>> >>
>> >>
>> https://github.com/apache/ignite/tree/d83f1ec0a4d010e57fecc12ffd3d1f8346ded61c
>> >>
>> >>
>> >> --
>> >> Best Regards, Vyacheslav D.
>> >>
>>
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>



-- 
Best Regards, Vyacheslav D.

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Nikolay Izhikov]

Dmitry.

From my point of view, WAL encryption should be done in Phase-1.
We should provide only production ready features to the users, isn't it?

Ticket for a phase-1 created - https://issues.apache.org/jira/browse/IGNITE-8485

I'm starting working on it and expecting to implement it in a couple months.

В Пн, 14/05/2018 в 17:33 +0300, Dmitry Pavlov пишет:
> Hi Nickolay,
> 
> Thank you for sharing results.
> 
> I would suggest to make phase 1 as small as possible, for example, skipping
> WAL encryption or something like that.
> 
> It would not be full TDE implementation, but will allow us to move by small
> steps, it also allows us to merge smaller changes to master.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> пн, 14 мая 2018 г. в 16:53, Nikolay Izhikov :
> 
> > Hello, guys.
> > 
> > We had private discussion about TDE with Dmitriy Pavlov, Vladimir Ozerov
> > and Anton Vinogradov.
> > 
> > Some decisions was made I want to approve with communtiy:
> > 
> >   1. Current design of TDE is OK. We can start work on implementation.
> > 
> >   2. We should split implementation to phases.
> >  So we can focus on basic features and deliver it to users.
> >  And after it, extend it for a full compliance with enterprise
> > standarts such as PCI DSS and others.
> > 
> > I propose following plan:
> > 
> >   Phase-1. Basic TDE:
> > 1. Usage of standard JKS, Java Security.
> > 
> > 2. Persistent Data Encryption/Decryption.
> > 
> >   Phase-2. Keys rotation:
> > 1. Key regeneration.
> > 
> > 2. CEK and MEK rotation.
> > 
> > Other phases to be added.
> > 
> > Thoughts or objections?
> > 
> > В Пн, 07/05/2018 в 10:19 +, Dmitry Pavlov пишет:
> > > Hi, just 2 remarks,
> > > 
> > > 1) We should somehow separate issue with disc corruption and incorrect
> > 
> > key.
> > > For incorrect key I suggest to adopt Key Check Value (KCV) techique. It
> > 
> > is
> > > some heading bytes (e.g. 3 bytes) of encrypted 00...00 block using this
> > > key. KCV allow us to check key decrypted correctly and check key before
> > > processing storage.
> > > 
> > > 2) I'm not sure about AES CBC, would it be applied only to one page, or
> > 
> > to
> > > whole store? We need random access to pages for example for page
> > > replacement process, so we need to reset CBC init vector (IV) to some
> > 
> > well
> > > known value. If we set IV to 0 for each page dectyption, it means CBC is
> > > applied only in scope of page. Continious IV accumulating CBC mode for
> > > whole store seems to be not working. Same question about IV is applicable
> > > to WAL records.
> > > 
> > > Could we consider ECB? or limit CBC with zero or well known IV?
> > > 
> > > Sincerely,
> > > Dmitriy Pavlov
> > > 
> > > сб, 5 мая 2018 г. в 17:49, Nikolay Izhikov :
> > > 
> > > > Hello, Guys.
> > > > 
> > > > Here are answers to the TDE design questions.
> > > > I will create FAQ in IEP-18 with this answers, also.
> > > > 
> > > > > 1. MEK, CEK rotation. Should we provide the way to change(regenerate)
> > > > 
> > > > MEK, CEK from time to time?
> > > > 
> > > > Yes. PCI DSS are require it. See 3.6.4, 3.6.5 sections.
> > > > 
> > > > > 2. Does CEK(table keys) relate to user access permission?
> > > > 
> > > > No. It is prohibited by PCI DSS. 3.4.1: "Decryption keys must not be
> > > > associated with user accounts"
> > > > 
> > > > > 3. WAL encryption. How will it be implemented?
> > > > 
> > > > LogicalRecord – key+value for encrypted cache entries are encrypted.
> > > > PhysicalRecord.FullPageSnaphot - regular page encryption algorithm
> > 
> > used.
> > > > PhysicalRecord.DeltaRecord - delta is encrypted.
> > > > 
> > > > > 4. We should keep CEKs in MetaStore.
> > > > 
> > > > Yes, we should.
> > > > We can't store CEKs in KeyStore because of PCI DSS 3.5.3.c:
> > > > "Key-encrypting keys are stored separately from data-encrypting keys."
> > > > 
> > > > > 5. How should we handle following case
> > > > 
> > > > I propose to not include the node to cluster with the appropriate
> > > > exception message.
> > > > 
> > > > > 6. Public API to deal with CEK should be provided.
> > > > 
> > > > The first draft of public API for TDE -
> > > > https://github.com/nizhikov/ignite/pull/12
> > > > 
> > > > > 7. Can each node use own CEK? What are pros and cons for that?
> > > > 
> > > > I propose to use one CEK per cache.
> > > > CEK would be the same on every cluster node.
> > > > It simplifies Ignite cluster management, backup procedure(like saving
> > 
> > key
> > > > copies on some external device), etc.
> > > > 
> > > > > 8. How we can ensure that decryption succeed? In case CEK is broken.
> > 
> > It
> > > > 
> > > > can be broken because of memory corruption, network errors, etc.
> > > > 
> > > > Currently, page integrity is checked by CRC.
> > > > I propose to reuse this procedure to ensure decryption integrity.
> > > > 
> > > > > 9. Specific encryption algorithm has to be chosen prior an
> > > > 
> > > > 

[jira] [Created] (IGNITE-8485) TDE - Phase-1

[Nikolay Izhikov (JIRA)]

Nikolay Izhikov created IGNITE-8485:
---

 Summary: TDE - Phase-1
 Key: IGNITE-8485
 URL: https://issues.apache.org/jira/browse/IGNITE-8485
 Project: Ignite
  Issue Type: Sub-task
Reporter: Nikolay Izhikov
Assignee: Nikolay Izhikov
 Fix For: 2.6


Basic support for a Transparent Data Encryption should be implemented:

1. Usage of standard JKS, Java Security.

2. Persistent Data Encryption/Decryption.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3988: IGNITE-5965 fix Flaky failure of GridServiceProce...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3988


---

Re: Transparent Data Encryption (TDE) in Apache Ignite

[Dmitry Pavlov]

Hi Nickolay,

Thank you for sharing results.

I would suggest to make phase 1 as small as possible, for example, skipping
WAL encryption or something like that.

It would not be full TDE implementation, but will allow us to move by small
steps, it also allows us to merge smaller changes to master.

Sincerely,
Dmitriy Pavlov

пн, 14 мая 2018 г. в 16:53, Nikolay Izhikov :

> Hello, guys.
>
> We had private discussion about TDE with Dmitriy Pavlov, Vladimir Ozerov
> and Anton Vinogradov.
>
> Some decisions was made I want to approve with communtiy:
>
>   1. Current design of TDE is OK. We can start work on implementation.
>
>   2. We should split implementation to phases.
>  So we can focus on basic features and deliver it to users.
>  And after it, extend it for a full compliance with enterprise
> standarts such as PCI DSS and others.
>
> I propose following plan:
>
>   Phase-1. Basic TDE:
> 1. Usage of standard JKS, Java Security.
>
> 2. Persistent Data Encryption/Decryption.
>
>   Phase-2. Keys rotation:
> 1. Key regeneration.
>
> 2. CEK and MEK rotation.
>
> Other phases to be added.
>
> Thoughts or objections?
>
> В Пн, 07/05/2018 в 10:19 +, Dmitry Pavlov пишет:
> > Hi, just 2 remarks,
> >
> > 1) We should somehow separate issue with disc corruption and incorrect
> key.
> > For incorrect key I suggest to adopt Key Check Value (KCV) techique. It
> is
> > some heading bytes (e.g. 3 bytes) of encrypted 00...00 block using this
> > key. KCV allow us to check key decrypted correctly and check key before
> > processing storage.
> >
> > 2) I'm not sure about AES CBC, would it be applied only to one page, or
> to
> > whole store? We need random access to pages for example for page
> > replacement process, so we need to reset CBC init vector (IV) to some
> well
> > known value. If we set IV to 0 for each page dectyption, it means CBC is
> > applied only in scope of page. Continious IV accumulating CBC mode for
> > whole store seems to be not working. Same question about IV is applicable
> > to WAL records.
> >
> > Could we consider ECB? or limit CBC with zero or well known IV?
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > сб, 5 мая 2018 г. в 17:49, Nikolay Izhikov :
> >
> > > Hello, Guys.
> > >
> > > Here are answers to the TDE design questions.
> > > I will create FAQ in IEP-18 with this answers, also.
> > >
> > > > 1. MEK, CEK rotation. Should we provide the way to change(regenerate)
> > >
> > > MEK, CEK from time to time?
> > >
> > > Yes. PCI DSS are require it. See 3.6.4, 3.6.5 sections.
> > >
> > > > 2. Does CEK(table keys) relate to user access permission?
> > >
> > > No. It is prohibited by PCI DSS. 3.4.1: "Decryption keys must not be
> > > associated with user accounts"
> > >
> > > > 3. WAL encryption. How will it be implemented?
> > >
> > > LogicalRecord – key+value for encrypted cache entries are encrypted.
> > > PhysicalRecord.FullPageSnaphot - regular page encryption algorithm
> used.
> > > PhysicalRecord.DeltaRecord - delta is encrypted.
> > >
> > > > 4. We should keep CEKs in MetaStore.
> > >
> > > Yes, we should.
> > > We can't store CEKs in KeyStore because of PCI DSS 3.5.3.c:
> > > "Key-encrypting keys are stored separately from data-encrypting keys."
> > >
> > > > 5. How should we handle following case
> > >
> > > I propose to not include the node to cluster with the appropriate
> > > exception message.
> > >
> > > > 6. Public API to deal with CEK should be provided.
> > >
> > > The first draft of public API for TDE -
> > > https://github.com/nizhikov/ignite/pull/12
> > >
> > > > 7. Can each node use own CEK? What are pros and cons for that?
> > >
> > > I propose to use one CEK per cache.
> > > CEK would be the same on every cluster node.
> > > It simplifies Ignite cluster management, backup procedure(like saving
> key
> > > copies on some external device), etc.
> > >
> > > > 8. How we can ensure that decryption succeed? In case CEK is broken.
> It
> > >
> > > can be broken because of memory corruption, network errors, etc.
> > >
> > > Currently, page integrity is checked by CRC.
> > > I propose to reuse this procedure to ensure decryption integrity.
> > >
> > > > 9. Specific encryption algorithm has to be chosen prior an
> > >
> > > implementation.
> > >
> > > AES CBC – 256, 192, 128 bits.
> > >
> > > > 10. Page integrity are checked by CRC. How this process would be
> changed
> > >
> > > for encrypted pages?
> > >
> > > The process doesn't change:
> > > 1. Decrypt page.
> > > 2. Check CRC.
> > >
> > > > 11. Page header has well-known content. This can be used for
> > >
> > > known-plain-text attacks. We should measure the treatment and find the
> way
> > > to deal with it.
> > >
> > > This question requires additional research.
> > > We have to understand: Is there any issue here?
> > > But at first glance, We can encrypt blocks in reverse order – from
> last to
> > > first.
> > > Last blocks don't have well know so there 

Re: Tests coverage report of Ignite 2.5 core module

[Dmitry Pavlov]

Ok, thank you. I missed that I should download the archive first.

I've found for example that method truncate in FSyncWalManager is not
covered.

Which test suites were executed?

пн, 14 мая 2018 г. в 17:16, Vyacheslav Daradur :

> Hi, Dmitry,
>
> Download and unpack the archive first:
>
> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip
>
>
> On Mon, May 14, 2018 at 5:12 PM, Dmitry Pavlov 
> wrote:
> > Hi Vyacheslav,
> >
> >
> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0_subpath=%2Fcoverage-report%2Findex.html
> >
> > says error 400 for me.
> >
> > Sincerely,
> > Dmitriy Pavlov
> >
> > пн, 14 мая 2018 г. в 17:06, Vyacheslav Daradur :
> >
> >> Hi, Igniters!
> >>
> >> I’ve made tests coverage report of core module using Jacoco 0.8.1
> >>
> >> I’d like to share results with the community. Here is a link:
> >>
> >>
> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0
> >>
> >> For viewing the report unpack the archive and open:
> >> coverage-report/index.html
> >>
> >> Also, there is ‘tests-classes.txt’ which includes names of tests
> >> classes which have been used.
> >>
> >> The report was generated from branch: ‘ignite-2-5’, commit:
> >>
> >>
> https://github.com/apache/ignite/tree/d83f1ec0a4d010e57fecc12ffd3d1f8346ded61c
> >>
> >>
> >> --
> >> Best Regards, Vyacheslav D.
> >>
>
>
>
> --
> Best Regards, Vyacheslav D.
>

Re: Tests coverage report of Ignite 2.5 core module

[Vyacheslav Daradur]

Hi, Dmitry,

Download and unpack the archive first:
https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip


On Mon, May 14, 2018 at 5:12 PM, Dmitry Pavlov  wrote:
> Hi Vyacheslav,
>
> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0_subpath=%2Fcoverage-report%2Findex.html
>
> says error 400 for me.
>
> Sincerely,
> Dmitriy Pavlov
>
> пн, 14 мая 2018 г. в 17:06, Vyacheslav Daradur :
>
>> Hi, Igniters!
>>
>> I’ve made tests coverage report of core module using Jacoco 0.8.1
>>
>> I’d like to share results with the community. Here is a link:
>>
>> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0
>>
>> For viewing the report unpack the archive and open:
>> coverage-report/index.html
>>
>> Also, there is ‘tests-classes.txt’ which includes names of tests
>> classes which have been used.
>>
>> The report was generated from branch: ‘ignite-2-5’, commit:
>>
>> https://github.com/apache/ignite/tree/d83f1ec0a4d010e57fecc12ffd3d1f8346ded61c
>>
>>
>> --
>> Best Regards, Vyacheslav D.
>>



-- 
Best Regards, Vyacheslav D.

[GitHub] ignite pull request #3976: IGNITE-8456: Print warning if IGNITE_HOME & WORK ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3976


---

Re: Tests coverage report of Ignite 2.5 core module

[Dmitry Pavlov]

Hi Vyacheslav,

https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0_subpath=%2Fcoverage-report%2Findex.html

says error 400 for me.

Sincerely,
Dmitriy Pavlov

пн, 14 мая 2018 г. в 17:06, Vyacheslav Daradur :

> Hi, Igniters!
>
> I’ve made tests coverage report of core module using Jacoco 0.8.1
>
> I’d like to share results with the community. Here is a link:
>
> https://www.dropbox.com/s/rdgs1svvojm757x/ignite-2.5-core-module-tests-coverage-report-rev-d83f1ec.zip?dl=0
>
> For viewing the report unpack the archive and open:
> coverage-report/index.html
>
> Also, there is ‘tests-classes.txt’ which includes names of tests
> classes which have been used.
>
> The report was generated from branch: ‘ignite-2-5’, commit:
>
> https://github.com/apache/ignite/tree/d83f1ec0a4d010e57fecc12ffd3d1f8346ded61c
>
>
> --
> Best Regards, Vyacheslav D.
>

[jira] [Created] (IGNITE-8484) Web Console lacks 'collocated' flag on SQL query tab

[Ilya Kasnacheev (JIRA)]

Ilya Kasnacheev created IGNITE-8484:
---

 Summary: Web Console lacks 'collocated' flag on SQL query tab
 Key: IGNITE-8484
 URL: https://issues.apache.org/jira/browse/IGNITE-8484
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 2.3
Reporter: Ilya Kasnacheev


For some queries, this flag is needed to avoid endless running and/or memory 
depletion.

There's setEnforceJoinOrder() flag and setDistributedJoins() flag but not 
setCollocated() flag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: async operation is not fair async

[Alexey Goncharuk]

Another +1 for the true asynchronous approach. I remember a while ago one
of the Ignite users raised a similar question regarding the *async method
being blocked on establishing a TCP connection.

As far as deadlocks go, I have a counter-example. Currently, we check the
thread-local chain only for a single cache, so if I run the following code:
cache1.getAsync(k1);
cache2.getAsync(k2);
then the deadlock is still possible, and I did not see a single user
complaining about unexpected deadlocks. Rather than implementing this
cross-cache chain (which would probably add another overhead), I would make
it consistent and allow operations to be run in parallel.

There are many use-cases when having true async operations dramatically
improve performance. Consider, for example, a streaming example when keys
are being pushed by a client to a cluster. Currently, to run effective
processing, the user will have to use a data streamer with custom keys
receiver which may be a huge usability downside. Async operations can
utilize the cluster resources very efficiently.

Finally, if we want to be on the safe side, we can keep the operation chain
inside a transaction. I see absolutely no point in maintaining this chain
outside of transactions.

--AG

2018-05-14 16:01 GMT+03:00 Dmitriy Govorukhin 
:

> Andrey,
>
> Do you prefer change behavior at runtime?
> I guess will be better have different methods for getting cache instance
> with fair and not fair sync.
>
> On Mon, May 14, 2018 at 3:39 PM, Andrey Gura  wrote:
>
> > +1 for fair async operations.
> >
> > But I don't like idea use withFairSync() method. We added xxxAsync()
> > methods recently and withAsync() is deprecated.
> >
> > I think we should just make methods are async in nature and provide
> > ability of switching to the old behaviour using flag or property.
> >
> > On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
> >  wrote:
> > > Vladimir,
> > >
> > > In general I agree, but I do get greatly *close-minded* (pun intended)
> > > whenever users' code that worked for the past several years all of a
> > sudden
> > > gets deadlocked after an upgrade. Making this feature optional is even
> > > worse and more confusing. In this case the best action is no action at
> > all.
> > >
> > > BTW, would be interesting to find out how Oracle async driver behaves
> in
> > > this case.
> > >
> > > D.
> > >
> > >
> > >
> > >
> > >
> > > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov  >
> > > wrote:
> > >
> > >> Guys,
> > >>
> > >> To build a great product we should be open minded and look to the
> > future,
> > >> not to the past.
> > >>
> > >> Dima raised very valid point - why async is not async? Current
> > programming
> > >> culture and demanding performance requirements pushes users towards
> > >> reactive-style programming. I do not want my thread to ever be
> blocked.
> > >> Instead, I want to send a number of concurrent commands and optionally
> > >> subscribe to final result. So trully async API makes total sense to
> me.
> > >>
> > >> But personally, my primary interest in this area is SQL. Oracle is
> > >> preparing new async driver. ADBA - async database access. It was
> > presented
> > >> on recent JavaOne [1]. It is under active development right now - juse
> > >> weave through the mailing list [2]. Some prototypes are already there
> > [3].
> > >> PostgreSQL community even started adopted it [4]!
> > >>
> > >> I am not pushing for immediate actions, but at least we should
> > understand
> > >> which way the wind is blowing. As a mid-term goals I would propose to
> > >> finally remove thread ID from our PESSIMISTIC transactions to allow
> for
> > >> suspend/resume in different threads. And as a next step I would think
> on
> > >> adopting async cache and SQL APIs.
> > >>
> > >> Vladimir.
> > >>
> > >> [1]
> > >> http://www.oracle.com/technetwork/database/
> > application-development/jdbc/
> > >> con1491-3961036.pdf
> > >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> > >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> > >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> > >>
> > >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > >> wrote:
> > >>
> > >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> > >> > dmitriy.govoruk...@gmail.com> wrote:
> > >> >
> > >> > > I will edit IGNITE-8475, and remove all part that belong to the
> > public
> > >> > api.
> > >> > > Is it acceptable for you?
> > >> > >
> > >> >
> > >> > Everything is acceptable, as long as the public API is safe :)
> > >> >
> > >>
> >
>

[GitHub] ignite pull request #3993: IGNITE-8238: Operation can fails with unexpected ...

[SharplEr]

GitHub user SharplEr opened a pull request:

https://github.com/apache/ignite/pull/3993

IGNITE-8238: Operation can fails with unexpected RuntimeException when node 
is stopping.

For [IGNITE-8238](https://issues.apache.org/jira/browse/IGNITE-8238)

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/SharplEr/ignite ignite-8238

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3993.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3993






---

[GitHub] ignite pull request #2776: IGNITE-6502: Print warning if -Djava.net.preferIP...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/2776


---

Re: async operation is not fair async

[Dmitriy Govorukhin]

Andrey,

Do you prefer change behavior at runtime?
I guess will be better have different methods for getting cache instance
with fair and not fair sync.

On Mon, May 14, 2018 at 3:39 PM, Andrey Gura  wrote:

> +1 for fair async operations.
>
> But I don't like idea use withFairSync() method. We added xxxAsync()
> methods recently and withAsync() is deprecated.
>
> I think we should just make methods are async in nature and provide
> ability of switching to the old behaviour using flag or property.
>
> On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
>  wrote:
> > Vladimir,
> >
> > In general I agree, but I do get greatly *close-minded* (pun intended)
> > whenever users' code that worked for the past several years all of a
> sudden
> > gets deadlocked after an upgrade. Making this feature optional is even
> > worse and more confusing. In this case the best action is no action at
> all.
> >
> > BTW, would be interesting to find out how Oracle async driver behaves in
> > this case.
> >
> > D.
> >
> >
> >
> >
> >
> > On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov 
> > wrote:
> >
> >> Guys,
> >>
> >> To build a great product we should be open minded and look to the
> future,
> >> not to the past.
> >>
> >> Dima raised very valid point - why async is not async? Current
> programming
> >> culture and demanding performance requirements pushes users towards
> >> reactive-style programming. I do not want my thread to ever be blocked.
> >> Instead, I want to send a number of concurrent commands and optionally
> >> subscribe to final result. So trully async API makes total sense to me.
> >>
> >> But personally, my primary interest in this area is SQL. Oracle is
> >> preparing new async driver. ADBA - async database access. It was
> presented
> >> on recent JavaOne [1]. It is under active development right now - juse
> >> weave through the mailing list [2]. Some prototypes are already there
> [3].
> >> PostgreSQL community even started adopted it [4]!
> >>
> >> I am not pushing for immediate actions, but at least we should
> understand
> >> which way the wind is blowing. As a mid-term goals I would propose to
> >> finally remove thread ID from our PESSIMISTIC transactions to allow for
> >> suspend/resume in different threads. And as a next step I would think on
> >> adopting async cache and SQL APIs.
> >>
> >> Vladimir.
> >>
> >> [1]
> >> http://www.oracle.com/technetwork/database/
> application-development/jdbc/
> >> con1491-3961036.pdf
> >> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
> >> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
> >> [4] https://github.com/pgjdbc/pgjdbc/issues/978
> >>
> >> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> >> wrote:
> >>
> >> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
> >> > dmitriy.govoruk...@gmail.com> wrote:
> >> >
> >> > > I will edit IGNITE-8475, and remove all part that belong to the
> public
> >> > api.
> >> > > Is it acceptable for you?
> >> > >
> >> >
> >> > Everything is acceptable, as long as the public API is safe :)
> >> >
> >>
>

Re: Apache Ignite 2.5 release

[Andrey Gura]

Igniters,

we are ready to start process of releasing Apache Ignite 2.5.

So code freeze. No more commits to ignite-2.5 branch.

Thanks!


On Thu, May 10, 2018 at 8:02 PM, Andrey Gura  wrote:
> Igniters,
>
> Almost all issues for Apache Ignite 2.5 release are fixed. Issues that
> are still in Open state will be moved to 2.6 release.
>
> Code freeze for ignite-2.5 branch is planed for tomorrow.
>
> Thanks!
>
> On Tue, May 8, 2018 at 7:09 PM, Andrey Gura  wrote:
>> Thanks for fast update, Ivan.
>>
>> On Tue, May 8, 2018 at 5:59 PM, Ivan Rakov  wrote:
>>> Andrey,
>>>
>>> Sorry to disturb you again. I have a follow up fix for IGNITE-8429 -
>>> https://issues.apache.org/jira/browse/IGNITE-8453.
>>> Please merge it to master and ignite-2.5.
>>>
>>> Best Regards,
>>> Ivan Rakov
>>>
>>>
>>> On 08.05.2018 15:29, Vladimir Ozerov wrote:

 H2 version updated is moved to AI 2.6. There are several serious problems,
 main - spatial dependency replacement which is a breaking change.

 On Tue, May 8, 2018 at 10:17 AM, Pavel Tupitsyn 
 wrote:

> IGNITE-8434 merged to master, cherry-picked to ignite-2.5.
>
> Thank you, Andrey!
>
> On Mon, May 7, 2018 at 10:46 PM, Andrey Gura  wrote:
>
>> Pavel,
>>
>> issue is targeted to release 2.5 now. Thanks!
>>
>> On Mon, May 7, 2018 at 10:40 PM, Pavel Tupitsyn 
>> wrote:
>>>
>>> Igniters,
>>>
>>> There is an unpleasant bug in .NET [1]:
>>> Services do not work at all on .NET Core and sometimes on classic .NET.
>>>
>>> I'm almost done with the fix and I think we should include it in 2.5.
>>
>> Users
>>>
>>> are complaining [2].
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/IGNITE-8434
>>> [2]
>>> http://apache-ignite-users.70518.x6.nabble.com/Re-
>>
>> Service-Grid-and-net-core-2-td21318.html
>>>
>>> On Mon, May 7, 2018 at 6:44 PM, Vladimir Ozerov 
>>> wrote:
>>>
 Andrey,

 I realized that we missed one very important thing - H2 version
>
> upgrade

 [1]. This is very important thing because new version contain
>>
>> optimization

 to IN clause processing. All we need is to upgrade H2 vesion from
>>
>> 1.4.196

 to 1.4.197 and re-run tests. Can we target it to 2.5 release?

 [1] https://issues.apache.org/jira/browse/IGNITE-4150

 On Mon, May 7, 2018 at 6:07 PM, Andrey Gura  wrote:

> Ivan,
>
> ok, targeted to 2.5. Thanks!
>
> On Mon, May 7, 2018 at 5:24 PM, Dmitry Pavlov <
>
> dpavlov@gmail.com>
>
> wrote:
>>
>> Thank you, Andrey,
>>
>> I saw 1 more proposal in separate topic
>> http://apache-ignite-developers.2346864.n4.nabble.
>
> com/RE-Hi-can-we-squeeze-this-into-2-5-https-issues-apache-
> org-jira-browse-IGNITE-8439-td30161.html
>>
>> and 1 proposal here related to ML:
>> http://apache-ignite-developers.2346864.n4.nabble.
>
> com/Apache-Ignite-2-5-release-td28927i20.html#a30155
>>
>> but here Yuriy Babak probably should share his opinion.
>>
>> Sincerely,
>> Dmitriy Pavlov
>>
>> пн, 7 мая 2018 г. в 17:09, Andrey Gura :
>>>
>>> Sergey, Dmitry,
>>>
>>> I've targeted your issues to 2.5 release.
>>>
>>> Thanks!
>>>
>>> On Fri, May 4, 2018 at 5:51 PM, Dmitry Pavlov <
>>
>> dpavlov@gmail.com>
>>>
>>> wrote:

 I did not understand process.

 Is it required to receive someone reply that issue is agreed to
>>
>> be

 included
 into release?

 Or because there is no response I can cherry-pick anything to
>
> 2.5
>
> hoping

 it
 is lazy consensus; no one objected.

 пт, 4 мая 2018 г. в 17:44, Denis Magda :

> Igniters,
>
> I got lost. Were all the bugs reported in this thread fixed
>
> and
>
> merged
>
> into
> 2.5 branch? Is there any bug which fix is still in progress?
>
> Just want to know when the QA and bugs fixing stage is over.
>
> --
> Denis
>
> On Thu, May 3, 2018 at 9:14 AM, Dmitry Pavlov <

 dpavlov@gmail.com
>
> wrote:
>

Re: tracking SQL queries in 2.5

[Vladimir Ozerov]

Hi Dima,

This is not that simple unfortunately. We need to build the whole
infrastructure for cache-less SQL queries. This is not very complex, but
require considerable efforts. Let's do that in AI 2.6 scope.

On Sat, May 12, 2018 at 4:36 PM, Dmitriy Setrakyan 
wrote:

> Igniters,
>
> I just noticed this horrible ticket:
> https://issues.apache.org/jira/browse/IGNITE-6677
>
> Apparently, a couple of releases back the community added support for
> "CREATE TABLE" command and for some reason nobody checked that query
> metrics on the created tables do not work.
>
> Do we have any idea of what it would take to fix? If it is a simple fix, I
> would like to squeeze it into 2.5 release.
>
> D.
>

Re: async operation is not fair async

[Andrey Gura]

+1 for fair async operations.

But I don't like idea use withFairSync() method. We added xxxAsync()
methods recently and withAsync() is deprecated.

I think we should just make methods are async in nature and provide
ability of switching to the old behaviour using flag or property.

On Fri, May 11, 2018 at 11:00 PM, Dmitriy Setrakyan
 wrote:
> Vladimir,
>
> In general I agree, but I do get greatly *close-minded* (pun intended)
> whenever users' code that worked for the past several years all of a sudden
> gets deadlocked after an upgrade. Making this feature optional is even
> worse and more confusing. In this case the best action is no action at all.
>
> BTW, would be interesting to find out how Oracle async driver behaves in
> this case.
>
> D.
>
>
>
>
>
> On Fri, May 11, 2018 at 8:29 PM, Vladimir Ozerov 
> wrote:
>
>> Guys,
>>
>> To build a great product we should be open minded and look to the future,
>> not to the past.
>>
>> Dima raised very valid point - why async is not async? Current programming
>> culture and demanding performance requirements pushes users towards
>> reactive-style programming. I do not want my thread to ever be blocked.
>> Instead, I want to send a number of concurrent commands and optionally
>> subscribe to final result. So trully async API makes total sense to me.
>>
>> But personally, my primary interest in this area is SQL. Oracle is
>> preparing new async driver. ADBA - async database access. It was presented
>> on recent JavaOne [1]. It is under active development right now - juse
>> weave through the mailing list [2]. Some prototypes are already there [3].
>> PostgreSQL community even started adopted it [4]!
>>
>> I am not pushing for immediate actions, but at least we should understand
>> which way the wind is blowing. As a mid-term goals I would propose to
>> finally remove thread ID from our PESSIMISTIC transactions to allow for
>> suspend/resume in different threads. And as a next step I would think on
>> adopting async cache and SQL APIs.
>>
>> Vladimir.
>>
>> [1]
>> http://www.oracle.com/technetwork/database/application-development/jdbc/
>> con1491-3961036.pdf
>> [2] http://mail.openjdk.java.net/pipermail/jdbc-spec-discuss/
>> [3] https://github.com/oracle/oracle-db-examples/tree/master/java/AoJ
>> [4] https://github.com/pgjdbc/pgjdbc/issues/978
>>
>> On Fri, May 11, 2018 at 9:48 PM, Dmitriy Setrakyan 
>> wrote:
>>
>> > On Fri, May 11, 2018 at 7:46 PM, Dmitriy Govorukhin <
>> > dmitriy.govoruk...@gmail.com> wrote:
>> >
>> > > I will edit IGNITE-8475, and remove all part that belong to the public
>> > api.
>> > > Is it acceptable for you?
>> > >
>> >
>> > Everything is acceptable, as long as the public API is safe :)
>> >
>>

[jira] [Created] (IGNITE-8482) Skip 2-phase partition release wait in case of activation or dynamic caches start

[Pavel Kovalenko (JIRA)]

Pavel Kovalenko created IGNITE-8482:
---

 Summary: Skip 2-phase partition release wait in case of activation 
or dynamic caches start
 Key: IGNITE-8482
 URL: https://issues.apache.org/jira/browse/IGNITE-8482
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Affects Versions: 2.5
Reporter: Pavel Kovalenko
Assignee: Pavel Kovalenko
 Fix For: 2.6


Currently we perform 2-phase partitions release waiting on any type of 
distributed exchange. We can optimize this behaviour, skipping such waiting on 
cluster activation (obviously if we activate cluster it means that before 
activation no caches were running and there is no reason to wait for finishing 
operations) and on dynamic caches start.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-8483) Per partition TTL pending tree for in-memory caches.

[Andrew Mashenkov (JIRA)]

Andrew Mashenkov created IGNITE-8483:


 Summary: Per partition TTL pending tree for in-memory caches.
 Key: IGNITE-8483
 URL: https://issues.apache.org/jira/browse/IGNITE-8483
 Project: Ignite
  Issue Type: Improvement
  Components: cache
Reporter: Andrew Mashenkov


For in memory cache pending tree root page is allocated when ignite store is 
initializing even if no ExpiryPolicy is used. 
 We have to implement lazy PendingTree initialization and switch to 
per-partition based PendingTree as we've done for persistent caches.

Switching to per-partition based PendingTree without lazy initialization will 
bump us with 1024 unevictable pages per CacheGroup if no ExpiryPolicy is used.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3992: IGNITE-8469: add memory leak test case

[Mmuzaf]

GitHub user Mmuzaf opened a pull request:

https://github.com/apache/ignite/pull/3992

IGNITE-8469: add memory leak test case



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Mmuzaf/ignite leak-ignite-8469

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3992.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3992


commit 2c319879f2d79c72c57c90539fd0ce16a8b37e5a
Author: Maxim Muzafarov 
Date:   2018-05-14T12:20:33Z

IGNITE-8469: add memory leak test case




---

[GitHub] ignite pull request #3991: 50 time run test

[voipp]

GitHub user voipp opened a pull request:

https://github.com/apache/ignite/pull/3991

50 time run test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/voipp/ignite IGNITE-8161

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3991.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3991


commit 9fbff820c89327d8db6c371386275565f6a18d54
Author: voipp 
Date:   2018-05-14T11:51:07Z

50 time run test




---

Re: Integration with JHipster

[Andrey Gura]

Hi,

Ignite already can be used as Spring Cache and Hibernate L2 cache.
Only this two options are provided by JHipster.
So JHipster documentaton update should be enough.

On Sat, May 12, 2018 at 7:09 PM, Dmitriy Setrakyan
 wrote:
> Igniters,
>
> Jhipster is a very popular Spring Boot + Angular framework. Many caching
> frameworks integrate with it, including EhCache, Hazelcast, and Infinispan:
>
> https://www.jhipster.tech/using-cache/
>
> I think the integration is fairly simple and we should provide our own.
>
> Thoughts?
>
> D.

[jira] [Created] (IGNITE-8481) VisorValidateIndexesJob works very slowly in case of many partitions/keys for each partition.

[Alexei Scherbakov (JIRA)]

Alexei Scherbakov created IGNITE-8481:
-

 Summary: VisorValidateIndexesJob works very slowly in case of many 
partitions/keys for each partition.
 Key: IGNITE-8481
 URL: https://issues.apache.org/jira/browse/IGNITE-8481
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.5
Reporter: Alexei Scherbakov
 Fix For: 2.6
 Attachments: ignite.zip, thrdump-server.log

I tried to validate indexes using newly introduced VisorValidateIndexesTask 
from control.sh and found what on large data set it works very slowly. Process 
was not finished for 12 hours from start.

Looking through a thread dump I've noticed following problems:

1. ValidateIndexesClosure works not in optimal way by doing btree lookup for 
each index for each entry of each partition. It should be faster to validate by 
scanning index tree.

2. Thread dump shows contention on acquiring segment read lock by worker 
pool-XXX threads, but no obvious reason for holding write lock (no load on grid)

3. 
org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.Segment#partGeneration
 generates garbage on each page access.

Check attachment for log and thread dump.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3990: IGNITE-8474 skipForExchangeMerge property changed...

[sergey-chugunov-1985]

GitHub user sergey-chugunov-1985 opened a pull request:

https://github.com/apache/ignite/pull/3990

IGNITE-8474 skipForExchangeMerge property changed to 'true'



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8474

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3990.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3990


commit 853908d1f9eaaeea9dbbf8dbc1f63c02fc59b4a0
Author: Sergey Chugunov 
Date:   2018-05-14T10:52:58Z

IGNITE-8474 skipForExchangeMerge property changed to 'true'




---

Re: SqlLine script and Java9

[Petr Ivanov]

Tested, review and merged to master and ignite-2.5 branches (thanks to Andrey 
Gura for operativeness).



> On 14 May 2018, at 11:40, Petr Ivanov  wrote:
> 
> Prepared patch [1], review and testing is required.
> 
> 
> [1] https://github.com/apache/ignite/pull/3989 
> 
> 
> 
> 
>> On 14 May 2018, at 10:35, Petr Ivanov > > wrote:
>> 
>> Filed the ticket [1], will fix.
>> 
>> 
>> [1] https://issues.apache.org/jira/browse/IGNITE-8478 
>> 
>> 
>>> On 12 May 2018, at 20:10, Dmitriy Setrakyan >> > wrote:
>>> 
>>> Igniters,
>>> 
>>> Any reason SqlLine tool shipped with Ignite does not work with Java 9? This
>>> is the error I got.
>>> 
>>> 
 *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
 incorrect.*
 *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 
 1.8.**You
 can also download latest JDK at http://java.com/download 
 
 >.*
>>> 
>>> 
>>> D.
>> 
> 

[jira] [Created] (IGNITE-8480) Broken Ignite Examples test Support Spring Data 2.0

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-8480:
--

 Summary: Broken Ignite Examples test Support Spring Data 2.0
 Key: IGNITE-8480
 URL: https://issues.apache.org/jira/browse/IGNITE-8480
 Project: Ignite
  Issue Type: Improvement
Reporter: Dmitriy Pavlov
Assignee: Roman Meerson


https://issues.apache.org/jira/browse/IGNITE-6879 implementation was introduced 
support of spring data of version 2.0+ 

 

But example test is broken on TC after this change

[Examples 
|https://ci.ignite.apache.org/viewType.html?buildTypeId=IgniteTests24Java8_Examples=%3Cdefault%3E=buildTypeStatusDiv]
  IgniteExamplesSelfTestSuite: 
SpringDataExampleSelfTest.testSpringDataExample

 

Test history: 
[https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8=-9052304574499269123=%3Cdefault%3E=testDetails]

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3775: IGNITE-8138 Uptime output with days

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3775


---

[GitHub] ignite pull request #3972: Ignite 8138 2

[dspavlov]

Github user dspavlov closed the pull request at:

https://github.com/apache/ignite/pull/3972


---

Re: NodeJS thin client: full API

[Alexey Kosenchuk]

- Let's rename SqlDataProcessingExample.js to just SqlExample.js and
describe it as an example that shows primary APIs to use with Ignite as
with an SQL database.
- Let's rename SqlQueryExample.js to SqlQueryEntiriesExample.js.


Updated.

[GitHub] ignite pull request #3975: IGNITE-8422 Improve tests

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3975


---

[jira] [Created] (IGNITE-8479) Web console: query text is empty on opening query section

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-8479:
--

 Summary: Web console: query text is empty on opening query section
 Key: IGNITE-8479
 URL: https://issues.apache.org/jira/browse/IGNITE-8479
 Project: Ignite
  Issue Type: Bug
Reporter: Pavel Konstantinov
 Attachments: image-2018-05-14-16-39-34-686.png

On the first second the query text is empty and then it appears.
 !image-2018-05-14-16-39-34-686.png! 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #3692: Suspend resume benchmark

[voipp]

Github user voipp closed the pull request at:

https://github.com/apache/ignite/pull/3692


---

Re: SqlLine script and Java9

[Petr Ivanov]

Prepared patch [1], review and testing is required.


[1] https://github.com/apache/ignite/pull/3989 




> On 14 May 2018, at 10:35, Petr Ivanov  wrote:
> 
> Filed the ticket [1], will fix.
> 
> 
> [1] https://issues.apache.org/jira/browse/IGNITE-8478 
> 
> 
>> On 12 May 2018, at 20:10, Dmitriy Setrakyan > > wrote:
>> 
>> Igniters,
>> 
>> Any reason SqlLine tool shipped with Ignite does not work with Java 9? This
>> is the error I got.
>> 
>> 
>>> *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
>>> incorrect.*
>>> *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 1.8.**You
>>> can also download latest JDK at http://java.com/download 
>>> 
>>> >.*
>> 
>> 
>> D.
> 

[GitHub] ignite pull request #3989: IGNITE-8478 SqlLine run failure on Java 9

[vveider]

GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/3989

IGNITE-8478 SqlLine run failure on Java 9



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-8478

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3989.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3989


commit cf98ea393e269e4985a449c5871b47280f56b891
Author: Ivanov Petr 
Date:   2018-05-14T08:20:48Z

IGNITE-8478 SqlLine run failure on Java 9




---

[GitHub] ignite pull request #2975: Ignite-6083 Null value has appeared in entry proc...

[voipp]

Github user voipp closed the pull request at:

https://github.com/apache/ignite/pull/2975


---

Re: SqlLine script and Java9

[Petr Ivanov]

Filed the ticket [1], will fix.


[1] https://issues.apache.org/jira/browse/IGNITE-8478 


> On 12 May 2018, at 20:10, Dmitriy Setrakyan  wrote:
> 
> Igniters,
> 
> Any reason SqlLine tool shipped with Ignite does not work with Java 9? This
> is the error I got.
> 
> 
>> *The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
>> incorrect.*
>> *Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 1.8.**You
>> can also download latest JDK at http://java.com/download
>> .*
> 
> 
> D.

[jira] [Created] (IGNITE-8478) SqlLine run failure on Java 9

[Peter Ivanov (JIRA)]

Peter Ivanov created IGNITE-8478:


 Summary: SqlLine run failure on Java 9
 Key: IGNITE-8478
 URL: https://issues.apache.org/jira/browse/IGNITE-8478
 Project: Ignite
  Issue Type: Task
Reporter: Peter Ivanov
Assignee: Peter Ivanov


{code}
*The version of JAVA installed in C:\Program Files\Java\jdk-9.0.1 is
incorrect.*
*Please point JAVA_HOME variable to installation of JDK 1.7 or JDK 1.8.**You
can also download latest JDK at http://java.com/download
.*
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)