date:20180911

Hello, Denis.

> Could you please list the limitations of Phase 1?
> I'm curious what won't be supported in 2.7.

1. We will have ability to encrypt data stored on the disk for specific cache.

- There is no limitation on API usage or something for an encrypted 
cache.
- If some cache in a cache group is encrypted, all other caches in this 
group must be encrypted.
- Setting up master key(standard jdk key storage) is prerequisite and 
should be done by an administrator.
- `encryptionEnabled` flag setting up on cache creation and can't be 
changed in runtime.

2. We won't be able to change encryption keys for existing cache(key rotation 
procedure).
This will be implemented in Phase 2.

В Вт, 11/09/2018 в 23:41 -0400, Denis Magda пишет:
> Nikolay,
> 
> Could you please list the limitations of Phase 1? I'm curious what won't be
> supported in 2.7.
> 
> --
> Denis
> 
> On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov  wrote:
> 
> > > As I understand the plan is to get TDE Phase 1 released in 2.7, right?
> > 
> > Yes. We will release TDE in 2.7
> > 
> > > Could you also list what needs to be done at Phase 2 and how much time
> > 
> > it might take.
> > 
> > Yes, I think Dmitry Ryabov will send Phase 2 design
> > 
> > 
> > В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > > Hello, Denis.
> > > 
> > > Yes, Vladimir made 2 rounds of review.
> > > I planning to fix all issues with implementation in a few days.
> > > 
> > > 
> > > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > > > Hi Nikolay,
> > > > 
> > > > Has anybody started looking into your request? As I understand the
> > 
> > plan is
> > > > to get TDE Phase 1 released in 2.7, right?
> > > > 
> > 
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > > > 
> > > > Could you also list what needs to be done at Phase 2 and how much time
> > 
> > it
> > > > might take.
> > > > 
> > > > --
> > > > Denis
> > > > 
> > > > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov 
> > 
> > wrote:
> > > > 
> > > > > Hello, Igniters.
> > > > > 
> > > > > I want to share with you TDE implementation details.
> > > > > I think it can simplify review and acception of TDE implementation.
> > > > > This mail is copy of wiki page [1].
> > > > > 
> > > > > Please, share your thoughts.
> > > > > 
> > > > > TDE is ready for review [2].
> > > > > Please, let me know, who is able to make final review.
> > > > > 
> > > > > This document describes some internal details of TDE.Phase 1
> > > > > implementation [3].
> > > > > I suggest that reader familiar with the general design described in
> > 
> > IEP-18
> > > > > [4].
> > > > > 
> > > > > 
> > > > > Cache group key management and node join enhancements:
> > > > > 
> > > > > 1. GridEncryptionManager encapsulates all logic related to
> > 
> > key
> > > > > management:
> > > > > a. All group encryption keys are stored in MetaStore.
> > > > > 
> > > > > b. Joining node sends to cluster:
> > > > > * Master key digest.
> > > > > * All group keys saved locally (Map > > > > byte[]>). Keys send over a network in encrypted form.
> > > > > 
> > > > > c. Coordinator on new node join:
> > > > > * Compares new node master key digest with
> > 
> > the
> > > > > local master key digest.
> > > > > If differs then new node join is rejected.
> > > > > 
> > > > > * Compares local and received group keys.
> > > > > If some key differs then new node join is
> > > > > rejected.
> > > > > 
> > > > > d. All server nodes:
> > > > > * If some of received keys are new then they
> > 
> > save
> > > > > locally.
> > > > > 
> > > > > 2. Dynamic cache creation:
> > > > > a. On server node - Encryption key is generated and
> > 
> > added
> > > > > to DynamicCacheChangeRequest.
> > > > > 
> > > > > b. On client node:
> > > > > * Prior to creation of
> > 
> > DynamicCacheChangeRequest
> > > > > we have to get new encryption key from some server node.
> > > > > * New request added to solve this -
> > > > > GenerateEncryptionKeyRequest.
> > > > > 
> > > > > 
> > > > > WAL Record encryption:
> > > > > 
> > > > > 
> > > > > 1. New type of WAL record created – EncryptedRecord.
> > > > > 
> > > > > 2. EncryptedRecord is a container that stores some
> > > > > WalRecordCacheGroupAware in encrypted form inside.
> > > > > 
> > > > > 3. Write:
> > > > > Each record for an encrypted group that implements
> > > > > WalRecordCacheGroupAware written to WAL in encrypted form.
> > > > > Instead of that record we write EncryptedRecord with plain
> > 
> > record
> > > > > inside(PageSnapshot, PageDeltaRecord, etc).
> > > > > 
> > > > > 4. Rea

[jira] [Created] (IGNITE-9555) Web Console can stop due to an unexpected message via web sockets

2018-09-11 Thread Roman Guseinov (JIRA)

Roman Guseinov created IGNITE-9555:
--

 Summary: Web Console can stop due to an unexpected message via web 
sockets
 Key: IGNITE-9555
 URL: https://issues.apache.org/jira/browse/IGNITE-9555
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Reporter: Roman Guseinov
 Attachments: webconsole.log

Web Console shows error only if debug messages are enabled `DEBUG=mongodb-* 
./gridgain-web-console-linux`. Otherwise, it just stops.

Here is the error message:
{code:java}
/snapshot/backend/node_modules/ignite-web-console/app/browsersHandler.js:211
return cb('Invalid format of message: "node:rest"');
^

TypeError: cb is not a function
at Socket.module.exports.factory.nodeListeners.sock.on 
(/snapshot/backend/node_modules/ignite-web-console/app/browsersHandler.js:211:32)
at emitThree (events.js:136:13)
at Socket.emit (events.js:217:7)
at 
/snapshot/backend/node_modules/ignite-web-console/node_modules/socket.io/lib/socket.js:503:12
at _combinedTickCallback (internal/process/next_tick.js:131:7)
at process._tickCallback (internal/process/next_tick.js:180:9)
{code}
The full log is attached: [^webconsole.log].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4618: IGNITE-9311: add override annotation

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4618


---

Re: TDE Implementation details.

2018-09-11 Thread Denis Magda

Nikolay,

Could you please list the limitations of Phase 1? I'm curious what won't be
supported in 2.7.

--
Denis

On Tue, Sep 11, 2018 at 4:29 PM Nikolay Izhikov  wrote:

> > As I understand the plan is to get TDE Phase 1 released in 2.7, right?
>
> Yes. We will release TDE in 2.7
>
> > Could you also list what needs to be done at Phase 2 and how much time
> it might take.
>
> Yes, I think Dmitry Ryabov will send Phase 2 design
>
>
> В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> > Hello, Denis.
> >
> > Yes, Vladimir made 2 rounds of review.
> > I planning to fix all issues with implementation in a few days.
> >
> >
> > В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > > Hi Nikolay,
> > >
> > > Has anybody started looking into your request? As I understand the
> plan is
> > > to get TDE Phase 1 released in 2.7, right?
> > >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > >
> > > Could you also list what needs to be done at Phase 2 and how much time
> it
> > > might take.
> > >
> > > --
> > > Denis
> > >
> > > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov 
> wrote:
> > >
> > > > Hello, Igniters.
> > > >
> > > > I want to share with you TDE implementation details.
> > > > I think it can simplify review and acception of TDE implementation.
> > > > This mail is copy of wiki page [1].
> > > >
> > > > Please, share your thoughts.
> > > >
> > > > TDE is ready for review [2].
> > > > Please, let me know, who is able to make final review.
> > > >
> > > > This document describes some internal details of TDE.Phase 1
> > > > implementation [3].
> > > > I suggest that reader familiar with the general design described in
> IEP-18
> > > > [4].
> > > >
> > > >
> > > > Cache group key management and node join enhancements:
> > > >
> > > > 1. GridEncryptionManager encapsulates all logic related to
> key
> > > > management:
> > > > a. All group encryption keys are stored in MetaStore.
> > > >
> > > > b. Joining node sends to cluster:
> > > > * Master key digest.
> > > > * All group keys saved locally (Map > > > byte[]>). Keys send over a network in encrypted form.
> > > >
> > > > c. Coordinator on new node join:
> > > > * Compares new node master key digest with
> the
> > > > local master key digest.
> > > > If differs then new node join is rejected.
> > > >
> > > > * Compares local and received group keys.
> > > > If some key differs then new node join is
> > > > rejected.
> > > >
> > > > d. All server nodes:
> > > > * If some of received keys are new then they
> save
> > > > locally.
> > > >
> > > > 2. Dynamic cache creation:
> > > > a. On server node - Encryption key is generated and
> added
> > > > to DynamicCacheChangeRequest.
> > > >
> > > > b. On client node:
> > > > * Prior to creation of
> DynamicCacheChangeRequest
> > > > we have to get new encryption key from some server node.
> > > > * New request added to solve this -
> > > > GenerateEncryptionKeyRequest.
> > > >
> > > >
> > > > WAL Record encryption:
> > > >
> > > >
> > > > 1. New type of WAL record created – EncryptedRecord.
> > > >
> > > > 2. EncryptedRecord is a container that stores some
> > > > WalRecordCacheGroupAware in encrypted form inside.
> > > >
> > > > 3. Write:
> > > > Each record for an encrypted group that implements
> > > > WalRecordCacheGroupAware written to WAL in encrypted form.
> > > > Instead of that record we write EncryptedRecord with plain
> record
> > > > inside(PageSnapshot, PageDeltaRecord, etc).
> > > >
> > > > 4. Read: There are 3 different cases on EncryptedRecord read:
> > > > a. WAL restore – we read EncryptedRecord, decrypt
> internal
> > > > record and return it.
> > > >
> > > > b. Offline WAL
> iteration(StandaloneWalRecordsIterator) -
> > > > EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just
> return it
> > > > from an iterator.
> > > >
> > > > c. Meta storage restore process – On node start we
> restore
> > > > MetaStore.
> > > > When we do it no encryption keys are available,
> because
> > > > they are stored in MetaStore.
> > > > So we can’t decrypt EncryptedRecord and just return
> it
> > > > from an iterator.
> > > > We don't need decrypted record on this step to
> operate
> > > > properly.
> > > >
> > > >
> > > > Page encryption:
> > > >
> > > >
> > > > 1. We have to write on disk pages aligned on 2^n (2kb, 4kb,
> etc)
> > > > for gain maximum perfrormance.
> > > >
> > > > 2. There is a 16 byte overhead for and AES CBC mode.
> > > >
> > > > 3. So we have to preserve 16

[jira] [Created] (IGNITE-9554) Web console demo: do not auto-populate caches created via SQL

2018-09-11 Thread Pavel Konstantinov (JIRA)

Pavel Konstantinov created IGNITE-9554:
--

 Summary: Web console demo: do not auto-populate caches created via 
SQL
 Key: IGNITE-9554
 URL: https://issues.apache.org/jira/browse/IGNITE-9554
 Project: Ignite
  Issue Type: Improvement
Reporter: Pavel Konstantinov


If I executing an example SQL quires

CREATE TABLE Person(ID INTEGER PRIMARY KEY, NAME VARCHAR(100));
INSERT INTO Person(ID, NAME) VALUES (1, 'Ed'), (2, 'Ann'), (3, 'Emma');
SELECT * FROM Person;

I got 'result set is empty' due to inserted data were replaced with 
auto-generated.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9553) Web console: in Demo mode do not polulate caches created by SQL

2018-09-11 Thread Vasiliy Sisko (JIRA)

Vasiliy Sisko created IGNITE-9553:
-

 Summary: Web console: in Demo mode do not polulate caches created 
by SQL
 Key: IGNITE-9553
 URL: https://issues.apache.org/jira/browse/IGNITE-9553
 Project: Ignite
  Issue Type: Bug
  Components: wizards
Reporter: Vasiliy Sisko
Assignee: Vasiliy Sisko






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9552) Web console: add TypeScript support

2018-09-11 Thread Ilya Borisov (JIRA)

Ilya Borisov created IGNITE-9552:


 Summary: Web console: add TypeScript support
 Key: IGNITE-9552
 URL: https://issues.apache.org/jira/browse/IGNITE-9552
 Project: Ignite
  Issue Type: Improvement
  Components: wizards
Reporter: Ilya Borisov
Assignee: Ilya Borisov


What to do:
1. Add TypeScript plugin to babel config.
2. Update webpack configs to load .ts files with babel-loader.
3. Make sure eslint lint .ts files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4732: Mxbeans threads

2018-09-11 Thread DaveWHarvey

GitHub user DaveWHarvey opened a pull request:

https://github.com/apache/ignite/pull/4732

Mxbeans threads



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/percipiomedia/ignite mxbeans_threads

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4732.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4732


commit 5db720517d20c80a541cfc38d52c87d3992e0216
Author: Vasiliy Sisko 
Date:   2018-09-10T09:43:10Z

IGNITE-7460 Web Console: Fixed issue with "step" param of 
"evictionThreshold" input.

commit 92100729a933a70e9fcd3f52a7128339bd1aad31
Author: Sunny Chan 
Date:   2018-09-10T09:53:28Z

IGNITE-5960 Fixed race on continuous query registration and entry update. 
Fixes #2728.

commit 422ca49aa073b101253e74b6b55ac8e478ddd10b
Author: Aleksei Scherbakov 
Date:   2018-09-10T11:33:40Z

IGNITE-8509 Fixed and reworkd tx rollback tests - Fixes #4150.

Signed-off-by: Alexey Goncharuk 

commit 1cb551dd5f0d203c7325a3ffe964612e85dff0e4
Author: Vasiliy Sisko 
Date:   2018-09-10T15:14:34Z

IGNITE-8842 Web console: Fixed initial screen in demo mode.

commit 472954762bc2a8ad3703ae28e190637dd37e18f9
Author: Aleksei Scherbakov 
Date:   2018-09-10T15:43:41Z

IGNITE-9445 Use valid tag for page write unlock while reading cold page 
from disk - Fixes #4708.

Signed-off-by: Alexey Goncharuk 

commit 7d1f5d083cc4b8f286847e661b415657166ce179
Author: devozerov 
Date:   2018-09-10T19:55:55Z

Fixed a problem with optimized marshaller.

commit b188e2c319738fd30f5f87664167419d78347f57
Author: Ilya Borisov 
Date:   2018-09-11T03:05:07Z

IGNITE-9528 Web Console: Fixed handling of 
"LOAD_COMPLETE_CONFIGURATION_ERR" action in PageConfigure service.
   Previous implementation used to throw a complete store message instead 
of an error it wraps.
   Also updated clusterServiceConfiguration generator to fail gracefully if 
some of caches do not exist.

commit 788a726b14a34f212c766aaea18999d2169abb4c
Author: Alexey Platonov 
Date:   2018-09-11T08:39:54Z

IGNITE-8650 Fix flaky ZookeeperDiscovery testClientReconnect. - Fixes #4704.

Signed-off-by: Dmitriy Govorukhin 

commit db7b2eee21ee40d40ab90519cbfc64fe2aa4d191
Author: ibessonov 
Date:   2018-09-11T09:53:56Z

IGNITE-9518 Fix getPagesFillFactor returns NaN for empty region - Fixes 
#4717.

Signed-off-by: Dmitriy Govorukhin 

commit 4cc5a05ba34034243798ef8f55306262e07b756a
Author: ibessonov 
Date:   2018-09-11T12:12:19Z

IGNITE-9535 Missing annotation added to 
GridClusterStateProcessor.ClientChangeGlobalStateComputeRequest class. - Fixes 
#4721.

Signed-off-by: Dmitriy Govorukhin 

commit 0db6db8ef18742468652fa68c932a5acd6f9ce93
Author: Alexey Goncharuk 
Date:   2018-09-11T13:47:56Z

IGNITE-9084 Fixed conflicting messages order

commit 932b301911cf8b0340f0b2b4737b9f3928981217
Author: ipavlukhin 
Date:   2018-09-11T14:01:47Z

IGNITE-9516: MVCC: workaround for hanging test. This closes #4728.

commit d38910efb4bbfacc62abf46274ffc061b56936d9
Author: devozerov 
Date:   2018-09-11T14:25:24Z

IGNITE-9202: .NET: Fixed failures in several tests which use distributed 
joins. The fix is to set 1 min timeout. Looks like on some environments it 
takes too long to join node and rebalance data, so we failed with timeout.

commit 3d901ad9f5447fdecb2b39c156ae673f0925d0ec
Author: Dmitriy Govorukhin 
Date:   2018-09-11T14:47:12Z

IGNITE-9426 Added IgniteAtomicSequence benchmarks - Fixes #4665.

Signed-off-by: Dmitriy Govorukhin 

commit b29970790e0e6657f5d03f2950ed070987527386
Author: shroman 
Date:   2018-09-12T01:10:56Z

IGNITE-9388: mesos IgniteProvider tries to access obolete ignite.run or 
download from slow archive. - Fixes #4639.

Signed-off-by: shroman 

commit 61390df528d11b94fa9828e8823c650f0072216e
Author: Dave Harvey 
Date:   2018-09-12T02:35:24Z

Merge branch 'master' of https://github.com/apache/ignite

commit d3d8ab176f6ed0fc735b3c801d3280c4c62fcad8
Author: Dave Harvey 
Date:   2018-08-03T18:11:38Z

Fix for IGNITE-7616
Correctly register threads and add new MXBEANS support for other thread
types.   Needed to weaken some data hiding to enable MXBeans




---

[GitHub] ignite pull request #4731: Mxbeans thread display improvements

2018-09-11 Thread DaveWHarvey

Github user DaveWHarvey closed the pull request at:

https://github.com/apache/ignite/pull/4731


---

[GitHub] ignite pull request #4731: Mxbeans thread display improvements

2018-09-11 Thread DaveWHarvey

GitHub user DaveWHarvey opened a pull request:

https://github.com/apache/ignite/pull/4731

Mxbeans thread display improvements



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/percipiomedia/ignite 
mxbeans_thread_display_improvements

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4731.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4731


commit 748332185a274ca61028c4b6e91ebcc163d6ef09
Author: Vasiliy Sisko 
Date:   2018-09-10T09:43:10Z

IGNITE-7460 Web Console: Fixed issue with "step" param of 
"evictionThreshold" input.

commit f95bb245afa751a51f600a26d34a7a684cbea5d8
Author: Sunny Chan 
Date:   2018-09-10T09:53:28Z

IGNITE-5960 Fixed race on continuous query registration and entry update. 
Fixes #2728.

commit 030a4864a8a6d0763a375c9a0580bf724ec90833
Author: Aleksei Scherbakov 
Date:   2018-09-10T11:33:40Z

IGNITE-8509 Fixed and reworkd tx rollback tests - Fixes #4150.

Signed-off-by: Alexey Goncharuk 

commit 5600a4a9688700ae055e187080c343a94f141109
Author: Vasiliy Sisko 
Date:   2018-09-10T15:14:34Z

IGNITE-8842 Web console: Fixed initial screen in demo mode.

commit 43cacfa8c9c666fe73b432f4541736c03591888a
Author: Aleksei Scherbakov 
Date:   2018-09-10T15:43:41Z

IGNITE-9445 Use valid tag for page write unlock while reading cold page 
from disk - Fixes #4708.

Signed-off-by: Alexey Goncharuk 

commit b4a055f02e7aa412710ce0b8a1ec7a4840e5275b
Author: devozerov 
Date:   2018-09-10T19:55:55Z

Fixed a problem with optimized marshaller.

commit 19db096c4c32f80160983d9b33ccdd4e7b537be3
Author: Ilya Borisov 
Date:   2018-09-11T03:05:07Z

IGNITE-9528 Web Console: Fixed handling of 
"LOAD_COMPLETE_CONFIGURATION_ERR" action in PageConfigure service.
   Previous implementation used to throw a complete store message instead 
of an error it wraps.
   Also updated clusterServiceConfiguration generator to fail gracefully if 
some of caches do not exist.

commit b49b8b9f4a5cd1b9674b41aa6c22e283e680d4d7
Author: Alexey Platonov 
Date:   2018-09-11T08:39:54Z

IGNITE-8650 Fix flaky ZookeeperDiscovery testClientReconnect. - Fixes #4704.

Signed-off-by: Dmitriy Govorukhin 

commit 836d47ff97e81dbf400844dd1f3fc2cf3b9ed72b
Author: ibessonov 
Date:   2018-09-11T09:53:56Z

IGNITE-9518 Fix getPagesFillFactor returns NaN for empty region - Fixes 
#4717.

Signed-off-by: Dmitriy Govorukhin 

commit 2ff43627a6b80c92a26528e40a20184785cca317
Author: ibessonov 
Date:   2018-09-11T12:12:19Z

IGNITE-9535 Missing annotation added to 
GridClusterStateProcessor.ClientChangeGlobalStateComputeRequest class. - Fixes 
#4721.

Signed-off-by: Dmitriy Govorukhin 

commit b6879cc6e5987013f3b384a65e9cf42155b153e6
Author: Alexey Goncharuk 
Date:   2018-09-11T13:47:56Z

IGNITE-9084 Fixed conflicting messages order

commit 85a0359ce6fc6272ab8ea3bf70fff504e1b58047
Author: ipavlukhin 
Date:   2018-09-11T14:01:47Z

IGNITE-9516: MVCC: workaround for hanging test. This closes #4728.

commit d60d379d4e8504a14a02d0a60cdf90724f60dfa0
Author: devozerov 
Date:   2018-09-11T14:25:24Z

IGNITE-9202: .NET: Fixed failures in several tests which use distributed 
joins. The fix is to set 1 min timeout. Looks like on some environments it 
takes too long to join node and rebalance data, so we failed with timeout.

commit e3252fbd2f3bc48985fdccd23bd6a981ccb36a60
Author: Dmitriy Govorukhin 
Date:   2018-09-11T14:47:12Z

IGNITE-9426 Added IgniteAtomicSequence benchmarks - Fixes #4665.

Signed-off-by: Dmitriy Govorukhin 

commit e85eea8c68c6b162c44a6e1384f0c4466cbbb5d5
Author: shroman 
Date:   2018-09-12T01:10:56Z

IGNITE-9388: mesos IgniteProvider tries to access obolete ignite.run or 
download from slow archive. - Fixes #4639.

Signed-off-by: shroman 

commit e16209cde9f3d898295126e43b64552590e909ad
Author: Dave Harvey 
Date:   2018-08-03T18:11:38Z

Fix for IGNITE-7616
Correctly register threads and add new MXBEANS support for other thread
types.   Needed to weaken some data hiding to enable MXBeans




---

[GitHub] ignite pull request #4639: IGNITE-9388: mesos IgniteProvider tries to access...

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4639


---

Re: TDE Implementation details.

> As I understand the plan is to get TDE Phase 1 released in 2.7, right?

Yes. We will release TDE in 2.7

> Could you also list what needs to be done at Phase 2 and how much time it 
> might take.

Yes, I think Dmitry Ryabov will send Phase 2 design 


В Вт, 11/09/2018 в 23:27 +0300, Nikolay Izhikov пишет:
> Hello, Denis.
> 
> Yes, Vladimir made 2 rounds of review.
> I planning to fix all issues with implementation in a few days.
> 
> 
> В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> > Hi Nikolay,
> > 
> > Has anybody started looking into your request? As I understand the plan is
> > to get TDE Phase 1 released in 2.7, right?
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > 
> > Could you also list what needs to be done at Phase 2 and how much time it
> > might take.
> > 
> > --
> > Denis
> > 
> > On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:
> > 
> > > Hello, Igniters.
> > > 
> > > I want to share with you TDE implementation details.
> > > I think it can simplify review and acception of TDE implementation.
> > > This mail is copy of wiki page [1].
> > > 
> > > Please, share your thoughts.
> > > 
> > > TDE is ready for review [2].
> > > Please, let me know, who is able to make final review.
> > > 
> > > This document describes some internal details of TDE.Phase 1
> > > implementation [3].
> > > I suggest that reader familiar with the general design described in IEP-18
> > > [4].
> > > 
> > > 
> > > Cache group key management and node join enhancements:
> > > 
> > > 1. GridEncryptionManager encapsulates all logic related to key
> > > management:
> > > a. All group encryption keys are stored in MetaStore.
> > > 
> > > b. Joining node sends to cluster:
> > > * Master key digest.
> > > * All group keys saved locally (Map > > byte[]>). Keys send over a network in encrypted form.
> > > 
> > > c. Coordinator on new node join:
> > > * Compares new node master key digest with the
> > > local master key digest.
> > > If differs then new node join is rejected.
> > > 
> > > * Compares local and received group keys.
> > > If some key differs then new node join is
> > > rejected.
> > > 
> > > d. All server nodes:
> > > * If some of received keys are new then they save
> > > locally.
> > > 
> > > 2. Dynamic cache creation:
> > > a. On server node - Encryption key is generated and added
> > > to DynamicCacheChangeRequest.
> > > 
> > > b. On client node:
> > > * Prior to creation of DynamicCacheChangeRequest
> > > we have to get new encryption key from some server node.
> > > * New request added to solve this -
> > > GenerateEncryptionKeyRequest.
> > > 
> > > 
> > > WAL Record encryption:
> > > 
> > > 
> > > 1. New type of WAL record created – EncryptedRecord.
> > > 
> > > 2. EncryptedRecord is a container that stores some
> > > WalRecordCacheGroupAware in encrypted form inside.
> > > 
> > > 3. Write:
> > > Each record for an encrypted group that implements
> > > WalRecordCacheGroupAware written to WAL in encrypted form.
> > > Instead of that record we write EncryptedRecord with plain record
> > > inside(PageSnapshot, PageDeltaRecord, etc).
> > > 
> > > 4. Read: There are 3 different cases on EncryptedRecord read:
> > > a. WAL restore – we read EncryptedRecord, decrypt internal
> > > record and return it.
> > > 
> > > b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> > > EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return 
> > > it
> > > from an iterator.
> > > 
> > > c. Meta storage restore process – On node start we restore
> > > MetaStore.
> > > When we do it no encryption keys are available, because
> > > they are stored in MetaStore.
> > > So we can’t decrypt EncryptedRecord and just return it
> > > from an iterator.
> > > We don't need decrypted record on this step to operate
> > > properly.
> > > 
> > > 
> > > Page encryption:
> > > 
> > > 
> > > 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc)
> > > for gain maximum perfrormance.
> > > 
> > > 2. There is a 16 byte overhead for and AES CBC mode.
> > > 
> > > 3. So we have to preserve 16 bytes in page in memory to get
> > > encrypted page size equal to 2^n when written it to disk.
> > > 
> > > 4. PageIO has many methods with pageSize parameter.
> > > So for encrypted cache groups page size is calculated as
> > > cfg.getPageSize() - 16 byte.
> > > 
> > > 
> > > [1]
> > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > > [2] https://github.com/

Re: TDE Implementation details.

Hello, Denis.

Yes, Vladimir made 2 rounds of review.
I planning to fix all issues with implementation in a few days.


В Вт, 11/09/2018 в 10:40 -0400, Denis Magda пишет:
> Hi Nikolay,
> 
> Has anybody started looking into your request? As I understand the plan is
> to get TDE Phase 1 released in 2.7, right?
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> 
> Could you also list what needs to be done at Phase 2 and how much time it
> might take.
> 
> --
> Denis
> 
> On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:
> 
> > Hello, Igniters.
> > 
> > I want to share with you TDE implementation details.
> > I think it can simplify review and acception of TDE implementation.
> > This mail is copy of wiki page [1].
> > 
> > Please, share your thoughts.
> > 
> > TDE is ready for review [2].
> > Please, let me know, who is able to make final review.
> > 
> > This document describes some internal details of TDE.Phase 1
> > implementation [3].
> > I suggest that reader familiar with the general design described in IEP-18
> > [4].
> > 
> > 
> > Cache group key management and node join enhancements:
> > 
> > 1. GridEncryptionManager encapsulates all logic related to key
> > management:
> > a. All group encryption keys are stored in MetaStore.
> > 
> > b. Joining node sends to cluster:
> > * Master key digest.
> > * All group keys saved locally (Map > byte[]>). Keys send over a network in encrypted form.
> > 
> > c. Coordinator on new node join:
> > * Compares new node master key digest with the
> > local master key digest.
> > If differs then new node join is rejected.
> > 
> > * Compares local and received group keys.
> > If some key differs then new node join is
> > rejected.
> > 
> > d. All server nodes:
> > * If some of received keys are new then they save
> > locally.
> > 
> > 2. Dynamic cache creation:
> > a. On server node - Encryption key is generated and added
> > to DynamicCacheChangeRequest.
> > 
> > b. On client node:
> > * Prior to creation of DynamicCacheChangeRequest
> > we have to get new encryption key from some server node.
> > * New request added to solve this -
> > GenerateEncryptionKeyRequest.
> > 
> > 
> > WAL Record encryption:
> > 
> > 
> > 1. New type of WAL record created – EncryptedRecord.
> > 
> > 2. EncryptedRecord is a container that stores some
> > WalRecordCacheGroupAware in encrypted form inside.
> > 
> > 3. Write:
> > Each record for an encrypted group that implements
> > WalRecordCacheGroupAware written to WAL in encrypted form.
> > Instead of that record we write EncryptedRecord with plain record
> > inside(PageSnapshot, PageDeltaRecord, etc).
> > 
> > 4. Read: There are 3 different cases on EncryptedRecord read:
> > a. WAL restore – we read EncryptedRecord, decrypt internal
> > record and return it.
> > 
> > b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> > EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return it
> > from an iterator.
> > 
> > c. Meta storage restore process – On node start we restore
> > MetaStore.
> > When we do it no encryption keys are available, because
> > they are stored in MetaStore.
> > So we can’t decrypt EncryptedRecord and just return it
> > from an iterator.
> > We don't need decrypted record on this step to operate
> > properly.
> > 
> > 
> > Page encryption:
> > 
> > 
> > 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc)
> > for gain maximum perfrormance.
> > 
> > 2. There is a 16 byte overhead for and AES CBC mode.
> > 
> > 3. So we have to preserve 16 bytes in page in memory to get
> > encrypted page size equal to 2^n when written it to disk.
> > 
> > 4. PageIO has many methods with pageSize parameter.
> > So for encrypted cache groups page size is calculated as
> > cfg.getPageSize() - 16 byte.
> > 
> > 
> > [1]
> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> > [2] https://github.com/apache/ignite/pull/4167
> > [3] https://issues.apache.org/jira/browse/IGNITE-8485
> > [4]
> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption
> > 

signature.asc
Description: This is a digitally signed message part

Re: Python thin client

2018-09-11 Thread Dmitry Melnichuk


Igor,

I have just commited an improvment to the HandshakeError message 
generation algorithm. I hope it is now easier to understand what expects 
what in case of binary protocol version mismatch.


Thank you for pointing this out.

On 9/12/18 2:13 AM, Igor Sapego wrote:

I managed to start tests, and now I'm getting the following message:

pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.

It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.

Best Regards,
Igor

[GitHub] ignite pull request #4730: Ignite 2.4.6.b4

2018-09-11 Thread slukyano

GitHub user slukyano opened a pull request:

https://github.com/apache/ignite/pull/4730

Ignite 2.4.6.b4



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-2.4.6.b4

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4730.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4730


commit debc906a25d3e2d65db58e16307fae6f08460eeb
Author: devozerov 
Date:   2018-02-27T09:13:52Z

Revert "IGNITE-7253: JDBC thin streaming: fixed default local buffer size, 
improved error messages in case of unsupported SQL statements."

This reverts commit d53504adb1d4276f79ede2401e2d1512fe0287ec.

commit d8cf9e042c0e9afe65508c006d9d1c39779d78b9
Author: devozerov 
Date:   2018-02-27T09:14:21Z

Revert "IGNITE-7253: Streaming mode for JDBC thin driver. This closes 
#3499."

This reverts commit bc331f9de716c30d6f733e28821ab44da7ed0cf7.

commit 975a7cdb68cb89da74ec78c3cf23f644050918ff
Author: devozerov 
Date:   2018-02-27T09:49:44Z

Merge branch 'ignite-2.4' into ignite-2.4.3

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/cache/persistence/wal/FsyncModeFileWriteAheadLogManager.java
#   parent/pom.xml

commit 74a54d23913bd7195c525d8e222b4e4047515843
Author: Ilya Lantukh 
Date:   2018-02-28T07:08:54Z

IGNITE-7836 Fixed handling of state change message when forceReassignment 
is false

commit 2a70ede048f59753061973495f83927f47452d66
Author: Andrey Novikov 
Date:   2018-01-19T03:05:03Z

IGNITE-6920 Web Console: Create default account for direct-install package.

(cherry picked from commit e5005d9)

commit 2f1ea2cdb76863008d4514f26845457bdeb7d6ed
Author: Andrey Novikov 
Date:   2018-01-19T04:35:30Z

IGNITE-6920 Minor fix.

(cherry picked from commit 9cc7cbf)

commit 62652f3fb1563ba149dcbccb80928d50b822ff36
Author: alexdel 
Date:   2018-01-25T08:49:28Z

IGNITE-7064 Web Console: Implemented basic E2E tests.

(cherry picked from commit ce96e4f)

commit 06e891f1161af598e0aa4665f7a6047637d1c476
Author: Dmitriy Shabalin 
Date:   2018-01-25T09:51:44Z

IGNITE-7522 Web Console: Fixed cluster selector state after cluster restart.

(cherry picked from commit ac3cfb8)

commit 291cb2c88118ccffebcf3383db629647faec1eee
Author: Dmitriy Shabalin 
Date:   2018-01-25T10:33:13Z

IGNITE-7529 Web Console: Refactor UIGrid column filters.

(cherry picked from commit 08658ea)

commit 443eafc718685e66c9a60058bd0ab56d88f9f0a6
Author: alexdel 
Date:   2018-01-25T14:38:36Z

IGNITE-7064 Web Console: Minor test fix.

(cherry picked from commit 4b6d9ad)

commit 6f1df5c40100363b5922734223a774ff1d6a008e
Author: Ilya Borisov 
Date:   2018-01-26T09:07:47Z

IGNITE-7031 Web Console: Refactored confirmation cancellation logic.

(cherry picked from commit 92ae3fe)

commit 16982825fb06ff2724ba4583781cc15443c4f46d
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:07:57Z

IGNITE-7610 Web Console: Profile page refactored to component.

(cherry picked from commit 958975e)

commit 9c6a52250e058c4546aef0d0ecee977c07076a1a
Author: Alexey Kuznetsov 
Date:   2018-02-02T10:09:37Z

IGNITE-7612 Web Console: Refactored mongoose schemas to separate module.

(cherry picked from commit 71db707)

commit 89e15830dedcb46f24d9cc9b922ba3b013331a18
Author: Vasiliy Sisko 
Date:   2018-02-12T10:22:10Z

Web Agent: Fixed wrong config of IGNITE_ATOMIC_CACHE_DELETE_HISTORY_SIZE in 
demo startup.

(cherry picked from commit 1a6e544)

commit 18966673570425192e1b89fbb2c63d164b47eaca
Author: Vasiliy Sisko 
Date:   2018-02-12T13:24:30Z

IGNITE-7578 Actualized client connector configuration.

(cherry picked from commit 819d746)

commit 237063efa35c54bb9e9800ecf63ea223ec20a9ef
Author: alexdel 
Date:   2018-02-19T04:25:24Z

IGNITE-7650 Extracted signin/signup form to separate page, improved landing 
page.

(cherry picked from commit 1925674)

commit 679aeca7a3ff60a9dd478966d3949107d302d5db
Author: Andrey Novikov 
Date:   2018-02-19T07:56:07Z

IGNITE-7650 Fixed headers.

(cherry picked from commit 67922b3)

commit 5d5a6a05ec49f895e8a5edd505e496dcfe58a208
Author: Vasiliy Sisko 
Date:   2018-02-21T04:21:02Z

IGNITE-6094 Web Agent: Enabled persistent in demo mode.

(cherry picked from commit 3c35900)

commit e35d8cfb06f52765959fc2e1883bf70fe0259f45
Author: Alexander Kalinin 
Date:   2018-02-21T07:03:20Z

IGNITE-7320 Web Console - Fixed table headers for Safari.

(cherry picked from commit 04a025b)

commit 20cb1439f48b9a3c985f65784af36ef2c6f45e4f
Author: Andrey Novikov 
Date:   2018-02-22T02:54:05Z

IGNITE-7650 Fixed counties codes.

(cherry picked from commit fc40261)

commit 50d1389cd60e1480

[jira] [Created] (IGNITE-9551) Provide support for on-heap/non serialised/read only caches

2018-09-11 Thread Steve Hostettler (JIRA)

Steve Hostettler created IGNITE-9551:

Summary: Provide support for on-heap/non serialised/read only
caches
Key: IGNITE-9551
URL: https://issues.apache.org/jira/browse/IGNITE-9551
Project: Ignite
Issue Type: Wish
Components: cache
Affects Versions: 2.7
Reporter: Steve Hostettler

We have several use cases in which we warm-up caches and then use them in a
read-only manner during the processsing. For instance, we exchange rates and
forex values. These caches are heavily used.

Ideally I would like to be able to tag caches (or get projections) with the
following features:
* Read Only : the cache are of course initiliazed but once they are marked as
initiliazed I do not need any locking anymore because I know that the only
access will be read only until they are destroyed. In micro-benchmarks that
create a new projection using GatewayProtectedCacheProxy(
(IgniteCacheProxy) delegate, opCtx.keepBinary(), false); I gain around
15%

* No Serialisation : These type of caches usually do not contain billions of
elements but almost all the fields are used all the time. Therefore the
overhead of the systematic binary marshalling is quite high.
* On Heap : as explained in
[http://apache-ignite-developers.2346864.n4.nabble.com/Questions-about-getAllInternal-in-GridLocalAtomicCache-td34662.html]
having everything off-heap comes at a cost. Espcially the need to put guards
and locking.

I would love to contribute but I need some guidance and reality check.

--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4729: IGNITE-9545 IgniteProjectionStartStopRestartSelfT...

2018-09-11 Thread oignatenko

GitHub user oignatenko opened a pull request:

https://github.com/apache/ignite/pull/4729

IGNITE-9545 IgniteProjectionStartStopRestartSelfTest: misleading javadocs 
etc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9545

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4729.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4729


commit 1eeca908a8076a8317947dac8a46845964d1d7ea
Author: Oleg Ignatenko 
Date:   2018-08-23T13:13:28Z

IGNITE-9348 ML examples improvements
- wip (logging improved)
-- verified with diffs overview, executing the examples and studying 
execution logs

commit e50b89c392568ba9b93935c4fa6c7f7f93f5ec6f
Author: Oleg Ignatenko 
Date:   2018-08-23T14:45:57Z

Revert "IGNITE-9348 ML examples improvements"

This reverts commit 1eeca908a8076a8317947dac8a46845964d1d7ea.

commit 474024b4c5bbdb3c0a4ed2f0a66238c8054c6674
Author: Oleg Ignatenko 
Date:   2018-08-23T14:57:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9642b233b5701bdad47ebea163079160227c582a
Author: Oleg Ignatenko 
Date:   2018-08-28T14:01:11Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 7fc16d013ab725d2ff2e1a1b042c983f11d0c4d4
Author: Oleg Ignatenko 
Date:   2018-08-28T15:13:02Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit d2caba67b156674f051f50faebeafe0871bf0914
Author: Oleg Ignatenko 
Date:   2018-08-29T13:14:07Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 16775dff51d71ea68b4a3dea98be552130c493ed
Author: Oleg Ignatenko 
Date:   2018-08-30T09:00:56Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aedb59929974fe205b949225c1a338c68c60cfc8
Author: Oleg Ignatenko 
Date:   2018-08-30T09:42:38Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 39c6482fcdca506aa33011ed21c98060b4a8c68b
Author: Oleg Ignatenko 
Date:   2018-08-30T11:28:05Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 33b32a2760a6559c78283b927e3191180d8ed9e1
Author: Oleg Ignatenko 
Date:   2018-08-30T12:31:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9531028ddd1aef9e95f7e8c8b528086739bbb1b0
Author: Oleg Ignatenko 
Date:   2018-08-30T14:06:34Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 28f22c6e2fffcb82717ba5da7be2cfd39715c4e3
Author: Oleg Ignatenko 
Date:   2018-08-30T16:41:51Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit aacac88db519187413b0fc5ff9d0e55b8f8cff22
Author: Oleg Ignatenko 
Date:   2018-08-31T10:12:32Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 897f920dde46022849b13f9fb86dba8e54119a56
Author: Oleg Ignatenko 
Date:   2018-08-31T13:57:14Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 114c79e54c1b316006ccc3ff22d20d902f9313df
Author: Oleg Ignatenko 
Date:   2018-08-31T17:39:16Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fd6b659bb8be1992618ce4ce91f568a0988b3afa
Author: Oleg Ignatenko 
Date:   2018-09-02T06:11:42Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 6ae6d1d3cf9743d8d466be0330511ddc8589e944
Author: Oleg Ignatenko 
Date:   2018-09-03T10:27:35Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit e8b27dbd3d0c1cbdb7a7659175f5c7bb447482bf
Author: Oleg Ignatenko 
Date:   2018-09-04T09:49:44Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 622c82efdd0aa182fadea6b7ffa5d4615521a3f5
Author: Oleg Ignatenko 
Date:   2018-09-05T10:50:28Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit fb844fe3751e2e8ae87e6b8030b0e4bd659df9c2
Author: Oleg Ignatenko 
Date:   2018-09-05T11:45:58Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 480ed78869277d7e32f725ab71bec9621f1ac03a
Author: Oleg Ignatenko 
Date:   2018-09-06T07:52:55Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit c99762498f617c0e98ea3062a43c0b30092166ef
Author: Oleg Ignatenko 
Date:   2018-09-06T14:45:04Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 2e17175225c72f747d370b5fee96f8be69d6d2cb
Author: Oleg Ignatenko 
Date:   2018-09-06T17:33:54Z

Merge branch 'master' of https://github.com/apache/ignite into master-ml

commit 9ebcd9a2fe5966b0bf42a95484395867c15d863f
Author: Oleg Ignatenko 
Date:   2018-09-07T13:38:51Z

Merge branch 'master' of

Re: Python thin client

2018-09-11 Thread Pavel Petroshenko

Hi Igor,

Did you have to do anything special to make the tests eventually run?

p.

On Tue, Sep 11, 2018 at 9:13 AM, Igor Sapego  wrote:

> I managed to start tests, and now I'm getting the following message:
>
> pyignite.exceptions.HandshakeError: Handshake error: Unsupported
> version. Expected protocol version: 1.1.0.
>
> It would be useful to print "Unexpected version" itself, because I can
> not understand what is the issue.
>
> Best Regards,
> Igor
>
>
> On Tue, Sep 11, 2018 at 10:34 AM Dmitry Melnichuk <
> dmitry.melnic...@nobitlost.com> wrote:
>
> > Igor,
> >
> > I literally followed the steps you describe, but unfortunately could not
> > reproduce the error.
> >
> > The only clue I got is that your site-packages already have the newer
> > version of attrs (18.2.0 against 18.1.0 as of pyignite requirements).
> > But this should not be an issue, since pytest-runner in this command
> > creates its own test environment, independent of the one it runs into.
> > It worked for me when I installed `attrs==18.2.0` locally and run tests
> > − everything went smooth.
> >
> > You can try uninstalling or downgrading your local attrs with
> >
> > ```
> > $ pip3 uninstall attrs
> > ```
> >
> > but I don't think it will actually help. Maybe it will raise another
> > error, so we could dig deeper into the problem.
> >
> > Alternatively, you can give me more details to recreate it (OS, Python
> > and pip versions). I'm not ruling out a bug in setuptools.
> >
> > On 9/11/18 2:13 AM, Igor Sapego wrote:
> > > Guys, I've cloned your repository, run pip3 install -e .
> > > then run pip3 install -r requirements/* over all the requirements,
> > > and finally run python ./setup.py pytest, but get the following output:
> > >
> > > running pytest
> > > Searching for attrs==18.1.0
> > > Best match: attrs 18.1.0
> > >
> > > Using /home/isapego/.local/lib/python3.6/site-packages
> > > running egg_info
> > > writing pyignite.egg-info/PKG-INFO
> > > writing dependency_links to pyignite.egg-info/dependency_links.txt
> > > writing requirements to pyignite.egg-info/requires.txt
> > > writing top-level names to pyignite.egg-info/top_level.txt
> > > reading manifest file 'pyignite.egg-info/SOURCES.txt'
> > > writing manifest file 'pyignite.egg-info/SOURCES.txt'
> > > running build_ext
> > > Traceback (most recent call last):
> > >File "./setup.py", line 98, in 
> > >  'Operating System :: OS Independent',
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/
> setuptools/__init__.py",
> > > line 140, in setup
> > >  return distutils.core.setup(**attrs)
> > >File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
> > >  dist.run_commands()
> > >File "/usr/lib/python3.6/distutils/dist.py", line 955, in
> run_commands
> > >  self.run_command(cmd)
> > >File "/usr/lib/python3.6/distutils/dist.py", line 974, in
> run_command
> > >  cmd_obj.run()
> > >File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py",
> line
> > 175,
> > > in run
> > >  with self.project_on_sys_path():
> > >File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
> > >  return next(self.gen)
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/
> setuptools/command/test.py",
> > > line 166, in project_on_sys_path
> > >  require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/pkg_
> resources/__init__.py",
> > > line 895, in require
> > >  needed = self.resolve(parse_requirements(requirements))
> > >File
> > >
> > "/home/isapego/.local/lib/python3.6/site-packages/pkg_
> resources/__init__.py",
> > > line 786, in resolve
> > >  raise VersionConflict(dist, req).with_context(dependent_req)
> > > pkg_resources.ContextualVersionConflict: (attrs 18.2.0
> > > (/home/isapego/.local/lib/python3.6/site-packages),
> > > Requirement.parse('attrs==18.1.0'), {'pyignite'})
> > >
> > > What am I doing wrong?
> > >
> > > Best Regards,
> > > Igor
> > >
> >
>

[jira] [Created] (IGNITE-9550) Cache get operation returns null for a lost partition

2018-09-11 Thread Pavel Vinokurov (JIRA)

Pavel Vinokurov created IGNITE-9550:
---

 Summary: Cache get operation returns null for a lost partition
 Key: IGNITE-9550
 URL: https://issues.apache.org/jira/browse/IGNITE-9550
 Project: Ignite
  Issue Type: Bug
  Components: cache
Affects Versions: 2.6
Reporter: Pavel Vinokurov
Assignee: Pavel Vinokurov
 Attachments: PartitionLostReproducer.java





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Python thin client

2018-09-11 Thread Igor Sapego

I managed to start tests, and now I'm getting the following message:

pyignite.exceptions.HandshakeError: Handshake error: Unsupported
version. Expected protocol version: 1.1.0.

It would be useful to print "Unexpected version" itself, because I can
not understand what is the issue.

Best Regards,
Igor


On Tue, Sep 11, 2018 at 10:34 AM Dmitry Melnichuk <
dmitry.melnic...@nobitlost.com> wrote:

> Igor,
>
> I literally followed the steps you describe, but unfortunately could not
> reproduce the error.
>
> The only clue I got is that your site-packages already have the newer
> version of attrs (18.2.0 against 18.1.0 as of pyignite requirements).
> But this should not be an issue, since pytest-runner in this command
> creates its own test environment, independent of the one it runs into.
> It worked for me when I installed `attrs==18.2.0` locally and run tests
> − everything went smooth.
>
> You can try uninstalling or downgrading your local attrs with
>
> ```
> $ pip3 uninstall attrs
> ```
>
> but I don't think it will actually help. Maybe it will raise another
> error, so we could dig deeper into the problem.
>
> Alternatively, you can give me more details to recreate it (OS, Python
> and pip versions). I'm not ruling out a bug in setuptools.
>
> On 9/11/18 2:13 AM, Igor Sapego wrote:
> > Guys, I've cloned your repository, run pip3 install -e .
> > then run pip3 install -r requirements/* over all the requirements,
> > and finally run python ./setup.py pytest, but get the following output:
> >
> > running pytest
> > Searching for attrs==18.1.0
> > Best match: attrs 18.1.0
> >
> > Using /home/isapego/.local/lib/python3.6/site-packages
> > running egg_info
> > writing pyignite.egg-info/PKG-INFO
> > writing dependency_links to pyignite.egg-info/dependency_links.txt
> > writing requirements to pyignite.egg-info/requires.txt
> > writing top-level names to pyignite.egg-info/top_level.txt
> > reading manifest file 'pyignite.egg-info/SOURCES.txt'
> > writing manifest file 'pyignite.egg-info/SOURCES.txt'
> > running build_ext
> > Traceback (most recent call last):
> >File "./setup.py", line 98, in 
> >  'Operating System :: OS Independent',
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/setuptools/__init__.py",
> > line 140, in setup
> >  return distutils.core.setup(**attrs)
> >File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
> >  dist.run_commands()
> >File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
> >  self.run_command(cmd)
> >File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
> >  cmd_obj.run()
> >File "/home/isapego/.local/lib/python3.6/site-packages/ptr.py", line
> 175,
> > in run
> >  with self.project_on_sys_path():
> >File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__
> >  return next(self.gen)
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/setuptools/command/test.py",
> > line 166, in project_on_sys_path
> >  require('%s==%s' % (ei_cmd.egg_name, ei_cmd.egg_version))
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 895, in require
> >  needed = self.resolve(parse_requirements(requirements))
> >File
> >
> "/home/isapego/.local/lib/python3.6/site-packages/pkg_resources/__init__.py",
> > line 786, in resolve
> >  raise VersionConflict(dist, req).with_context(dependent_req)
> > pkg_resources.ContextualVersionConflict: (attrs 18.2.0
> > (/home/isapego/.local/lib/python3.6/site-packages),
> > Requirement.parse('attrs==18.1.0'), {'pyignite'})
> >
> > What am I doing wrong?
> >
> > Best Regards,
> > Igor
> >
>

Re: Critical worker threads liveness checking drawbacks

2018-09-11 Thread Andrey Kuznetsov

David, Maxim!

Thanks a lot for you ideas. Unfortunately, I can't adopt all of them right
now: the scope is much broader than the scope of the change I implement. I
have had a talk to a group of Ignite commiters, and we agreed to complete
the change as follows.
- Blocking instructions in system-critical which may resonably last long
should be explicitly excluded from the monitoring.
- Failure handlers should have a setting to suppress some failures on
per-failure-type basis.
According to this I have updated the implementation: [1]

[1] https://github.com/apache/ignite/pull/4089

пн, 10 сент. 2018 г. в 22:35, David Harvey :

> When I've done this before,I've needed to find the oldest  thread, and kill
> the node running that.   From a language standpoint, Maxim's "without
> progress" better than "heartbeat".   For example, what I'm most interested
> in on a distributed system is which thread started the work it has not
> completed the earliest, and when did that thread last make forward
> process. You don't want to kill a node because a thread is waiting on a
> lock held by a thread that went off-node and has not gotten a response.
> If you don't understand the dependency relationships, you will make
> incorrect recovery decisions.
>
> On Mon, Sep 10, 2018 at 4:08 AM Maxim Muzafarov 
> wrote:
>
> > I think we should find exact answers to these questions:
> >  1. What `critical` issue exactly is?
> >  2. How can we find critical issues?
> >  3. How can we handle critical issues?
> >
> > First,
> >  - Ignore uninterruptable actions (e.g. worker\service shutdown)
> >  - Long I/O operations (should be a configurable timeout for each type of
> > usage)
> >  - Infinite loops
> >  - Stalled\deadlocked threads (and\or too many parked threads, exclude
> I/O)
> >
> > Second,
> >  - The working queue is without progress (e.g. disco, exchange queues)
> >  - Work hasn't been completed since the last heartbeat (checking
> > milestones)
> >  - Too many system resources used by a thread for the long period of time
> > (allocated memory, CPU)
> >  - Timing fields associated with each thread status exceeded a maximum
> time
> > limit.
> >
> > Third (not too many options here),
> >  - `log everything` should be the default behaviour in all these cases,
> > since it may be difficult to find the cause after the restart.
> >  - Wait some interval of time and kill the hanging node (cluster should
> be
> > configured stable enough)
> >
> > Questions,
> >  - Not sure, but can workers miss their heartbeat deadlines if CPU loads
> up
> > to 80%-90%? Bursts of momentary overloads can be
> > expected behaviour as a normal part of system operations.
> >  - Why do we decide that critical thread should monitor each other? For
> > instance, if all the tasks were blocked and unable to run,
> > node reset would never occur. As for me, a better solution is to use
> a
> > separate monitor thread or pool (maybe both with software
> > and hardware checks) that not only checks heartbeats but monitors the
> > other system as well.
> >
> > On Mon, 10 Sep 2018 at 00:07 David Harvey  wrote:
> >
> > > It would be safer to restart the entire cluster than to remove the last
> > > node for a cache that should be redundant.
> > >
> > > On Sun, Sep 9, 2018, 4:00 PM Andrey Gura  wrote:
> > >
> > > > Hi,
> > > >
> > > > I agree with Yakov that we can provide some option that manage worker
> > > > liveness checker behavior in case of observing that some worker is
> > > > blocked too long.
> > > > At least it will  some workaround for cases when node fails is too
> > > > annoying.
> > > >
> > > > Backups count threshold sounds good but I don't understand how it
> will
> > > > help in case of cluster hanging.
> > > >
> > > > The simplest solution here is alert in cases of blocking of some
> > > > critical worker (we can improve WorkersRegistry for this purpose and
> > > > expose list of blocked workers) and optionally call system configured
> > > > failure processor. BTW, failure processor can be extended in order to
> > > > perform any checks (e.g. backup count) and decide whether it should
> > > > stop node or not.
> > > > On Sat, Sep 8, 2018 at 3:42 PM Andrey Kuznetsov 
> > > wrote:
> > > > >
> > > > > David, Yakov, I understand your fears. But liveness checks deal
> with
> > > > > _critical_ conditions, i.e. when such a condition is met we
> conclude
> > > the
> > > > > node as totally broken, and there is no sense to keep it alive
> > > regardless
> > > > > the data it contains. If we want to give it a chance, then the
> > > condition
> > > > > (long fsync etc.) should not considered as critical at all.
> > > > >
> > > > > сб, 8 сент. 2018 г. в 15:18, Yakov Zhdanov :
> > > > >
> > > > > > Agree with David. We need to have an opporunity set backups count
> > > > threshold
> > > > > > (at runtime also!) that will not allow any automatic stop if
> there
> > > > will be
> > > > > > a data loss. Andrey, what do you think?
> > > > > >
> > > > > >

Speakers needed for Apache DC Roadshow

2018-09-11 Thread Rich Bowen

We need your help to make the Apache Washington DC Roadshow on Dec 4th a 
success.


What do we need most? Speakers!

We're bringing a unique DC flavor to this event by mixing Open Source 
Software with talks about Apache projects as well as OSS CyberSecurity, 
OSS in Government and and OSS Career advice.


Please take a look at: http://www.apachecon.com/usroadshow18/

(Note: You are receiving this message because you are subscribed to one 
or more mailing lists at The Apache Software Foundation.)


Rich, for the ApacheCon Planners

--
rbo...@apache.org
http://apachecon.com
@ApacheCon

[jira] [Created] (IGNITE-9549) control.sh add command to collect information on the distribution of partitions and reset lost partitions

2018-09-11 Thread Alexand Polyakov (JIRA)

Alexand Polyakov created IGNITE-9549:


 Summary: control.sh add command to collect information on the 
distribution of partitions and reset lost partitions
 Key: IGNITE-9549
 URL: https://issues.apache.org/jira/browse/IGNITE-9549
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexand Polyakov
Assignee: Alexand Polyakov


control.sh add command to collect information on the distribution of partitions

control.sh --cache distribution [nodeId|null] [cacheName1[,...,cacheNameN]] 
[--user-attributes userAttribute[,...,userAttributeN]]
return 
[next group: id=??, name=??]
groupId,partition,nodeId,primary,state,updateCounter,size,nodeAddresses[,userAttribute[,...,userAttributeN]]
...
groupId,partition,nodeId,primary,state,updateCounter,size,nodeAddresses[,userAttribute[,...,userAttributeN]]


reset lost partitions
control.sh --cache cacheName1[,...,cacheNameN]
return 
Reset LOST-partitions performed successfully. Cache group (name = '??', id = ??)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9548) Transaction with short timeout is not rolled back on primary node resulting in blocked PME

2018-09-11 Thread Alexei Scherbakov (JIRA)

Alexei Scherbakov created IGNITE-9548:
-

 Summary: Transaction with short timeout is not rolled back on 
primary node resulting in blocked PME
 Key: IGNITE-9548
 URL: https://issues.apache.org/jira/browse/IGNITE-9548
 Project: Ignite
  Issue Type: Bug
Reporter: Alexei Scherbakov
Assignee: Alexei Scherbakov


{noformat}
2018-09-10 12:38:24.237 [WARN 
][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.apache.ignite.internal.diagnostic]
 Pending transactions:
2018-09-10 12:38:24.242 [WARN 
][exchange-worker-#153%DPL_GRID%DplGridNodeName%][o.apache.ignite.internal.diagnostic]
 >>> [txVer=AffinityTopologyVersion [topVer=343, minorTopVer=0], exchWait=true, 
tx=GridDhtTxLocal [nearNodeId=eb94406c-a132-4998-bf22-b7d74960b866, nearFut
Id=b7cff46b561-0b500010-3ed6-4b79-8cc8-65b3b3b16738, nearMiniId=1, 
nearFinFutId=null, nearFinMiniId=0, nearXidVer=GridCacheVersion 
[topVer=147809766, order=1536687716227, nodeOrder=182], 
super=GridDhtTxLocalAdapter [nearOnOriginatingNode=false, nearNodes=[], 
dhtNodes=[],
 explicitLock=false, super=IgniteTxLocalAdapter [completedBase=null, 
sndTransformedVals=false, depEnabled=false, txState=IgniteTxStateImpl 
[activeCacheIds=[-1934881220], recovery=false, txMap=[IgniteTxEntry 
[key=KeyCacheObjectImpl [part=12715, val=ucp_ids_counter_name_DP
L_ucp_ids_section_name, hasValBytes=true], cacheId=-1934881220, 
txKey=IgniteTxKey [key=KeyCacheObjectImpl [part=12715, 
val=ucp_ids_counter_name_DPL_ucp_ids_section_name, hasValBytes=true], 
cacheId=-1934881220], val=[op=NOOP, val=null], prevVal=[op=NOOP, val=null], 
oldVal
=[op=NOOP, val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1, 
conflictVer=null, explicitVer=null, dhtVer=null, filters=[], 
filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=[], 
part=12715, super=GridDistributedCacheEntry [super=GridCach
eMapEntry [key=KeyCacheObjectImpl [part=12715, 
val=ucp_ids_counter_name_DPL_ucp_ids_section_name, hasValBytes=true], 
val=CacheObjectImpl [val=null, hasValBytes=true], startVer=1536665387604, 
ver=GridCacheVersion [topVer=147809766, order=1536737564543, nodeOrder=33], hash
=-864500235, extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc 
[locs=[GridCacheMvccCandidate [nodeId=a4823893-be8f-4b24-abca-0a28efde604a, 
ver=GridCacheVersion [topVer=147809766, order=1536737580715, nodeOrder=33], 
threadId=887, id=43118359, topVer=AffinityTopologyVers
ion [topVer=343, minorTopVer=0], reentry=null, 
otherNodeId=eb94406c-a132-4998-bf22-b7d74960b866, otherVer=GridCacheVersion 
[topVer=147809766, order=1536687716227, nodeOrder=182], mappedDhtNodes=null, 
mappedNearNodes=null, ownerVer=GridCacheVersion [topVer=147809766, orde
r=1536737580560, nodeOrder=33], serOrder=null, key=KeyCacheObjectImpl 
[part=12715, val=ucp_ids_counter_name_DPL_ucp_ids_section_name, 
hasValBytes=true], 
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
 prevV
er=null, nextVer=null]], rmts=null]], flags=2]]], prepared=0, locked=false, 
nodeId=null, locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=2, 
partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=147809766, 
order=1536737580715, nodeOrder=33
, super=IgniteTxAdapter [xidVer=GridCacheVersion [topVer=147809766, 
order=1536737580715, nodeOrder=33], writeVer=null, implicit=false, loc=true, 
threadId=887, startTime=1536416065902, 
nodeId=a4823893-be8f-4b24-abca-0a28efde604a, startVer=GridCacheVersion 
[topVer=14780976
6, order=1536737580715, nodeOrder=33], endVer=null, isolation=REPEATABLE_READ, 
concurrency=PESSIMISTIC, timeout=200, sysInvalidate=false, sys=false, plc=2, 
commitVer=null, finalizing=NONE, invalidParts=null, state=MARKED_ROLLBACK, 
timedOut=false, topVer=AffinityTopologyV
ersion [topVer=343, minorTopVer=0], duration=156238330ms, 
onePhaseCommit=false], size=1
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9547) Document DML operations prohibited inside transaction

2018-09-11 Thread Yury Gerzhedovich (JIRA)

Yury Gerzhedovich created IGNITE-9547:
-

 Summary: Document DML operations prohibited inside transaction
 Key: IGNITE-9547
 URL: https://issues.apache.org/jira/browse/IGNITE-9547
 Project: Ignite
  Issue Type: Task
  Components: documentation, sql
Reporter: Yury Gerzhedovich
 Fix For: 2.7


Docs says:

""Presently, DML supports the atomic mode only meaning that if there is a DML 
query that is executed as a part of an Ignite transaction then it will not be 
enlisted in the transaction's writing queue and will be executed right away""

However it's wrong.

We need to document that now any DML operations is prohibited and throw 
Exception in case it will be executed inside a transaction.

 

Also appeared new boolean property IGNITE_ALLOW_DML_INSIDE_TRANSACTION. it is 
necessary to emulate the old behavior. In case value is true then DML operation 
is allowed, but it be applied only after transaction will be commited.

By default value is false.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9546) Document GROUP_CONCAT aggregate function

2018-09-11 Thread Taras Ledkov (JIRA)

Taras Ledkov created IGNITE-9546:


 Summary: Document GROUP_CONCAT aggregate function
 Key: IGNITE-9546
 URL: https://issues.apache.org/jira/browse/IGNITE-9546
 Project: Ignite
  Issue Type: Task
  Components: sql
Affects Versions: 2.6
Reporter: Taras Ledkov
 Fix For: 2.7


We need to document GROUP_CONCAT aggregate function.
# Explain users that {{DISTINCT}} and {{ORDER BY}} is supported only for 
collocated {{GROUP BY}};
# String concatenation expressions are separated by {{||}} (not by comma as 
described at the 
[draft|https://dash.readme.io/project/apacheignite-sql/v2.6/docs/group_concat]);
 e.g.: {{GROUP_CONCAT(valueId || '=' || value SEPARATOR '; ')}}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4665: IGNITE-9426 IgniteAtomicSequence benchmarks

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4665


---

Re: TDE Implementation details.

2018-09-11 Thread Denis Magda

Hi Nikolay,

Has anybody started looking into your request? As I understand the plan is
to get TDE Phase 1 released in 2.7, right?
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473

Could you also list what needs to be done at Phase 2 and how much time it
might take.

--
Denis

On Thu, Aug 9, 2018 at 8:48 AM Nikolay Izhikov  wrote:

> Hello, Igniters.
>
> I want to share with you TDE implementation details.
> I think it can simplify review and acception of TDE implementation.
> This mail is copy of wiki page [1].
>
> Please, share your thoughts.
>
> TDE is ready for review [2].
> Please, let me know, who is able to make final review.
>
> This document describes some internal details of TDE.Phase 1
> implementation [3].
> I suggest that reader familiar with the general design described in IEP-18
> [4].
>
>
> Cache group key management and node join enhancements:
>
> 1. GridEncryptionManager encapsulates all logic related to key
> management:
> a. All group encryption keys are stored in MetaStore.
>
> b. Joining node sends to cluster:
> * Master key digest.
> * All group keys saved locally (Map byte[]>). Keys send over a network in encrypted form.
>
> c. Coordinator on new node join:
> * Compares new node master key digest with the
> local master key digest.
> If differs then new node join is rejected.
>
> * Compares local and received group keys.
> If some key differs then new node join is
> rejected.
>
> d. All server nodes:
> * If some of received keys are new then they save
> locally.
>
> 2. Dynamic cache creation:
> a. On server node - Encryption key is generated and added
> to DynamicCacheChangeRequest.
>
> b. On client node:
> * Prior to creation of DynamicCacheChangeRequest
> we have to get new encryption key from some server node.
> * New request added to solve this -
> GenerateEncryptionKeyRequest.
>
>
> WAL Record encryption:
>
>
> 1. New type of WAL record created – EncryptedRecord.
>
> 2. EncryptedRecord is a container that stores some
> WalRecordCacheGroupAware in encrypted form inside.
>
> 3. Write:
> Each record for an encrypted group that implements
> WalRecordCacheGroupAware written to WAL in encrypted form.
> Instead of that record we write EncryptedRecord with plain record
> inside(PageSnapshot, PageDeltaRecord, etc).
>
> 4. Read: There are 3 different cases on EncryptedRecord read:
> a. WAL restore – we read EncryptedRecord, decrypt internal
> record and return it.
>
> b. Offline WAL iteration(StandaloneWalRecordsIterator) -
> EncryptionSpi is null so wecan’t decrypt EncryptedRecord and just return it
> from an iterator.
>
> c. Meta storage restore process – On node start we restore
> MetaStore.
> When we do it no encryption keys are available, because
> they are stored in MetaStore.
> So we can’t decrypt EncryptedRecord and just return it
> from an iterator.
> We don't need decrypted record on this step to operate
> properly.
>
>
> Page encryption:
>
>
> 1. We have to write on disk pages aligned on 2^n (2kb, 4kb, etc)
> for gain maximum perfrormance.
>
> 2. There is a 16 byte overhead for and AES CBC mode.
>
> 3. So we have to preserve 16 bytes in page in memory to get
> encrypted page size equal to 2^n when written it to disk.
>
> 4. PageIO has many methods with pageSize parameter.
> So for encrypted cache groups page size is calculated as
> cfg.getPageSize() - 16 byte.
>
>
> [1]
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=89067473
> [2] https://github.com/apache/ignite/pull/4167
> [3] https://issues.apache.org/jira/browse/IGNITE-8485
> [4]
> https://cwiki.apache.org/confluence/display/IGNITE/IEP-18%3A+Transparent+Data+Encryption
>

[GitHub] SomeFire closed pull request #7: IGNITE-9542: new failure check fix

SomeFire closed pull request #7: IGNITE-9542: new failure check fix
URL: https://github.com/apache/ignite-teamcity-bot/pull/7
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js 
b/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
index 5652414..2b390f7 100644
--- a/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
+++ b/ignite-tc-helper-web/src/main/webapp/js/testfails-2.1.js
@@ -457,7 +457,9 @@ function showSuiteData(suite, settings) {
  * @returns {boolean} True - if test is new. False - otherwise.
  */
 function isNewFailedTest(testFailure) {
-return Number.parseFloat(testFailure.failureRate) < 4.0 || 
testFailure.flakyComments == null;
+var recent = testFailure.histBaseBranch.recent;
+
+return recent != null && Number.parseFloat(recent.failureRate) < 4.0 || 
testFailure.flakyComments == null;
 }
 
 function failureRateToColor(failureRate) {


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ignite pull request #4728: IGNITE-9516: workaround hanging vacuum after test

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4728


---

[jira] [Created] (IGNITE-9545) IgniteProjectionStartStopRestartSelfTest: misleading javadocs, required conditions are not described, inconvenient to configure locally

2018-09-11 Thread Oleg Ignatenko (JIRA)

Oleg Ignatenko created IGNITE-9545:
--

 Summary: IgniteProjectionStartStopRestartSelfTest: misleading 
javadocs, required conditions are not described, inconvenient to configure 
locally
 Key: IGNITE-9545
 URL: https://issues.apache.org/jira/browse/IGNITE-9545
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.6
Reporter: Oleg Ignatenko


This test has been [reported as 
flaky|https://ci.ignite.apache.org/project.html?projectId=IgniteTests24Java8&tab=testDetails&testNameId=2278553016619338221#analysis]
 at Teamcity. Looking closer shows that there are some issues with the test.

[IgniteProjectionStartStopRestartSelfTest|https://github.com/apache/ignite/blob/master/modules/ssh/src/test/java/org/apache/ignite/internal/IgniteProjectionStartStopRestartSelfTest.java]
 class javadocs provide instructions on how to configure test which have 
nothing to do with the way how it is actually configured. Not only the way is 
different but even respective property names are incorrect, which is easy to 
see from very first 3 statements in test code that initialize configuration.

Checking git history of this file shows that root cause for this issue is a 
change made about 4 years ago when obtaining test properties has changed from 
{{GridTestProperties.getProperty}} to {{System.getenv}} (back then, also 
property names have changed) but test javadoc was not updated to reflect that.

Another issue with javadoc which makes it unnecessarily difficult to 
investigate failures is that it doesn't explain that test expects configured 
target host to run ssh server and accept connections at configured port from 
user with specified credentials.

Javadocs need to be corrected.

Another issue with the test is the way it obtains the config (username and 
password): when I tried to do some quick experiments on my machine it turned 
out fairly difficult to set to what I wanted. When I tried to change test code 
to obtain config in the way how it was in the past (via {{GridTestProperties}}) 
it went much easier.

One good thing of the current way is it has proven to work well on Teamcity and 
because of that it makes sense to keep it. But on the other hand it looks 
desirable to augment it with fallback to the way that is more convenient for 
local experimenting.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4728: IGNITE-9516: workaround hanging vacuum after test

2018-09-11 Thread pavlukhin

GitHub user pavlukhin opened a pull request:

https://github.com/apache/ignite/pull/4728

IGNITE-9516: workaround hanging vacuum after test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9516

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4728.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4728


commit 1dfc1a5660196a5ce72c2fe559c915f97936cd60
Author: ipavlukhin 
Date:   2018-09-11T09:11:33Z

fail tests depending on Cache.size for cache API with issue reference, set 
timeout for vacuum cleanup after test




---

[GitHub] SomeFire opened a new pull request #7: IGNITE-9542: new failure check fix

SomeFire opened a new pull request #7: IGNITE-9542: new failure check fix
URL: https://github.com/apache/ignite-teamcity-bot/pull/7
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[jira] [Created] (IGNITE-9544) BinaryOutputStream#writeByte excessive copying after reaching 1Gb

2018-09-11 Thread Alexand Polyakov (JIRA)

Alexand Polyakov created IGNITE-9544:


 Summary: BinaryOutputStream#writeByte excessive copying after 
reaching 1Gb
 Key: IGNITE-9544
 URL: https://issues.apache.org/jira/browse/IGNITE-9544
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexand Polyakov


need correction 
org.apache.ignite.internal.binary.streams.BinaryAbstractOutputStream#capacity
{code:java}
if (newCap < reqCap)
    newCap=Integer.MAX_VALUE - 8;
{code}
see java.util.ArrayList#MAX_ARRAY_SIZE



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4727: MVCC branch

2018-09-11 Thread gvvinblade

GitHub user gvvinblade opened a pull request:

https://github.com/apache/ignite/pull/4727

MVCC branch



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-mvcc-b3

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4727.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4727


commit 197332fbe22f8b2ac21bf31ea444c4da218e56b4
Author: Igor Seliverstov 
Date:   2018-09-11T13:26:02Z

fake commit




---

[jira] [Created] (IGNITE-9543) MVCC TX: Add Mvcc atomicity mode to ConfigVariations tests when full cache API support is implemented.

Roman Kondakov created IGNITE-9543:
--

 Summary: MVCC TX: Add Mvcc atomicity mode to ConfigVariations 
tests when full cache API support is implemented.
 Key: IGNITE-9543
 URL: https://issues.apache.org/jira/browse/IGNITE-9543
 Project: Ignite
  Issue Type: Task
  Components: mvcc
Reporter: Roman Kondakov


We need to add  {{TRANSACTIONAL_SNAPSHOT}} atomicity mode to 
{{ConfigVariations#BASIC_CACHE_SET}} when full Cache API support is implemented 
for MVCC caches (i.e. near/local caches, read/write through, etc.).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4726: Ignite 2.5.1 p12 pme speedup

2018-09-11 Thread Jokser

GitHub user Jokser opened a pull request:

https://github.com/apache/ignite/pull/4726

Ignite 2.5.1 p12 pme speedup



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite 
ignite-2.5.1-p12-pme-speedup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4726.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4726


commit a7dbea16064bbd52907a770bb40c3a2445313db2
Author: Dmitriy Sorokin 
Date:   2018-04-17T11:48:44Z

IGNITE-8255 Possible name collisions in WorkersRegistry.

Signed-off-by: Andrey Gura 

commit b762d681b97ea121a8321eb66bf02f89a1d177cd
Author: Aleksey Plekhanov 
Date:   2018-04-17T12:56:36Z

IGNITE-8166 PME hangs when error occurs during checkpoint

Signed-off-by: Andrey Gura 

commit 8428b0e63e97c10277f6d9e1640e16528a772270
Author: Ivan Daschinskiy 
Date:   2018-04-17T15:05:42Z

IGNITE-8021 Delete cache config files when cache is destroyed - Fixes #3697.

Signed-off-by: Alexey Goncharuk 

commit cd59c8e64f05ca03c7da8dc35d027a14fcebf250
Author: Aleksey Plekhanov 
Date:   2018-04-17T15:27:53Z

IGNITE-8033 Fixed flaky failure of 
TxOptimisticDeadlockDetectionCrossCacheTest

Signed-off-by: Andrey Gura 

commit acfef907db8204ac93fc235770f36bf7f61269c3
Author: Ilya Kasnacheev 
Date:   2018-04-17T16:50:51Z

IGNITE-2766 Fix .net test. - Fixes #3853.

Signed-off-by: dpavlov 

(cherry picked from commit 96cb795)

commit 6cea78e4e13fe43555b78dcd683366f54c6816ff
Author: Andrey Kuznetsov 
Date:   2018-04-17T16:58:43Z

IGNITE-7770 Test testRandomMixedTxConfigurations partialy fixed

Signed-off-by: Andrey Gura 

commit e394693a7389b4daff328827abdb1dcd28783f66
Author: Maxim Muzafarov 
Date:   2018-04-17T18:18:36Z

IGNITE-8301 testReconnectCacheDestroyedAndCreated should excpect recreated 
client cache - Fixes #3856.

Signed-off-by: dpavlov 

(cherry picked from commit 56be24b)

commit 4685ebe5f5dda4023980398806e222fada895e26
Author: Vasiliy Sisko 
Date:   2018-04-18T03:44:44Z

IGNITE-8140 Web Console: Fixed code generation for large numbers in 
configuration params.

(cherry picked from commit eda5fe7)

commit 5efc589fcabffdb29cd6dfe0e7323bc91db47703
Author: Ilya Borisov 
Date:   2018-04-18T04:39:41Z

IGNITE-8294 Web Console: Move "Beta" ribbon to the left.

(cherry picked from commit 69606e4)

commit af46856d7f7ec88aa663865a108900abb8314ffe
Author: oleg-ostanin 
Date:   2018-04-17T17:58:53Z

IGNITE-8274 sqlline.sh script uses JAVA_HOME now

Signed-off-by: Andrey Gura 

(cherry picked from commit c3ff274)

commit ba4e337068ebfcc4bbb93509166f118225fe0cb4
Author: Dmitriy Shabalin 
Date:   2018-04-18T11:43:13Z

IGNITE-8298 Web Console: Fixed tables UI issues.

(cherry picked from commit a050436)

commit 25b25f271bc89d77013e1cda2bad30d941e1c8ad
Author: Pavel Kovalenko 
Date:   2018-04-18T12:57:45Z

IGNITE-8122 Restore partition state from WAL if no checkpoints are done - 
Fixes #3745.

Signed-off-by: Alexey Goncharuk 

commit 4ab86f13cbc7fef68f5fd39354e242fe6cca285a
Author: skalashnikov 
Date:   2018-04-18T13:37:20Z

IGNITE-7512 Check for null before validateKeyAndValue in 
GridDhtAtomicCache.updateWithBatch - Fixes #3429.

Signed-off-by: Alexey Goncharuk 

commit 7f318da5b357cca1a7e483a37d2556fbd56fc2a7
Author: Ilya Lantukh 
Date:   2018-04-18T14:56:56Z

IGNITE-8017 Disable WAL during initial rebalancing

commit b80fdbedc11b48aa89c6f29146363cec6f552abb
Author: Ilya Lantukh 
Date:   2018-04-18T16:04:39Z

IGNITE-8276 Fixed incorrect assertion during initialValue - Fixes #3827.

Signed-off-by: Alexey Goncharuk 

commit 5cd32329fe5f303eaf369519771ee22f2a2cf822
Author: Pavel Kovalenko 
Date:   2018-04-18T16:41:44Z

IGNITE-8116 Fixed historical rebalance from WAL

commit 442d87d5799c27ef7258c7ac36e3e0b312676713
Author: Pavel Kovalenko 
Date:   2018-04-18T18:03:32Z

IGNITE-8243 Fixed possible memory leak. Added latch manager to diagnostic 
messages. - Fixes #3850.

Signed-off-by: dpavlov 

(cherry picked from commit 6c6f094)

commit 5bc9fe9fa4e51c9af677f8120528746f345d990f
Author: dpavlov 
Date:   2018-04-18T18:15:17Z

IGNITE-8302 Reduce test IO consumption in case Direct IO is enabled, fixes 
flakyness of testPageRecoveryAfterFileCorruption - Fixes #3865.

Signed-off-by: dpavlov 

(cherry picked from commit d853ebb)

commit 3696c064f02567f471420ced50e2786a4f83d8c9
Author: dpavlov 
Date:   2018-04-18T18:16:35Z

IGNITE-8302 Licenses fix for Reduce test IO consumption in case Direct IO

(cherry picked from commit e7716f4)

commit 061a5a82c95f4f926c7789764901b70383152df1
Author: shroman 
Date:   2018-04-18T10:38:52Z

IGNITE-

[GitHub] ignite pull request #4725: IGNITE-7764: MVCC TX: make cache basic operations...

2018-09-11 Thread AMashenkov

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/4725

IGNITE-7764: MVCC TX: make cache basic operations support Mvcc tx mode.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7764

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4725.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4725


commit ed99e64194b880230051b011b3a3a8e234f51654
Author: Andrey V. Mashenkov 
Date:   2018-09-06T07:45:56Z

IGNITE-9464: WIP. Tests added.

Signed-off-by: Andrey V. Mashenkov 

commit 427c4aecbc3f88e351d5da74ada07d72bcae52d1
Author: Andrey V. Mashenkov 
Date:   2018-09-06T14:28:53Z

IGNITE-9464: WIP. Fix Get\GetAll operations.

Signed-off-by: Andrey V. Mashenkov 

commit 1f410ff3cf58b4dea619ddc7ee5ba00f378046b2
Author: Andrey V. Mashenkov 
Date:   2018-09-07T19:26:49Z

IGNITE-9464: WIP. Fix Put\PutAll operations. Naive implementation, 
optimization needed.

Signed-off-by: Andrey V. Mashenkov 

commit b66d8369f5226768f668650b5c56d44926f714cf
Author: Andrey V. Mashenkov 
Date:   2018-09-08T15:01:38Z

IGNITE-9464: WIP. Fix test.

Signed-off-by: Andrey V. Mashenkov 

commit fc500e945280b1d59a72a61ceaf655a9d86538c6
Author: Andrey V. Mashenkov 
Date:   2018-09-10T10:06:06Z

IGNITE-9464: WIP. Refactored query enlist futures.

Generify query enlist futures.

Signed-off-by: Andrey V. Mashenkov 




---

[GitHub] ignite pull request #4724: IGNITE-9519: PK as complex type should can be kee...

2018-09-11 Thread ygerzhedovich

GitHub user ygerzhedovich opened a pull request:

https://github.com/apache/ignite/pull/4724

IGNITE-9519: PK as complex type should can be keep at inline index



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9519

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4724.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4724


commit 8b96932e622afdd79fb4cc62b6e5ce0ddf0f497f
Author: Yury Gerzhedovich 
Date:   2018-09-10T15:42:14Z

IGNITE-9519: PK as complex type should can be keep at inline index




---

[GitHub] ignite pull request #4723: IGNITE-9249

2018-09-11 Thread ilantukh

GitHub user ilantukh opened a pull request:

https://github.com/apache/ignite/pull/4723

IGNITE-9249



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9249

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4723.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4723


commit 3be4dcc6da6649fb04f99f61c31cebb29d03c0fe
Author: Ilya Lantukh 
Date:   2018-08-10T13:23:28Z

IGNITE-9249 : Configured node join timeout for all tests.

commit a5ef89937689f29a10930ff9b38f525635d498b2
Author: Ilya Lantukh 
Date:   2018-09-11T12:43:21Z

IGNITE-9249 : Join timeout should be checked even with shared IP finder.




---

[GitHub] ignite pull request #4721: IGNITE-9535 ClientChangeGlobalStateComputeRequest...

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4721


---

[GitHub] asfgit closed pull request #3: IGNITE-9333 Add statistics page

asfgit closed pull request #3: IGNITE-9333 Add statistics page
URL: https://github.com/apache/ignite-teamcity-bot/pull/3
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
index 9782cbf..d9d8d1d 100644
--- a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
+++ b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/ITeamcity.java
@@ -33,6 +33,7 @@
 import org.apache.ignite.ci.tcmodel.conf.BuildType;
 import org.apache.ignite.ci.tcmodel.hist.BuildRef;
 import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.issues.IssuesUsagesList;
 import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrence;
 import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrences;
 import org.apache.ignite.ci.tcmodel.result.stat.Statistics;
@@ -52,6 +53,7 @@
 public interface ITeamcity extends AutoCloseable {
 
 String DEFAULT = "";
+long DEFAULT_BUILDS_COUNT = 1000;
 
 CompletableFuture> getProjectSuites(String projectId);
 
@@ -62,7 +64,17 @@
  * @param branch
  * @return list of builds in historical order, recent builds coming last
  */
-List getFinishedBuilds(String projectId, String branch);
+default List getFinishedBuilds(String projectId, String branch) {
+return getFinishedBuilds(projectId, branch, null);
+};
+
+/**
+ * @param projectId suite ID (string without spaces)
+ * @param branch
+ * @param cnt builds count
+ * @return list of builds in historical order, recent builds coming last
+ */
+List getFinishedBuilds(String projectId, String branch, Long 
cnt);
 
 /**
  * Includes snapshot dependencies failed builds into list
@@ -71,7 +83,19 @@
  * @param branch branch in TC identification
  * @return list of builds in historical order, recent builds coming last
  */
-List getFinishedBuildsIncludeSnDepFailed(String projectId, 
String branch);
+default List getFinishedBuildsIncludeSnDepFailed(String 
projectId, String branch){
+return getFinishedBuilds(projectId, branch, null);
+};
+
+/**
+ * Includes snapshot dependencies failed builds into list
+ *
+ * @param projectId suite ID (string without spaces)
+ * @param branch branch in TC identification
+ * @param cnt builds count
+ * @return list of builds in historical order, recent builds coming last
+ */
+List getFinishedBuildsIncludeSnDepFailed(String projectId, 
String branch, Long cnt);
 
 /**   */
 CompletableFuture> getRunningBuilds(@Nullable String 
branch);
@@ -80,7 +104,11 @@
 CompletableFuture> getQueuedBuilds(@Nullable String branch);
 
 default int[] getBuildNumbersFromHistory(String projectId, String 
branchNameForHist) {
-return getFinishedBuilds(projectId, 
branchNameForHist).stream().mapToInt(BuildRef::getId).toArray();
+return getBuildNumbersFromHistory(projectId, branchNameForHist, null);
+}
+
+default int[] getBuildNumbersFromHistory(String projectId, String 
branchNameForHist, Long cnt) {
+return getFinishedBuilds(projectId, branchNameForHist, 
cnt).stream().mapToInt(BuildRef::getId).toArray();
 }
 
 Build getBuild(String href);
@@ -114,6 +142,8 @@ default Build getBuild(int id) {
 
 ChangesList getChangesList(String href);
 
+IssuesUsagesList getIssuesUsagesList(String href);
+
 /**
  * Runs deep collection of all related statistics for particular build
  *
diff --git 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
index 99f61bb..126ad98 100644
--- 
a/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
+++ 
b/ignite-tc-helper-web/src/main/java/org/apache/ignite/ci/IgnitePersistentTeamcity.java
@@ -36,6 +36,7 @@
 import java.util.function.BiFunction;
 import java.util.function.Function;
 import java.util.function.Predicate;
+import java.util.stream.Collectors;
 import java.util.stream.Stream;
 import java.util.stream.StreamSupport;
 import javax.annotation.Nullable;
@@ -57,6 +58,7 @@
 import org.apache.ignite.ci.tcmodel.conf.BuildType;
 import org.apache.ignite.ci.tcmodel.hist.BuildRef;
 import org.apache.ignite.ci.tcmodel.result.Build;
+import org.apache.ignite.ci.tcmodel.result.issues.IssuesUsagesList;
 import org.apache.ignite.ci.tcmodel.result.problems.ProblemOccurrences;
 import org.apache.ignite.ci.tcmodel.result.stat.Statistics;
 import org.apache.ignite.ci.tcmodel.result.tests.TestOccurrence;
@@ -84,6 +86,7 @@

[jira] [Created] (IGNITE-9542) Ignite TC bot: provide separated base/current branch history for PR page

2018-09-11 Thread Dmitriy Pavlov (JIRA)

Dmitriy Pavlov created IGNITE-9542:
--

 Summary: Ignite TC bot: provide separated base/current branch 
history for PR page
 Key: IGNITE-9542
 URL: https://issues.apache.org/jira/browse/IGNITE-9542
 Project: Ignite
  Issue Type: Bug
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov


https://issues.apache.org/jira/browse/IGNITE-9376
works incorrectly without separation of test history for the base and for a 
shown branch.

It is suggested to refactor provided data for the current and for the base 
branch.

It will allow showing blockers based on statistics from the base branch.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Critical worker threads liveness checking drawbacks

2018-09-11 Thread vgrigorev

Reliability of ignite is very important to me, so please consider following
idea:

- Important threads as WAL writer (as a sample of any critical thread)
must not do any blocking action, by this way:
   - WAL thread  must be management thread for all WAL operations
   - Child, worker thread of WAL writer must do separate operations which
implements concrete WAL writings
   - Operations are separate units of work, countable by it's heartbeat for
sample and has characteristics
   and ids. 
   - Operations written in queue and has state.
   - If hung occur in a concrete operation, this operation may be cancelled,
(all child operations in a cluster too) and all others operations continue
to work, with failed operation go to recovery state or report user about
fail
   - If WAL child thread do infinite blocking operation, it's need to kill
this working thread and start new with same queue of operations of WAL type

So, we become able :
- always know what concrete operation  are in hung, (not that whole main WAL
thread hung) so can better decide want to do.
- WAL thread operations newer irresponsive, at minimum it reports that it
long doing some operation and just can insert next operation queue or
propose fail
- report size of queue and else full detail information about what happening
and allow to decide precisely - fail concrete user operations, clean
resources, spawn new working thread or else, and continue to work without
painful node or cluster restart
- minimal cleanless possible (just some operations)
- balance operations with queues, also implementing backpressure, so make
sure that optimal performance load is kept and cluster will not go to
degradation from some local oversaturations
- newer see that node hung, but just degrade and being in fully controlled
state 

- WAL thread operations check management functions can be encapsulated to
special class with that functionality and called from else main threads as
now.

Sorry for any inconvenience, I'm new to writing here



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[jira] [Created] (IGNITE-9541) Add the comparison for two general statistics "RunAll" for master in the date interval

2018-09-11 Thread Nikolai Kulagin (JIRA)

Nikolai Kulagin created IGNITE-9541:
---

 Summary: Add the comparison for two general statistics "RunAll" 
for master in the date interval
 Key: IGNITE-9541
 URL: https://issues.apache.org/jira/browse/IGNITE-9541
 Project: Ignite
  Issue Type: Sub-task
Reporter: Nikolai Kulagin


Based on IGNITE-9333 add the comparison for two general statistics "RunAll" for 
master in the date interval



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Apache Ignite 2.7 release

2018-09-11 Thread Ilya Kasnacheev

Hello!

Thank you for the clarifications! Your change looks good to me now.

Regards,
-- 
Ilya Kasnacheev


вт, 11 сент. 2018 г. в 5:53, Roman Shtykh :

> Ilya,
>
> The "latest" version is the default, and resolved by
> https://ignite.apache.org/latest which is used by our web site when a
> user download the latest Ignite version. And I think this is the authority
> to judge of the latest official release (pom.xml you suggest can have
> SNAPSHOTs etc.).
> Also, as I explained during our review sessions, ignite-mesos-2.6.0 is a
> driver and doesn't mean you need to have Ignite 2.6.0. User can run any
> version of Ignite he/she specifies. By default, it's "latest" but a user
> can specify any version needed, even from a non-archive URL.
>
> In short, what we have now
> 1. mesos driver (ignite-mesos-x.x.x) will use "latest" version by default
> -> it will try to resolve the latest officially releases version of Apache
> Ignite, find the closest mirror and download Ignite in a minute. If the
> version resolution fails, we fall back to the slow apache archive (as you
> suggest; in my opinion we better fail-fast instead of waiting for hours to
> download, so the user can choose another download option (3))
> 2. If the user specifies the version explicitly, it goes to the slow
> apache archive.
> 3. The user can put ignite zip file on his/her http server and provide the
> URL as a parameter to the driver, if options 1 and 2 don't work.
>
> As you see, there are 3 options. And I just fix the 1st one with
> https://issues.apache.org/jira/browse/IGNITE-9388 and don't change the
> original logic (which I find reasonable) documented on our site -- I don't
> see how it blocks anything.
>
> Roman Shtykh
>
>
> On Monday, September 10, 2018, 6:16:15 p.m. GMT+9, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>
> Hello!
>
> There's still two issues with the submission.
>
> The first one is that we're downloading "latest" version from preferred
> mirror but a specified version, such as "2.6", we're also going to download
> from "slow" archive.apache.org/dist.
> That's a great limitation for this change, since most real deployments of
> Apache Ignite will have their Ignite version pegged to a specific release.
> But in this case there's no win in download speed.
> *In my opinion it is a blocker.*
>
> The second one is that we can't download anything when we failed to
> resolve "latest". My idea is that we should try and download last known
> version in this case, which can be pushed to source from pom.xml, as we
> already do with URLs. So if you could not resolve "latest" you will
> download 2.7.0.
>
> Buuut, maybe it's not necessary, maybe we should just *discourage
> "latest"*, which is in my opinion almost always a bad idea.
>
> WDYT?
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вс, 9 сент. 2018 г. в 5:47, Roman Shtykh :
>
> Hi Ilya,
>
> Sorry, missed that.
> Added now.
>
> --
> Roman Shtykh
>
>
> On Thursday, September 6, 2018, 6:16:58 p.m. GMT+9, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>
> Hello!
>
> The last of my requests still standing is that we should fall-back to
> single URL download in case of error with 'latest' version. Everything else
> looks good to me.
>
> Can we do that? I'm really worried that Apache API will go sour.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> чт, 6 сент. 2018 г. в 8:56, Roman Shtykh :
>
> Hi Ilya,
>
> Thanks again.
>
> 1) Done.
> 2) Used catch() for latest version.
>
> Please see my comments on github.
> --
> Roman Shtykh
>
>
> On Wednesday, September 5, 2018, 11:30:10 p.m. GMT+9, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>
> Hello!
>
> I've left a new wave of replies.
>
> Basically, 1) let's keep DOWNLOAD_URL_PATTERN string value inlined so
> that it will work even if build process is broken (would be useful for e.g.
> developing out of IDE)
> And also I urge you to catch() around new fragile Apache JSON API
> resolving, and download the 'current' version instead, as defined by
> ignite-mesos version.
>
> This is because this module is not under continuouos scrutiny so extra
> care should be applied.
> --
> Ilya Kasnacheev
>
>
> вт, 4 сент. 2018 г. в 13:42, Roman Shtykh :
>
> Thanks, Ilya!
> I will check your comments, and discuss it at JIRA.
>
> --
> Roman Shtykh
>
>
> On Tuesday, September 4, 2018, 7:17:53 p.m. GMT+9, Ilya Kasnacheev <
> ilya.kasnach...@gmail.com> wrote:
>
>
> Hello!
>
> IGNITE-9408  looks
> good to me and may be merged right away.
>
> IGNITE-9388  needs
> more work in my opinion, I have commented the PR. I also advice having test
> for this functionality.
>
> Regards,
> --
> Ilya Kasnacheev
>
>
> вт, 4 сент. 2018 г. в 6:52, Roman Shtykh :
>
> Igniters,
> I would like Mesos integration update be included in the upcoming
> release.Can anyone review prs for the following issues?
> IGNITE-9388: mesos IgniteProvider tries to access obsole

[jira] [Created] (IGNITE-9540) MVCC TX: make cache invoke\invokeAll operations support Mvcc tx mode.

2018-09-11 Thread Andrew Mashenkov (JIRA)

Andrew Mashenkov created IGNITE-9540:


 Summary: MVCC TX: make cache invoke\invokeAll operations support 
Mvcc tx mode.
 Key: IGNITE-9540
 URL: https://issues.apache.org/jira/browse/IGNITE-9540
 Project: Ignite
  Issue Type: New Feature
Reporter: Andrew Mashenkov






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Support TextQuery for ThinClient

2018-09-11 Thread Igor Sapego

Great,

Let us know if you need any kind of help from the community.

Best Regards,
Igor


On Tue, Sep 11, 2018 at 10:48 AM Pavel Tupitsyn 
wrote:

> Awesome!
> I've filed a JIRA ticket [1], please assign to yourself, click Start
> Progress, and follow the contribution path [2].
> Looking forward to a pull request from you!
>
> [1] https://issues.apache.org/jira/browse/IGNITE-9529
> [2]
>
> https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute#HowtoContribute-1.CreateGitHubpull-request
>
> On Mon, Sep 10, 2018 at 10:54 PM Tâm Nguyễn Mạnh <
> nguyenmanhtam...@gmail.com>
> wrote:
>
> > Hi,
> > Yes, im going to add into .Net Thin Client.
> > I Already looked into implementation of other Thin Client’s methods. Its
> > quite easy to implement TextQuery for .Net Thi Client.
> > Actually I did it on my personal branch and I would love to push it into
> > Ignite git.
> >
> >
> >
> >
> > Vào Th 3, 11 thg 9, 2018 lúc 2:47 SA Pavel Tupitsyn <
> ptupit...@apache.org>
> > đã viết:
> >
> > > Hi,
> > >
> > > Text query should be easy to implement in thin client protocol, all the
> > > plumbing is there for cursors and so on.
> > > We just need a new operation similar to existing ones.
> > >
> > > Are you going to add it to .NET Thin Client?
> > >
> > > Pavel
> > >
> > > On Mon, Sep 10, 2018 at 8:22 PM Dmitriy Pavlov 
> > > wrote:
> > >
> > > > Hi, sure, feel free to open a ticket.  About implementation, I
> > personally
> > > > do not mind, but I can't represent an opinion of all community
> members.
> > > >
> > > > I hope TextQuery/Thin Client developers and maintainers will step in
> > and
> > > > share their vision.
> > > >
> > > > пн, 10 сент. 2018 г. в 16:31, Tâm Nguyễn Mạnh <
> > > nguyenmanhtam...@gmail.com
> > > > >:
> > > >
> > > > > Hi,
> > > > > I would like to implement this feature. Can we open a ticket for
> > that ?
> > > > >
> > > > > Vào Th 2, 10 thg 9, 2018 lúc 8:27 CH Dmitriy Pavlov <
> > > > dpavlov@gmail.com
> > > > > >
> > > > > đã viết:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I've didn't heard about such plans. TextQuery is not widely used,
> > as
> > > > far
> > > > > as
> > > > > > I know.
> > > > > >
> > > > > > Hope this helps.
> > > > > >
> > > > > > Sincerely,
> > > > > > Dmitriy Pavlov
> > > > > >
> > > > > > пт, 7 сент. 2018 г. в 4:57, Tâm Nguyễn Mạnh <
> > > > nguyenmanhtam...@gmail.com
> > > > > >:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Is there any plan about support TextQuery for ThinClient ?
> > > > > > >
> > > > > > > --
> > > > > > > Thanks & Best Regards
> > > > > > >
> > > > > > > Tam, Nguyen Manh
> > > > > > >
> > > > > >
> > > > > --
> > > > > Thanks & Best Regards
> > > > >
> > > > > Tam, Nguyen Manh
> > > > >
> > > >
> > >
> > --
> > Thanks & Best Regards
> >
> > Tam, Nguyen Manh
> >
>

Re: PHP thin client

2018-09-11 Thread Igor Sapego

1) I meant auto-generated doc, this page: [1]. By the way,
many methods do not have any description as well.
2) You can, for example, print object as you generate and
then get them from cache.
3) Yes, 10800 should be the default port.
4) You are right, I've just used server with SSL enabled.
Everything works now.
5) Ok, can you just provide a list of directories/files, that
should be copied to binary release?

[1] -
https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html

Best Regards,
Igor


On Tue, Sep 11, 2018 at 1:31 AM Alexey Kosenchuk <
alexey.kosenc...@nobitlost.com> wrote:

> Hi Igor,
>
> thanks for the review.
>
> Pls see below...
>
>  >> 1. Main page for documentation is empty.
>
> What is the main page for documentation?
>
> As wrote,
> the auto-generated API spec is here:
>
> https://rawgit.com/nobitlost/ignite/ignite-7783-docs/modules/platforms/php/api_docs/html/index.html
>
> All other docs are here:
>
> https://github.com/nobitlost/ignite/blob/ignite-7783-docs/modules/platforms/php/README.md
> Going to be placed on readme.io by Prachi (thanks!)
> https://issues.apache.org/jira/browse/IGNITE-9523
>
>  >> 2. More output for Auth example is needed. This is not a test, after
> all, but example.
>
> Will add a log when a connection happens with the details of the
> connection. Nothing else to output in this trivial example.
> In fact, the real profit for a user here is the source code with an
> example of TLS/auth cfg for the client.
>
>  >> 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1,
> they fail
>
> Do you mean 10800 should be the default port when not specified by a user?
> Will add.
>
>  >> 4. If I try run test with
> APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1:10800, they fail
>
> Double checked exactly in the same environment you have. It works.
>
> Please try again.
> Do you use a server with all default settings? Did not you try with the
> server after the Auth example?
> Do other examples work with the same server?
> If you still see the problem with the tests, please send the server's
> log to us. And/or switch the client's debug on - call setDebug(true) -
> and share the client's output.
>
>  >> 5. When "maven package" command is executed on Ignite, no php
> directory appears
>
> Need a help from experts / release engineer.
>
> Thanks,
> -Alexey
>
> 10.09.2018 15:34, Igor Sapego пишет:
> > By the way, I used Ubuntu 18.04, PHP 7.2.7 and what seems
> > to be PhpUnit 7.3 (not sure here).
> >
> > Best Regards,
> > Igor
> >
> >
> > On Mon, Sep 10, 2018 at 3:28 PM Igor Sapego  wrote:
> >
> >> Guys, I've reviewed the API (which looks good), run tests and examples
> and
> >> here are my commments:
> >>
> >> 1. Main page for documentation is empty.
> >>
> >> 2. More output for Auth example is needed. This is not a test, after
> all,
> >> but example.
> >>
> >> 3. If I try run test with APACHE_IGNITE_CLIENT_ENDPOINTS=127.0.0.1, they
> >> fail with the following message:
> >> Apache\Ignite\Exception\NoConnectionException: [127.0.0.1] Failed to
> parse
> >> address "127.0.0.1" in
> >>
> /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php:107
> >> Stack trace:
> >> #0
> >>
> /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Internal/Connection/ClientFailoverSocket.php(54):
> >>
> Apache\Ignite\Internal\Connection\ClientFailoverSocket->failoverConnect()
> >> #1
> >>
> /home/isapego/work/ignite/modules/platforms/php/src/Apache/Ignite/Client.php(61):
> >>
> Apache\Ignite\Internal\Connection\ClientFailoverSocket->connect(Object(Apache\Ignite\ClientConfiguration))
> >> #2
> >>
> /home/isapego/work/ignite/modules/platforms/php/tests/TestingHelper.php(52):
> >> Apache\Ignite\Client->connect(Object(Apache\Ignite\ClientConfiguration))
> >> #3
> >>
> /home/isapego/work/ignite/modules/platforms/php/tests/SqlQueryTest.php(49):
> >> Apache\Ignite\Tests\TestingHelper::init()
> >> #4
> >>
> /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/Framework/TestSuite.php(703):
> >> Apache\Ignite\Tests\SqlQueryTestCase::setUpBeforeClass()
> >> #5
> >>
> /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/Framework/TestSuite.php(750):
> >> PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
> >> #6
> >>
> /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(587):
> >> PHPUnit\Framework\TestSuite->run(Object(PHPUnit\Framework\TestResult))
> >> #7
> >>
> /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/TextUI/Command.php(203):
> >> PHPUnit\TextUI\TestRunner->doRun(Object(PHPUnit\Framework\TestSuite),
> >> Array, true)
> >> #8
> >>
> /home/isapego/work/ignite/modules/platforms/php/vendor/phpunit/phpunit/src/TextUI/Command.php(159):
> >> PHPUnit\TextUI\Command->run(Array, true)
> >> #9
> >>
> /home/isapego/work/ignite/modules/platforms/php/

[jira] [Created] (IGNITE-9538) MVCC TX: Send partition update counters to backup nodes on prepare state.

2018-09-11 Thread Igor Seliverstov (JIRA)

Igor Seliverstov created IGNITE-9538:


 Summary: MVCC TX: Send partition update counters to backup nodes 
on prepare state.
 Key: IGNITE-9538
 URL: https://issues.apache.org/jira/browse/IGNITE-9538
 Project: Ignite
  Issue Type: Task
  Components: cache, mvcc
Reporter: Igor Seliverstov
Assignee: Igor Seliverstov
 Fix For: 2.7


There are several issues with partition update counters consistency in 
transactional caches. The next approach solves most of them:
 # Count per-partition updates
 # on prepare state on primary node update current partition counter 
incrementing it by per-partition updates count and send initial value with 
updates count to backup nodes
 # on backup nodes hold all pending updates and update partition update counter 
applying the lowest gapless update (on tx finish).
 # on historical rebalance use partition update counter as start point.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9539) Add SQL column precision existence check if scale parameter is setted

2018-09-11 Thread PetrovMikhail (JIRA)

PetrovMikhail created IGNITE-9539:
-

 Summary: Add SQL column precision existence check if scale 
parameter is setted
 Key: IGNITE-9539
 URL: https://issues.apache.org/jira/browse/IGNITE-9539
 Project: Ignite
  Issue Type: Bug
Reporter: PetrovMikhail
Assignee: PetrovMikhail


We should check situation when user sets only scale parameter without precision 
for column, while creating SQL table. According to 
[http://www.h2database.com/html/datatypes.html#decimal_type] it's not valid 
behavior.

Currently, we ignore it.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4722: IGNITE-9531: ZookeeperDiscovery testClientReconne...

2018-09-11 Thread avplatonov

GitHub user avplatonov opened a pull request:

https://github.com/apache/ignite/pull/4722

IGNITE-9531: ZookeeperDiscovery testClientReconnect is flaky in master



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9531

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4722.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4722


commit f6ab10a1a78f4c17ba133acdbae76516b5c0d0ac
Author: Alexey Platonov 
Date:   2018-09-11T10:55:27Z

add test assertions

commit 5d79313bd92d66c562365d8287536b011c0597cc
Author: Alexey Platonov 
Date:   2018-09-11T10:55:44Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-9531




---

Re: Ignite Spark Bugs

Hello, Denis.

Thanks for providing reproducers for bugs!

I taked a look at your project and be able to reproduce some of errors.
I will create a ticket and start investigation in a next few days.

В Вт, 11/09/2018 в 12:28 +0300, Dmitriy Pavlov пишет:
> Hi Denis,
> 
> Thank you for bringing this here and for your efforts to reproduce it.
> Would you like to create an issue and contribute these test into Ignite
> code base?
> 
> If you would like to proceed with the patch submission process, please
> sign-in to Apache JIRA and share your JIRA username, I will add you as the
> contributor.
> 
> You can count on my support.
> 
> Sincerely,
> Dmitriy Pavlov
> 
> вт, 11 сент. 2018 г. в 10:51, Денис Костин :
> 
> > Hello everybody!
> > 
> > I found a few bugs in the actual version of Apache Ignite Spark (2.6.0)
> > and described in my GitHub: https://github.com/x-x-z/IgniteSparkBugs
> > 
> > I wrote a test for each case:
> > https://github.com/x-x-z/IgniteSparkBugs/blob/master/src/test/java/org/xxz/ignite_spark_bugs/IgniteSparkBugsApplicationTests.java
> > 
> > Check, please.
> > 
> > 
> > Best regards,
> > Denis Kostin
> > 

signature.asc
Description: This is a digitally signed message part

Re: Class field ThreadLocal. Why not static?

2018-09-11 Thread Maxim Muzafarov

Alexey, Ivan,

Agree. Keeping strong references to the Thread object is the source of
memory leak with ThreadLocals variables
and the values that it stores. ThreadLocalMap is bound to the Thread
lifespan [1], so I think when we are using
everything right all will be GC'ed correctly.
Is this memory leaks with ThreadLocal's you mean, Alexey? If not, please,
share your example.

Also, agree that these usages should be bound to the component lifespan.
But for `FileWriteAheadLogManager`
I think this variable used not semantically right. I've dumped all threads
(total ~49 threads)
that are using `lastWalPtr` in `FileWriteAheadLogManager`. For instance,
 * exchange-worker-#40%wal.IgniteWalRecoveryTest0%
 * sys-#148%wal.IgniteWalRecoveryTest1%
 * db-checkpoint-thread-#129%wal.IgniteWalRecoveryTest2%
Suppose everything would be OK here for `static` and `non-static` case of
ThreadLocal.

[1]
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/tip/src/share/classes/java/lang/Thread.java#l760

On Tue, 11 Sep 2018 at 13:05 Павлухин Иван  wrote:

> Dmitriy,
>
> Could you point to some piece of code implementing described pattern?
>
> 2018-09-11 13:02 GMT+03:00 Павлухин Иван :
>
> > Alex,
> >
> > ThreadLocal subclass is used in IgniteH2Indexing for simple access to H2
> > Connection from current thread. Such subclass has a capability to create
> > connection if one does not exist, so obtaining connection is merely
> > ThreadLocal.get. Also there are scheduled routines to cleanup connections
> > and associated with them statement cache after some expiration time. For
> > that reason Map is maintained. As query can
> > run on user thread we need to cleanup mentioned map to avoid a leak when
> > Thread is terminated. So we need to check thread status in cleanup
> routines
> > and remove entries for terminated Threads. And historically there was no
> > cleanup for terminated threads and leak was possible. And also great care
> > must be taken in order to avoid cyclic reference between ThreadLocal
> > instance and a stored value. Which easily could occur if the stored value
> > is covered by multiple layers of abstraction.
> >
> > And I am describing some historical state. Now machinery in
> IgniteH2Indexing
> > is even more complex (I hope we will have a chance to improve it).
> >
> > 2018-09-11 11:00 GMT+03:00 Alexey Goncharuk  >:
> >
> >> Ivan,
> >>
> >> Can you elaborate on the issue with the thread local cleanup you've
> faced?
> >>
> >> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван :
> >>
> >> > Guys,
> >> >
> >> > As we know ThreadLocal is an instrument which should be used with
> great
> >> > care. And I recently faced with problems related to proper cleanup of
> >> > ThreadLocal which is not needed anymore. In my opinion the best thing
> >> (in
> >> > ideal world) is to get rid of ThreadLocal where possible, but I guess
> >> that
> >> > it is quite hard (in real world).
> >> >
> >> > Also, a question comes to my mind. As ThreadLocal is so common in our
> >> code,
> >> > could you suggest some guidance or code fragments which address proper
> >> > ThreadLocal
> >> >  lifecycle control and especially cleanup?
> >> >
> >> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk <
> alexey.goncha...@gmail.com
> >> >:
> >> >
> >> > > Maxim,
> >> > >
> >> > > Ignite supports starting multiple instances of Ignite in the same
> VM,
> >> so
> >> > > having static thread locals for the fields you mentioned does not
> >> work.
> >> > >
> >> > > Generally, I think thread-local should be bound to the lifespan of
> the
> >> > > component it describes. Static thread-locals are hard to clean-up
> and
> >> > they
> >> > > often lead to leaks, so I would rather changed existing static
> >> > > thread-locals to be non-static.
> >> > >
> >> > > --AG
> >> > >
> >> > > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov :
> >> > >
> >> > > > Igniters,
> >> > > >
> >> > > > According to javadoc [1] class ThreadLocal:
> >> > > > `ThreadLocal instances are typically private *static* fields in
> >> classes
> >> > > > that wish to associate state with a thread (e.g., a user ID or
> >> > > Transaction
> >> > > > ID).`
> >> > > >
> >> > > > So, AFAIK non-static ThreadLocal usage means as `per thread - per
> >> class
> >> > > > instance`. What the real cases of using non-static ThreadLocal
> class
> >> > > fields
> >> > > > in Ignite code project? When we need it?
> >> > > >
> >> > > > In Ignite code project I've found ThreadLocal usage as:
> >> > > >  - non-static - 67
> >> > > >  - static  - 68
> >> > > >
> >> > > > Back to my example, I've checked FileWriteAheadLogManager. It has:
> >> > > > 1) private final ThreadLocal interrupted [2]
> >> > > > 2) private final ThreadLocal lastWALPtr [3]
> >> > > > I think both of these fields should be set and used as `static`.
> Can
> >> > > anyone
> >> > > > confirm it?
> >> > > >
> >> > > >
> >> > > > [1]
> >> > https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html
> >> > > > [2]
> >> > > >
> >> > > > https://githu

Re: Class field ThreadLocal. Why not static?

2018-09-11 Thread Павлухин Иван

Dmitriy,

Could you point to some piece of code implementing described pattern?

2018-09-11 13:02 GMT+03:00 Павлухин Иван :

> Alex,
>
> ThreadLocal subclass is used in IgniteH2Indexing for simple access to H2
> Connection from current thread. Such subclass has a capability to create
> connection if one does not exist, so obtaining connection is merely
> ThreadLocal.get. Also there are scheduled routines to cleanup connections
> and associated with them statement cache after some expiration time. For
> that reason Map is maintained. As query can
> run on user thread we need to cleanup mentioned map to avoid a leak when
> Thread is terminated. So we need to check thread status in cleanup routines
> and remove entries for terminated Threads. And historically there was no
> cleanup for terminated threads and leak was possible. And also great care
> must be taken in order to avoid cyclic reference between ThreadLocal
> instance and a stored value. Which easily could occur if the stored value
> is covered by multiple layers of abstraction.
>
> And I am describing some historical state. Now machinery in IgniteH2Indexing
> is even more complex (I hope we will have a chance to improve it).
>
> 2018-09-11 11:00 GMT+03:00 Alexey Goncharuk :
>
>> Ivan,
>>
>> Can you elaborate on the issue with the thread local cleanup you've faced?
>>
>> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван :
>>
>> > Guys,
>> >
>> > As we know ThreadLocal is an instrument which should be used with great
>> > care. And I recently faced with problems related to proper cleanup of
>> > ThreadLocal which is not needed anymore. In my opinion the best thing
>> (in
>> > ideal world) is to get rid of ThreadLocal where possible, but I guess
>> that
>> > it is quite hard (in real world).
>> >
>> > Also, a question comes to my mind. As ThreadLocal is so common in our
>> code,
>> > could you suggest some guidance or code fragments which address proper
>> > ThreadLocal
>> >  lifecycle control and especially cleanup?
>> >
>> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk > >:
>> >
>> > > Maxim,
>> > >
>> > > Ignite supports starting multiple instances of Ignite in the same VM,
>> so
>> > > having static thread locals for the fields you mentioned does not
>> work.
>> > >
>> > > Generally, I think thread-local should be bound to the lifespan of the
>> > > component it describes. Static thread-locals are hard to clean-up and
>> > they
>> > > often lead to leaks, so I would rather changed existing static
>> > > thread-locals to be non-static.
>> > >
>> > > --AG
>> > >
>> > > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov :
>> > >
>> > > > Igniters,
>> > > >
>> > > > According to javadoc [1] class ThreadLocal:
>> > > > `ThreadLocal instances are typically private *static* fields in
>> classes
>> > > > that wish to associate state with a thread (e.g., a user ID or
>> > > Transaction
>> > > > ID).`
>> > > >
>> > > > So, AFAIK non-static ThreadLocal usage means as `per thread - per
>> class
>> > > > instance`. What the real cases of using non-static ThreadLocal class
>> > > fields
>> > > > in Ignite code project? When we need it?
>> > > >
>> > > > In Ignite code project I've found ThreadLocal usage as:
>> > > >  - non-static - 67
>> > > >  - static  - 68
>> > > >
>> > > > Back to my example, I've checked FileWriteAheadLogManager. It has:
>> > > > 1) private final ThreadLocal interrupted [2]
>> > > > 2) private final ThreadLocal lastWALPtr [3]
>> > > > I think both of these fields should be set and used as `static`. Can
>> > > anyone
>> > > > confirm it?
>> > > >
>> > > >
>> > > > [1]
>> > https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html
>> > > > [2]
>> > > >
>> > > > https://github.com/apache/ignite/blob/master/modules/
>> > > core/src/main/java/org/apache/ignite/internal/processors/
>> > > cache/persistence/wal/FileWriteAheadLogManager.java#L253
>> > > > [3]
>> > > >
>> > > > https://github.com/apache/ignite/blob/master/modules/
>> > > core/src/main/java/org/apache/ignite/internal/processors/
>> > > cache/persistence/wal/FileWriteAheadLogManager.java#L340
>> > > > --
>> > > > --
>> > > > Maxim Muzafarov
>> > > >
>> > >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Ivan Pavlukhin
>> >
>>
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>



-- 
Best regards,
Ivan Pavlukhin

Re: Class field ThreadLocal. Why not static?

2018-09-11 Thread Павлухин Иван

Alex,

ThreadLocal subclass is used in IgniteH2Indexing for simple access to H2
Connection from current thread. Such subclass has a capability to create
connection if one does not exist, so obtaining connection is merely
ThreadLocal.get. Also there are scheduled routines to cleanup connections
and associated with them statement cache after some expiration time. For
that reason Map is maintained. As query can
run on user thread we need to cleanup mentioned map to avoid a leak when
Thread is terminated. So we need to check thread status in cleanup routines
and remove entries for terminated Threads. And historically there was no
cleanup for terminated threads and leak was possible. And also great care
must be taken in order to avoid cyclic reference between ThreadLocal
instance and a stored value. Which easily could occur if the stored value
is covered by multiple layers of abstraction.

And I am describing some historical state. Now machinery in IgniteH2Indexing
is even more complex (I hope we will have a chance to improve it).

2018-09-11 11:00 GMT+03:00 Alexey Goncharuk :

> Ivan,
>
> Can you elaborate on the issue with the thread local cleanup you've faced?
>
> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван :
>
> > Guys,
> >
> > As we know ThreadLocal is an instrument which should be used with great
> > care. And I recently faced with problems related to proper cleanup of
> > ThreadLocal which is not needed anymore. In my opinion the best thing (in
> > ideal world) is to get rid of ThreadLocal where possible, but I guess
> that
> > it is quite hard (in real world).
> >
> > Also, a question comes to my mind. As ThreadLocal is so common in our
> code,
> > could you suggest some guidance or code fragments which address proper
> > ThreadLocal
> >  lifecycle control and especially cleanup?
> >
> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk  >:
> >
> > > Maxim,
> > >
> > > Ignite supports starting multiple instances of Ignite in the same VM,
> so
> > > having static thread locals for the fields you mentioned does not work.
> > >
> > > Generally, I think thread-local should be bound to the lifespan of the
> > > component it describes. Static thread-locals are hard to clean-up and
> > they
> > > often lead to leaks, so I would rather changed existing static
> > > thread-locals to be non-static.
> > >
> > > --AG
> > >
> > > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > > According to javadoc [1] class ThreadLocal:
> > > > `ThreadLocal instances are typically private *static* fields in
> classes
> > > > that wish to associate state with a thread (e.g., a user ID or
> > > Transaction
> > > > ID).`
> > > >
> > > > So, AFAIK non-static ThreadLocal usage means as `per thread - per
> class
> > > > instance`. What the real cases of using non-static ThreadLocal class
> > > fields
> > > > in Ignite code project? When we need it?
> > > >
> > > > In Ignite code project I've found ThreadLocal usage as:
> > > >  - non-static - 67
> > > >  - static  - 68
> > > >
> > > > Back to my example, I've checked FileWriteAheadLogManager. It has:
> > > > 1) private final ThreadLocal interrupted [2]
> > > > 2) private final ThreadLocal lastWALPtr [3]
> > > > I think both of these fields should be set and used as `static`. Can
> > > anyone
> > > > confirm it?
> > > >
> > > >
> > > > [1]
> > https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html
> > > > [2]
> > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/processors/
> > > cache/persistence/wal/FileWriteAheadLogManager.java#L253
> > > > [3]
> > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/processors/
> > > cache/persistence/wal/FileWriteAheadLogManager.java#L340
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>



-- 
Best regards,
Ivan Pavlukhin

[GitHub] SomeFire opened a new pull request #6: IGNITE-9377 Handle print crit failures from RunAll to the JIRA ticket

SomeFire opened a new pull request #6: IGNITE-9377 Handle print crit failures 
from RunAll to the JIRA ticket
URL: https://github.com/apache/ignite-teamcity-bot/pull/6
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

[GitHub] ignite pull request #4717: IGNITE-9518 getPagesFillFactor returns NaN for em...

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4717


---

Re: Class field ThreadLocal. Why not static?

2018-09-11 Thread Dmitriy Pavlov

Hi Ivan,

I can imagine cases with temporary native or heap byte buffers in thread
locals used for IO operations.

These buffers are often Thread Local and I find it to be a perfectly ok
case.

вт, 11 сент. 2018 г. в 11:00, Alexey Goncharuk :

> Ivan,
>
> Can you elaborate on the issue with the thread local cleanup you've faced?
>
> вт, 11 сент. 2018 г. в 9:13, Павлухин Иван :
>
> > Guys,
> >
> > As we know ThreadLocal is an instrument which should be used with great
> > care. And I recently faced with problems related to proper cleanup of
> > ThreadLocal which is not needed anymore. In my opinion the best thing (in
> > ideal world) is to get rid of ThreadLocal where possible, but I guess
> that
> > it is quite hard (in real world).
> >
> > Also, a question comes to my mind. As ThreadLocal is so common in our
> code,
> > could you suggest some guidance or code fragments which address proper
> > ThreadLocal
> >  lifecycle control and especially cleanup?
> >
> > 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk  >:
> >
> > > Maxim,
> > >
> > > Ignite supports starting multiple instances of Ignite in the same VM,
> so
> > > having static thread locals for the fields you mentioned does not work.
> > >
> > > Generally, I think thread-local should be bound to the lifespan of the
> > > component it describes. Static thread-locals are hard to clean-up and
> > they
> > > often lead to leaks, so I would rather changed existing static
> > > thread-locals to be non-static.
> > >
> > > --AG
> > >
> > > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov :
> > >
> > > > Igniters,
> > > >
> > > > According to javadoc [1] class ThreadLocal:
> > > > `ThreadLocal instances are typically private *static* fields in
> classes
> > > > that wish to associate state with a thread (e.g., a user ID or
> > > Transaction
> > > > ID).`
> > > >
> > > > So, AFAIK non-static ThreadLocal usage means as `per thread - per
> class
> > > > instance`. What the real cases of using non-static ThreadLocal class
> > > fields
> > > > in Ignite code project? When we need it?
> > > >
> > > > In Ignite code project I've found ThreadLocal usage as:
> > > >  - non-static - 67
> > > >  - static  - 68
> > > >
> > > > Back to my example, I've checked FileWriteAheadLogManager. It has:
> > > > 1) private final ThreadLocal interrupted [2]
> > > > 2) private final ThreadLocal lastWALPtr [3]
> > > > I think both of these fields should be set and used as `static`. Can
> > > anyone
> > > > confirm it?
> > > >
> > > >
> > > > [1]
> > https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html
> > > > [2]
> > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/processors/
> > > cache/persistence/wal/FileWriteAheadLogManager.java#L253
> > > > [3]
> > > >
> > > > https://github.com/apache/ignite/blob/master/modules/
> > > core/src/main/java/org/apache/ignite/internal/processors/
> > > cache/persistence/wal/FileWriteAheadLogManager.java#L340
> > > > --
> > > > --
> > > > Maxim Muzafarov
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Ivan Pavlukhin
> >
>

Re: IGNITE-7482 Cursor in TextQuery fetches all data in first call to next() or hasNext()

2018-09-11 Thread Dmitriy Pavlov

Hi Tam Manh Nguyen,

I've added you to the list of contributors, so now you can assign an issue
to yourself.

Welcome to the Apache Ignite Community!

Sincerely,
Dmitriy Pavlov

пн, 10 сент. 2018 г. в 23:23, Tâm Nguyễn Mạnh :

> Hi,
>
> I just registered. Here is my jira account: nguyenmanhtam...@gmail.com
>
> Thank you,
> Tamnm
>
> On Mon, Sep 10, 2018 at 3:25 PM Alexey Goncharuk <
> alexey.goncha...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Please send your jira account ID so we can add you to the contributors
> > list. Then you will be able to assign tickets to yourself and contribute
> to
> > the project according to the process.
> >
> > You can get more info here:
> >
> > https://cwiki.apache.org/confluence/display/IGNITE/Development+Process
> > https://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute
> >
> > --AG
> >
> > пн, 10 сент. 2018 г. в 9:16, Tâm Nguyễn Mạnh  >:
> >
> > > Hi,
> > > I have not been assigned yet. But i really want to.
> > >
> > > On Fri, Sep 7, 2018 at 4:13 PM Ilya Kasnacheev <
> > ilya.kasnach...@gmail.com>
> > > wrote:
> > >
> > > > Hello!
> > > >
> > > > Can you please frame it as Github pull request as per our process? Do
> > you
> > > > have ticket for that?
> > > >
> > > > Regards,
> > > > --
> > > > Ilya Kasnacheev
> > > >
> > > >
> > > > пт, 7 сент. 2018 г. в 5:08, Tâm Nguyễn Mạnh <
> > nguyenmanhtam...@gmail.com
> > > >:
> > > >
> > > > >
> > > > >
> > > >
> > >
> >
> modules\indexing\src\main\java\org\apache\ignite\internal\processors\query\h2\opt\GridLuceneIndex.java
> > > > > ```java
> > > > > /*
> > > > >  * Licensed to the Apache Software Foundation (ASF) under one or
> more
> > > > >  * contributor license agreements.  See the NOTICE file distributed
> > > with
> > > > >  * this work for additional information regarding copyright
> > ownership.
> > > > >  * The ASF licenses this file to You under the Apache License,
> > Version
> > > > 2.0
> > > > >  * (the "License"); you may not use this file except in compliance
> > with
> > > > >  * the License.  You may obtain a copy of the License at
> > > > >  *
> > > > >  *  http://www.apache.org/licenses/LICENSE-2.0
> > > > >  *
> > > > >  * Unless required by applicable law or agreed to in writing,
> > software
> > > > >  * distributed under the License is distributed on an "AS IS"
> BASIS,
> > > > >  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> > > > implied.
> > > > >  * See the License for the specific language governing permissions
> > and
> > > > >  * limitations under the License.
> > > > >  */
> > > > >
> > > > > package org.apache.ignite.internal.processors.query.h2.opt;
> > > > >
> > > > > import java.io.IOException;
> > > > > import java.util.Collection;
> > > > > import java.util.concurrent.atomic.AtomicLong;
> > > > > import org.apache.ignite.IgniteCheckedException;
> > > > > import org.apache.ignite.internal.GridKernalContext;
> > > > > import org.apache.ignite.internal.processors.cache.CacheObject;
> > > > > import
> > org.apache.ignite.internal.processors.cache.CacheObjectContext;
> > > > > import
> > > > >
> org.apache.ignite.internal.processors.cache.version.GridCacheVersion;
> > > > > import
> > > > >
> org.apache.ignite.internal.processors.query.GridQueryIndexDescriptor;
> > > > > import
> > > > org.apache.ignite.internal.processors.query.GridQueryTypeDescriptor;
> > > > > import org.apache.ignite.internal.util.GridAtomicLong;
> > > > > import
> org.apache.ignite.internal.util.GridCloseableIteratorAdapter;
> > > > > import org.apache.ignite.internal.util.lang.GridCloseableIterator;
> > > > > import
> > org.apache.ignite.internal.util.offheap.unsafe.GridUnsafeMemory;
> > > > > import org.apache.ignite.internal.util.typedef.internal.U;
> > > > > import org.apache.ignite.lang.IgniteBiTuple;
> > > > > import org.apache.ignite.spi.indexing.IndexingQueryFilter;
> > > > > import org.apache.ignite.spi.indexing.IndexingQueryCacheFilter;
> > > > > import org.apache.lucene.analysis.standard.StandardAnalyzer;
> > > > > import org.apache.lucene.document.Document;
> > > > > import org.apache.lucene.document.Field;
> > > > > import org.apache.lucene.document.LongField;
> > > > > import org.apache.lucene.document.StoredField;
> > > > > import org.apache.lucene.document.StringField;
> > > > > import org.apache.lucene.document.TextField;
> > > > > import org.apache.lucene.index.DirectoryReader;
> > > > > import org.apache.lucene.index.IndexReader;
> > > > > import org.apache.lucene.index.IndexWriter;
> > > > > import org.apache.lucene.index.IndexWriterConfig;
> > > > > import org.apache.lucene.index.Term;
> > > > > import org.apache.lucene.queryparser.classic.MultiFieldQueryParser;
> > > > > import org.apache.lucene.search.BooleanClause;
> > > > > import org.apache.lucene.search.BooleanQuery;
> > > > > import org.apache.lucene.search.IndexSearcher;
> > > > > import org.apache.lucene.search.NumericRangeQuery;
> > > > > import org.apache.lucene.search.Query;
> > > > > import org.apache.l

[jira] [Created] (IGNITE-9537) .NET: Change MVCC enabled configuration.

Roman Kondakov created IGNITE-9537:
--

 Summary: .NET: Change MVCC enabled configuration.
 Key: IGNITE-9537
 URL: https://issues.apache.org/jira/browse/IGNITE-9537
 Project: Ignite
  Issue Type: Task
  Components: mvcc, platforms
Reporter: Roman Kondakov
 Fix For: 2.7


MVCC enabled configuration has changed from the node global flag to per-cache 
atomicity mode setting {{CacheAtomicityMode#TRANSACTIONAL_SNAPSHOT}}. We need 
to reflect this change in .NET API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Ignite Spark Bugs

2018-09-11 Thread Dmitriy Pavlov

Hi Denis,

Thank you for bringing this here and for your efforts to reproduce it.
Would you like to create an issue and contribute these test into Ignite
code base?

If you would like to proceed with the patch submission process, please
sign-in to Apache JIRA and share your JIRA username, I will add you as the
contributor.

You can count on my support.

Sincerely,
Dmitriy Pavlov

вт, 11 сент. 2018 г. в 10:51, Денис Костин :

> Hello everybody!
>
> I found a few bugs in the actual version of Apache Ignite Spark (2.6.0)
> and described in my GitHub: https://github.com/x-x-z/IgniteSparkBugs
>
> I wrote a test for each case:
> https://github.com/x-x-z/IgniteSparkBugs/blob/master/src/test/java/org/xxz/ignite_spark_bugs/IgniteSparkBugsApplicationTests.java
>
> Check, please.
>
>
> Best regards,
> Denis Kostin
>

[jira] [Created] (IGNITE-9536) CPP/ODBC: Change MVCC enabled configuration.

Roman Kondakov created IGNITE-9536:
--

 Summary: CPP/ODBC: Change MVCC enabled configuration.
 Key: IGNITE-9536
 URL: https://issues.apache.org/jira/browse/IGNITE-9536
 Project: Ignite
  Issue Type: Task
  Components: mvcc, odbc
Reporter: Roman Kondakov
 Fix For: 2.7


MVCC enabled configuration has changed from the node global flag to per-cache 
atomicity mode setting {{CacheAtomicityMode#TRANSACTIONAL_SNAPSHOT}}. We need 
to reflect this change in CPP/ODBC API.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4721: IGNITE-9535 ClientChangeGlobalStateComputeRequest...

2018-09-11 Thread ibessonov

GitHub user ibessonov opened a pull request:

https://github.com/apache/ignite/pull/4721

IGNITE-9535 ClientChangeGlobalStateComputeRequest is missing @GridInternal 
annotation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-9535

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4721.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4721


commit 3ef6f4946a2d78de56db77f0b53f85044efd5095
Author: ibessonov 
Date:   2018-09-11T09:02:26Z

IGNITE-9535 Missing annotation added to 
GridClusterStateProcessor.ClientChangeGlobalStateComputeRequest class.




---

[jira] [Created] (IGNITE-9535) ClientChangeGlobalStateComputeRequest is missing @GridInternal annotation

2018-09-11 Thread Ivan Bessonov (JIRA)

Ivan Bessonov created IGNITE-9535:
-

 Summary: ClientChangeGlobalStateComputeRequest is missing 
@GridInternal annotation
 Key: IGNITE-9535
 URL: https://issues.apache.org/jira/browse/IGNITE-9535
 Project: Ignite
  Issue Type: Bug
Reporter: Ivan Bessonov
Assignee: Ivan Bessonov


Problematic class: 
org.apache.ignite.internal.processors.cluster.GridClusterStateProcessor.ClientChangeGlobalStateComputeRequest;

@GridInternal annotation must be added.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4720: IGNITE-9532 Binary mode for Ignite Queue

2018-09-11 Thread udaykale

GitHub user udaykale opened a pull request:

https://github.com/apache/ignite/pull/4720

IGNITE-9532 Binary mode for Ignite Queue



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/udaykale/ignite IGNITE-9532

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4720.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4720


commit abf6cebf989305c4bc5d5cb51f2b8415b34cdd5f
Author: uday 
Date:   2018-09-11T08:40:22Z

IGNITE-9532 Binary mode for Ignite Queue




---

[GitHub] ignite pull request #4704: IGNITE-8650: ZookeeperDiscovery testClientReconne...

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/4704


---

[jira] [Created] (IGNITE-9534) Wrong error message in bin/include/functions.sh

2018-09-11 Thread Ilya Suntsov (JIRA)

Ilya Suntsov created IGNITE-9534:


 Summary: Wrong error message in bin/include/functions.sh
 Key: IGNITE-9534
 URL: https://issues.apache.org/jira/browse/IGNITE-9534
 Project: Ignite
  Issue Type: Bug
  Components: general
Affects Versions: 2.6
Reporter: Ilya Suntsov


In the case when we don't have java installed on the host I see a message:

{noformat}

echo $0", ERROR:"

            echo "JAVA_HOME environment variable is not found."

            echo "Please point JAVA_HOME variable to location of JDK 1.7 or JDK 
1.8."

            echo "You can also download latest JDK at http://java.com/download";

{noformat}

Java versions should be changed to:

{noformat}

 JDK 1.8 or JDK 9

{noformat}

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9533) Monitoring of meta-data discrepancies on all cluster nodes at runtime

2018-09-11 Thread Dmitriy Gladkikh (JIRA)

Dmitriy Gladkikh created IGNITE-9533:


 Summary: Monitoring of meta-data discrepancies on all cluster 
nodes at runtime
 Key: IGNITE-9533
 URL: https://issues.apache.org/jira/browse/IGNITE-9533
 Project: Ignite
  Issue Type: Improvement
Reporter: Dmitriy Gladkikh


It is required to develop a mechanism that allows to determine the 
discrepancies between meta-data (binary_meta/marshaller) on all cluster nodes 
at runtime. This check must be performed periodically. Periodicity should be 
configured.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (IGNITE-9532) Binary mode for Ignite Queue

2018-09-11 Thread Uday Kale (JIRA)

Uday Kale created IGNITE-9532:
-

 Summary: Binary mode for Ignite Queue
 Key: IGNITE-9532
 URL: https://issues.apache.org/jira/browse/IGNITE-9532
 Project: Ignite
  Issue Type: New Feature
  Components: binary, data structures
Reporter: Uday Kale
Assignee: Uday Kale


Add binary mode (withKeepBinary) to Data Structures Queue.

This will allow retrieving values in binary format when needed or when class is 
not available, and will allow implementing the structures in other platforms 
(.NET, C++).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] ignite pull request #4711: IGNITE-6346

2018-09-11 Thread xtern

GitHub user xtern reopened a pull request:

https://github.com/apache/ignite/pull/4711

IGNITE-6346



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xtern/ignite IGNITE-6346

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/4711.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4711


commit 2efaac326aa6b0a4120fa3b1f845db54dd313fce
Author: kaligula...@gmail.com 
Date:   2017-09-26T19:15:04Z

IGNITE-6346 Fix: distributed set does not work in REPLICATED mode

Changes:
1. Fixed bug
2. Added tests

commit 9d24708537ec47baac989d1a747cc716093457b4
Author: pereslegin-pa 
Date:   2018-09-10T09:56:49Z

IGNITE-6346 Code cleanup.




---

[GitHub] ignite pull request #4711: IGNITE-6346

2018-09-11 Thread xtern

Github user xtern closed the pull request at:

https://github.com/apache/ignite/pull/4711


---

[jira] [Created] (IGNITE-9531) ZookeeperDiscovery testClientReconnect is flaky in master

2018-09-11 Thread Alexey Platonov (JIRA)

Alexey Platonov created IGNITE-9531:
---

 Summary: ZookeeperDiscovery testClientReconnect is flaky in master
 Key: IGNITE-9531
 URL: https://issues.apache.org/jira/browse/IGNITE-9531
 Project: Ignite
  Issue Type: Bug
Reporter: Alexey Platonov
Assignee: Alexey Platonov
 Fix For: 2.8


The test IgniteClientReconnectCacheTest#testReconnectMultinode(LongHistory) 
periodically fails with timeouts in master.
>From the logs I see that the hang is caused by one of the two assertion errors:
{code}
java.lang.AssertionError
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.checkClientsStatus(ZookeeperDiscoveryImpl.java:1345)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.access$2300(ZookeeperDiscoveryImpl.java:108)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl$CheckClientsStatusCallback.processResult0(ZookeeperDiscoveryImpl.java:4332)
at 
org.apache.ignite.spi.discovery.zk.internal.ZkAbstractChildrenCallback.processResult(ZkAbstractChildrenCallback.java:42)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient$ChildrenCallbackWrapper.processResult(ZookeeperClient.java:1132)
at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
{code}
or 
{code}
java.lang.AssertionError
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.checkClientsStatus(ZookeeperDiscoveryImpl.java:1388)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl.access$2300(ZookeeperDiscoveryImpl.java:108)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperDiscoveryImpl$CheckClientsStatusCallback.processResult0(ZookeeperDiscoveryImpl.java:4332)
at 
org.apache.ignite.spi.discovery.zk.internal.ZkAbstractChildrenCallback.processResult(ZkAbstractChildrenCallback.java:42)
at 
org.apache.ignite.spi.discovery.zk.internal.ZookeeperClient$ChildrenCallbackWrapper.processResult(ZookeeperClient.java:1132)
at 
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:590)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
{code}

The test failure can be rarely reproduced locally (run repeatedly with CPU 
stress enabled).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Class field ThreadLocal. Why not static?

2018-09-11 Thread Alexey Goncharuk

Ivan,

Can you elaborate on the issue with the thread local cleanup you've faced?

вт, 11 сент. 2018 г. в 9:13, Павлухин Иван :

> Guys,
>
> As we know ThreadLocal is an instrument which should be used with great
> care. And I recently faced with problems related to proper cleanup of
> ThreadLocal which is not needed anymore. In my opinion the best thing (in
> ideal world) is to get rid of ThreadLocal where possible, but I guess that
> it is quite hard (in real world).
>
> Also, a question comes to my mind. As ThreadLocal is so common in our code,
> could you suggest some guidance or code fragments which address proper
> ThreadLocal
>  lifecycle control and especially cleanup?
>
> 2018-09-10 12:46 GMT+03:00 Alexey Goncharuk :
>
> > Maxim,
> >
> > Ignite supports starting multiple instances of Ignite in the same VM, so
> > having static thread locals for the fields you mentioned does not work.
> >
> > Generally, I think thread-local should be bound to the lifespan of the
> > component it describes. Static thread-locals are hard to clean-up and
> they
> > often lead to leaks, so I would rather changed existing static
> > thread-locals to be non-static.
> >
> > --AG
> >
> > пн, 10 сент. 2018 г. в 11:54, Maxim Muzafarov :
> >
> > > Igniters,
> > >
> > > According to javadoc [1] class ThreadLocal:
> > > `ThreadLocal instances are typically private *static* fields in classes
> > > that wish to associate state with a thread (e.g., a user ID or
> > Transaction
> > > ID).`
> > >
> > > So, AFAIK non-static ThreadLocal usage means as `per thread - per class
> > > instance`. What the real cases of using non-static ThreadLocal class
> > fields
> > > in Ignite code project? When we need it?
> > >
> > > In Ignite code project I've found ThreadLocal usage as:
> > >  - non-static - 67
> > >  - static  - 68
> > >
> > > Back to my example, I've checked FileWriteAheadLogManager. It has:
> > > 1) private final ThreadLocal interrupted [2]
> > > 2) private final ThreadLocal lastWALPtr [3]
> > > I think both of these fields should be set and used as `static`. Can
> > anyone
> > > confirm it?
> > >
> > >
> > > [1]
> https://docs.oracle.com/javase/8/docs/api/java/lang/ThreadLocal.html
> > > [2]
> > >
> > > https://github.com/apache/ignite/blob/master/modules/
> > core/src/main/java/org/apache/ignite/internal/processors/
> > cache/persistence/wal/FileWriteAheadLogManager.java#L253
> > > [3]
> > >
> > > https://github.com/apache/ignite/blob/master/modules/
> > core/src/main/java/org/apache/ignite/internal/processors/
> > cache/persistence/wal/FileWriteAheadLogManager.java#L340
> > > --
> > > --
> > > Maxim Muzafarov
> > >
> >
>
>
>
> --
> Best regards,
> Ivan Pavlukhin
>

[jira] [Created] (IGNITE-9530) MVCC TX: Local caches support.