Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

>>
My main concern here is code complexity. Yakov, how difficult it is to
stick a new node in an arbitrary spot of a discovery ring?
>>

Dmitry, I think this is not hard. At least I don't see any issue now.

>>
I think the NodeComparator approach will work. User can chose how to sort
nodes from one rack before nodes from another rack. Same goes for subnets,
or data centers.
>>

Dmitry, can you please explain why you enforce user to write code? This
does not seem convenient to me at all. If user wants to write code then he
can do it for calculating proper arc_id.

Another point I already posted to this thread - this is very error prone.

>>
I am strongly against giving user an opportunity to point exact place in
the ring with somewhat like this interface [int getIdex(Node newNode,
List currentRing)]. This is very error prone and may require tricky
consistency checks just to make sure that implementation of this interface
is consistent along the topology.
With "arcs" approach user can automatically assign proper ids basing on
physical network topology and network routes.
>>

I still think arc_id is better:
1. No code from user side. Only env variable or system property on a
machine.
2. All code inside Ignite - easy to fix and change if required.
3. All benefits of comparator are still available.

Alex, I still don't get how you (and other guys as well) want to deal with
latencies here. I would like you explain how you solve this - you have 1000
IP addresses, and you need to sort them in your beloved latency order, but
please note that you need to get exactly the same ring on all of these 1000
machines.

--Yakov

[jira] [Created] (IGNITE-4496) Review all logging for sensitive data leak

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-4496:
--

 Summary: Review all logging for sensitive data leak
 Key: IGNITE-4496
 URL: https://issues.apache.org/jira/browse/IGNITE-4496
 Project: Ignite
  Issue Type: Improvement
Reporter: Alexandr Kuramshin
Assignee: Alexandr Kuramshin


While sensitive logging option added and toString() methods fixed, not all 
logging was checked for sensitive data leak



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request #1385: IGNITE-4016: SQL: parallelize sql queries over ca...

[AMashenkov]

GitHub user AMashenkov opened a pull request:

https://github.com/apache/ignite/pull/1385

IGNITE-4016: SQL: parallelize sql queries over cache local partitions

Tree index splitted into segmentes in GridH2StripedTreeIndex
Tests added

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4106

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1385.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1385


commit d013500bd897505ec6cbfb2f25d54f1f938f2f83
Author: AMRepo 
Date:   2016-12-26T14:54:01Z

Tree index splitted into segmentes in GridH2StripedTreeIndex

Tests added




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1198: IGNITE-4106: SQL: parallelize sql queries over ca...

[AMashenkov]

Github user AMashenkov closed the pull request at:

https://github.com/apache/ignite/pull/1198


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4494) .NET: Optimize ExamplesTest.TestRemoteNodes

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-4494:
--

 Summary: .NET: Optimize ExamplesTest.TestRemoteNodes
 Key: IGNITE-4494
 URL: https://issues.apache.org/jira/browse/IGNITE-4494
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 1.8
Reporter: Pavel Tupitsyn
Assignee: Pavel Tupitsyn
 Fix For: 2.0


Currently for each example test with remote node we do the following:
* Start monitoring node
* Start a standalone node
* Stop monitoring node
* Start example

But since IGNITE-3427 is implemented, all examples have the same config. We can 
start standalone node(s) once and run all examples, saving a lot of time 
(currently example tests are the most time consuming).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Александр Меньшиков]

> Can you please explain why this is better than arc approach?

We had a misunderstanding. Everything okay with arc approach. But we must
choose how nodes will determine "ARC_ID", and i think it can be calculated
from latency values. If users will be able to set "ARC_ID" in config file
then they can set different 'ARC_ID' on all nodes, and, in fact, point
exact place in the ring, what we would like to avoid.

2016-12-26 15:36 GMT+03:00 Vyacheslav Daradur :

> >>
> Vyacheslav, I agree that latency increase in the way you describe, but I
> still don't understand how we use this information in discovery. Latency
> may differ from time to time depending on many factors. I still think that
> arc approach is more intuitive for user and easier to implement.
> >>
>
> Way of latency increase is just a main idea.
>
> I suggest to connect new node on some priority.
> General approach:
> --
> if [ there are same host node ] then [ connect with it ]
> else if [ there are same subnet nodes] then [ connect with one of them ]
>  // how to choose node from a set of subnet? - choose with min latency each
> other
> else [ connect to remote nodes ] // how to choose node from a set of
> remotes? - choose with min latency each other
> --
> Maybe we can describe another intermediate steps.
>
>
> 2016-12-26 15:08 GMT+03:00 Yakov Zhdanov :
>
> > >>
> > I just want to understand which benefits we get when implement what we're
> > talking about. If major benefit is reduced latency of ring messages, then
> > the assignment 'ARC ID' in accordance with latency value is quite
> > enough. But if there are any hidden problems because of the large number
> of
> > reconnection (like I described in first message in this discussion), then
> > better to find a way to determine real physical location.
> > >>
> >
> > I suggest to solve ring building up and reducing number of reconnects
> > separately. If we have AxB-C-D-A then A will try to reconnect to B, then
> to
> > C, then to D. This is how discovery works now. I agree this should be
> fixed
> > and I have couple ideas on how we can do it but let's separate these
> ones.
> >
> > >>
> > Okey, then i think Vyacheslav's idea (using latency values) is quite
> enough
> > when we can't determine real physical location.
> > >>
> >
> > Can you please explain why this is better than arc approach?
> >
> > --Yakov
> >
>

[jira] [Created] (IGNITE-4493) ODBC: Add missing diagnostic records in case of API errors

[Sergey Kalashnikov (JIRA)]

Sergey Kalashnikov created IGNITE-4493:
--

 Summary: ODBC: Add missing diagnostic records in case of API errors
 Key: IGNITE-4493
 URL: https://issues.apache.org/jira/browse/IGNITE-4493
 Project: Ignite
  Issue Type: Bug
  Components: odbc
Affects Versions: 1.8
Reporter: Sergey Kalashnikov
Assignee: Sergey Kalashnikov
 Fix For: 2.0


The following functions in Ignite ODBC driver do not strictly follow the API 
documentation with regard to diagnostics. It should be possible to obtain 
additional information about errors via SQLGetDiagRec() call.

SQLAllocHandle
SQLFreeHandle
SQLFreeStmt
SQLBindCol
SQLFetchScroll
SQLBindParameter




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (IGNITE-4492) Add MBean for StripedExecutor

[Yakov Zhdanov (JIRA)]

Yakov Zhdanov created IGNITE-4492:
-

 Summary: Add MBean for StripedExecutor
 Key: IGNITE-4492
 URL: https://issues.apache.org/jira/browse/IGNITE-4492
 Project: Ignite
  Issue Type: Improvement
Affects Versions: 1.9
Reporter: Yakov Zhdanov
Priority: Minor
 Fix For: 2.0


# Add MBean interface and implementation as we have for 
org.apache.ignite.internal.ThreadPoolMXBeanAdapter
# Register MBean together with other pools MBeans
# Unregister MBean on Ignite stop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] ignite pull request #1384: IGNITE-4470 Added support for log file configurat...

[skalashnikov]

GitHub user skalashnikov opened a pull request:

https://github.com/apache/ignite/pull/1384

IGNITE-4470 Added support for log file configuration via environmental 
variable IGNITE_ODBC_LOG_PATH



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4470

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1384.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1384


commit 16d133630e0f2089e29c9b6df5a37e7bf73b90ab
Author: skalashnikov 
Date:   2016-12-22T10:11:37Z

IGNITE-4470 added support for log file configuration via environment 
variable IGNITE_ODBC_LOG_PATH

commit 7a0945d6f3a2b553eed5fd42e4fb8e1abb8382f2
Author: skalashnikov 
Date:   2016-12-26T12:27:55Z

Merge branch 'master' of https://github.com/apache/ignite into ignite-4470




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Vyacheslav Daradur]

>>
Vyacheslav, I agree that latency increase in the way you describe, but I
still don't understand how we use this information in discovery. Latency
may differ from time to time depending on many factors. I still think that
arc approach is more intuitive for user and easier to implement.
>>

Way of latency increase is just a main idea.

I suggest to connect new node on some priority.
General approach:
--
if [ there are same host node ] then [ connect with it ]
else if [ there are same subnet nodes] then [ connect with one of them ]
 // how to choose node from a set of subnet? - choose with min latency each
other
else [ connect to remote nodes ] // how to choose node from a set of
remotes? - choose with min latency each other
--
Maybe we can describe another intermediate steps.


2016-12-26 15:08 GMT+03:00 Yakov Zhdanov :

> >>
> I just want to understand which benefits we get when implement what we're
> talking about. If major benefit is reduced latency of ring messages, then
> the assignment 'ARC ID' in accordance with latency value is quite
> enough. But if there are any hidden problems because of the large number of
> reconnection (like I described in first message in this discussion), then
> better to find a way to determine real physical location.
> >>
>
> I suggest to solve ring building up and reducing number of reconnects
> separately. If we have AxB-C-D-A then A will try to reconnect to B, then to
> C, then to D. This is how discovery works now. I agree this should be fixed
> and I have couple ideas on how we can do it but let's separate these ones.
>
> >>
> Okey, then i think Vyacheslav's idea (using latency values) is quite enough
> when we can't determine real physical location.
> >>
>
> Can you please explain why this is better than arc approach?
>
> --Yakov
>

Re: Make async API great again

[Yakov Zhdanov]

me too.

--Yakov

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

>>
I just want to understand which benefits we get when implement what we're
talking about. If major benefit is reduced latency of ring messages, then
the assignment 'ARC ID' in accordance with latency value is quite
enough. But if there are any hidden problems because of the large number of
reconnection (like I described in first message in this discussion), then
better to find a way to determine real physical location.
>>

I suggest to solve ring building up and reducing number of reconnects
separately. If we have AxB-C-D-A then A will try to reconnect to B, then to
C, then to D. This is how discovery works now. I agree this should be fixed
and I have couple ideas on how we can do it but let's separate these ones.

>>
Okey, then i think Vyacheslav's idea (using latency values) is quite enough
when we can't determine real physical location.
>>

Can you please explain why this is better than arc approach?

--Yakov

[GitHub] ignite pull request #1380: IGNITE-4486 Added serialVersionUID to AttributeNo...

[alexpaschenko]

Github user alexpaschenko closed the pull request at:

https://github.com/apache/ignite/pull/1380


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] ignite pull request #1383: IGNITE-3875

[devozerov]

GitHub user devozerov opened a pull request:

https://github.com/apache/ignite/pull/1383

IGNITE-3875



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-3875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1383.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1383


commit 9b25a643ac97b2a832179d669397d552d2a3
Author: AMRepo 
Date:   2016-10-19T16:05:19Z

Implemented

commit 0862db26623db5bde0c744f4b3536a10fd7b3e52
Author: AMRepo 
Date:   2016-10-19T16:05:19Z

Implemented

commit f9304121945f07b3785d07518e435bc964233f2f
Author: AMRepo 
Date:   2016-10-19T18:46:33Z

Minors

commit c17ff0e716cdbaf759e0f34bbbf7834b87198d0a
Author: AMRepo 
Date:   2016-10-19T18:55:50Z

Merge remote-tracking branch 'origin/ignite-3875' into ignite-3875

# Conflicts:
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/datastreamer/DataStreamProcessor.java

commit c8f78c3e4a69fb001c8995bb1f9c204da7415403
Author: Andrey V. Mashenkov 
Date:   2016-10-20T10:12:21Z

Merge branch 'ignite-1.6.11' into ignite-3875

commit 380a0b8a01a2ae4c606dbf67b588896ca7be435b
Author: vozerov-gridgain 
Date:   2016-10-20T11:58:55Z

Fixed receiver serialization problem.

commit 0f927608fe7b10f6c79b24b87b5eeecd46f68717
Author: vozerov-gridgain 
Date:   2016-10-20T11:59:15Z

Merge remote-tracking branch 'upstream/ignite-3875' into ignite-3875

commit 448d6f594dfb484aa471c186fec80bb0e54949d6
Author: Andrey V. Mashenkov 
Date:   2016-10-20T12:05:37Z

Fixed receiver serialization problem.

commit c893da70a9757b16b0799adc8eaa29fa1b03d06e
Author: tledkov-gridgain 
Date:   2016-12-21T11:54:33Z

IGNITE-4399: IGFS: Merged IgfsSecondaryFileSystem and 
IgfsSecondaryFileSystemV2 interfaces. This closes #1346.

commit c5882a85f4e3a1f61723ac54fd92f087684df6da
Author: devozerov 
Date:   2016-12-26T11:15:42Z

Merge branch 'master' into ignite-2.0

commit 248eaa710b11982831f71ed6733d125a5bae6871
Author: devozerov 
Date:   2016-12-26T11:21:50Z

Merge branch 'ignite-2.0' into ignite-3875

# Conflicts:
#   modules/core/src/main/java/org/apache/ignite/internal/IgnitionEx.java
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/communication/GridIoManager.java
#   
modules/core/src/main/java/org/apache/ignite/internal/managers/communication/GridIoPolicy.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/datastreamer/DataStreamProcessor.java
#   
modules/core/src/main/java/org/apache/ignite/internal/processors/datastreamer/DataStreamerImpl.java
#   
modules/core/src/test/java/org/apache/ignite/testframework/junits/GridTestKernalContext.java

commit 1d0a10c8a48de51e122821e2c3b47b0fff83ac7f
Author: devozerov 
Date:   2016-12-26T11:23:41Z

Fixes after merge.

commit 1883359d5ddc8deaddebb35b30e135ad18333070
Author: devozerov 
Date:   2016-12-26T11:23:51Z

Fixes after merge.

commit 13e514e99a715ce8a276c7fd1aa12f384a727015
Author: devozerov 
Date:   2016-12-26T11:31:45Z

Fixes after review.

commit 34aef7c17a59b9fdcc3df70382cb84dca1577a4b
Author: devozerov 
Date:   2016-12-26T11:39:58Z

Minors.

commit b39ae48535ed5ca1ec0bab7b8d64d6b1fe5d3715
Author: devozerov 
Date:   2016-12-26T11:42:07Z

Minors.

commit 87f103f92aeda9939e536b878f281b57aa1c31f2
Author: devozerov 
Date:   2016-12-26T11:44:02Z

Minors.

commit af5e0482d767bd79fa152c486d991628d928827c
Author: devozerov 
Date:   2016-12-26T11:47:26Z

Minors.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (IGNITE-4491) Commutation loss between two nodes leads to hang whole cluster.

[Vladislav Pyatkov (JIRA)]

Vladislav Pyatkov created IGNITE-4491:
-

 Summary: Commutation loss between two nodes leads to hang whole 
cluster.
 Key: IGNITE-4491
 URL: https://issues.apache.org/jira/browse/IGNITE-4491
 Project: Ignite
  Issue Type: Bug
Affects Versions: 1.8
Reporter: Vladislav Pyatkov
Priority: Critical


Reproduction steps:
1) Start nodes:

DC1   DC2

1 (10.116.172.1)  8 (10.116.64.11)
2 (10.116.172.2)  7 (10.116.64.12)
3 (10.116.172.3)  6 (10.116.64.13)
4 (10.116.172.4)  5 (10.116.64.14)

each node have client which run in same host with server (look source in 
attachment).

2) Drop connection

Between 1-8,

1 (10.116.172.1)  8 (10.116.64.11)

Drop all input and output traffic
Invoke from 10.116.172.1
iptables -A INPUT -s 10.116.64.11 -j DROP
iptables -A OUTPUT -d 10.116.64.11 -j DROP

Between  4-5

4 (10.116.172.4)  5 (10.116.64.14)

Invoke from 10.116.172.4
iptables -A INPUT -s 10.116.64.14 -j DROP
iptables -A OUTPUT -d 10.116.64.14 -j DROP

3) Stop the grid, after several seconds

If you are looking into logs, you can find which node was segmented (pay 
attention, which clients did not segmented.), after drop traffic:
[12:04:33,914][INFO][disco-event-worker-#211%null%][GridDiscoveryManager] 
Topology snapshot [ver=18, servers=6, clients=8, CPUs=456, heap=68.0GB]

And all operations stopped at the same time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Александр Меньшиков]

> I am afraid I did not understand this at all. Please elaborate.

I just want to understand which benefits we get when implement what we're
talking about. If major benefit is reduced latency of ring messages, then
the assignment 'ARC ID' in accordance with latency value is quite
enough. But if there are any hidden problems because of the large number of
reconnection (like I described in first message in this discussion), then
better to find a way to determine real physical location.

> And, yes, proper built ring should reduce latency of ring messages IMO.

Okey, then i think Vyacheslav's idea (using latency values) is quite enough
when we can't determine real physical location.

2016-12-26 13:03 GMT+03:00 Yakov Zhdanov :

> >Then, as I understand it, a lot of reconnection in the ring cannot create
> even temporary but major problems for performance. And in general this
> optimization will change practically nothing. Or am I missing some things?
>
> I am afraid I did not understand this at all. Please elaborate.
>
> I did not suggest any reconnections or ring rebuild. All I suggest is to
> control over ring building process with arcs. And, yes, proper built ring
> should reduce latency of ring messages IMO.
>
> --Yakov
>

IGNITE-4487 - NPE on query execution

[Александр Меньшиков]

Hello everyone.

I want to pick up *https://issues.apache.org/jira/browse/IGNITE-4487
* as my
first issue.

Please add me as contributor.

I already found that: in inner class
'GridCacheQueryAdapter.ScanQueryFallbackClosableIterator' in constructor is
called with method 'init()', but method 'init()' cannot be called with an
empty field 'nodes'. In source code it looks like:

private ScanQueryFallbackClosableIterator(int part, GridCacheQueryAdapter
qry,
GridCacheQueryManager qryMgr, GridCacheContext cctx) {
this.qry = qry;
this.qryMgr = qryMgr;
this.cctx = cctx;
this.part = part;

nodes = fallbacks(cctx.discovery().topologyVersionEx());
// !!! Here nodes.isEmpty()==true, and init() will fail in the
future. !!!
init();
}

I can fix it by adding some check in code, but i must know what behavior
are best in this case? As I understand it, the list of nodes is empty if
there are no nodes with the current partition, which means data loss, and
either need to return a meaningful exception, or ignore this situation. But
maybe I missed something.

Re: Make async API great again

[Vladimir Ozerov]

Answered.

On Mon, Dec 26, 2016 at 12:51 PM, Yakov Zhdanov  wrote:

> Vladimir, I commented in https://issues.apache.org/jira/browse/IGNITE-4480
> and https://issues.apache.org/jira/browse/IGNITE-4479 and
> https://issues.apache.org/jira/browse/IGNITE-4476
>
> I agree that current approach for async API is not good at all and needs to
> be fixed.
>
> As far as callback semantics, it does not seem to be a solution since user
> may not pass executor parameter and will get the same starvation again. We
> just need to add proper documentation and maybe implement smarter
> starvation checker similar to one in striped pool.
>
>
>
> --Yakov
>
> 2016-12-22 14:09 GMT+03:00 Vladimir Ozerov :
>
> > And several more improvements to future API:
> > 1) Remove startTime() and duration() methods:
> > https://issues.apache.org/jira/browse/IGNITE-4479
> > 2) Fix broken cancellation semantics:
> > https://issues.apache.org/jira/browse/IGNITE-4480
> >
> > On Thu, Dec 22, 2016 at 1:40 PM, Vladimir Ozerov 
> > wrote:
> >
> > > Gents,
> > >
> > > I created tickets for all proposed improvements:
> > > 1) Nice async API: https://issues.apache.org/jira/browse/IGNITE-4475
> > > 2) Do not process IO messages synchronously for local node:
> > > https://issues.apache.org/jira/browse/IGNITE-4476
> > > 3) Better IgniteFuture API and callback semantics: https://issues.
> > > apache.org/jira/browse/IGNITE-4477
> > >
> > > Please review it and let me know if you have any comments.
> > >
> > > Vladimir.
> > >
> > > On Wed, Dec 21, 2016 at 4:32 AM, Dmitriy Setrakyan <
> > dsetrak...@apache.org>
> > > wrote:
> > >
> > >> Would be nice if someone would prototype a new cache API and post the
> > >> generated javadoc here. I think we all will benefit from reviewing it.
> > >>
> > >> On Tue, Dec 20, 2016 at 12:17 PM, Vladimir Ozerov <
> voze...@gridgain.com
> > >
> > >> wrote:
> > >>
> > >> > Async API rework is mechanical addition of ~100 methods through
> > >> copy-paste.
> > >> > Should not take more than a day to implement and more than another
> day
> > >> to
> > >> > rework tests.
> > >> >
> > >> > On Tue, Dec 20, 2016 at 10:00 PM, Dmitriy Setrakyan <
> > >> dsetrak...@apache.org
> > >> > >
> > >> > wrote:
> > >> >
> > >> > > How difficult is this change? Does not look like it can be done
> > >> > overnight.
> > >> > >
> > >> > > On Tue, Dec 20, 2016 at 10:46 AM, Vladimir Ozerov <
> > >> voze...@gridgain.com>
> > >> > > wrote:
> > >> > >
> > >> > > > We already discussed this several months ago in other thread.
> > >> > > >
> > >> > > > "Async" methods is the most simple and straight API possible.
> .NET
> > >> > world
> > >> > > > goes this way all over their frameworks and nobody died.
> Hazelcast
> > >> also
> > >> > > > goes this way. Java goes this way (see CompletableFuture). This
> is
> > >> > common
> > >> > > > and well-known practice. The most impacted part of our API will
> be
> > >> > cache,
> > >> > > > +33 new methods. Though, I do not see how it can affect learning
> > >> curve.
> > >> > > >
> > >> > > > Agree that we should deprecate AsyncSupport gradually and remove
> > it
> > >> no
> > >> > > > earlier than in Apache Ignite 3.0.
> > >> > > >
> > >> > > > On Tue, Dec 20, 2016 at 9:31 PM, Dmitriy Setrakyan <
> > >> > > dsetrak...@apache.org>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > On Tue, Dec 20, 2016 at 10:28 AM, Sergi Vladykin <
> > >> > > > sergi.vlady...@gmail.com
> > >> > > > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > +1 For removing withAsync. It is a broken design.
> > >> > > > > >
> > >> > > > >
> > >> > > > > Sergi, do you also want to add all the async methods to the
> main
> > >> API
> > >> > or
> > >> > > > do
> > >> > > > > you have some other design in mind?
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: IGNITE-4464 - JclLogger ignores IGNITE_QUIET system property

[Vyacheslav Daradur]

Vladimir,

Thanks, I will choose another issue.

2016-12-26 13:07 GMT+03:00 Vladimir Ozerov :

> Hi Vyacheslav,
>
> I believe this issue was already resolved as a part of IGNITE-4248 [1].
> I'll close it as duplicate. Please choose another issue. I'll added you to
> contributor's list.
>
> [1] https://issues.apache.org/jira/browse/IGNITE-4248
>
> Vladimir.
>
> On Mon, Dec 26, 2016 at 12:45 PM, Vyacheslav Daradur 
> wrote:
>
> > Hello everyone.
> >
> > I want to pick up https://issues.apache.org/jira/browse/IGNITE-4464 as
> my
> > first issue.
> >
> > Please add me as contributor.
>

Re: Striped pool configuration

[Yakov Zhdanov]

Dmitry, for example, running queries from old fashioned pool seems to
perform better, but short-term cache updates perform better when run on
striped pool.

--Yakov

Re: IGNITE-4464 - JclLogger ignores IGNITE_QUIET system property

[Vladimir Ozerov]

Hi Vyacheslav,

I believe this issue was already resolved as a part of IGNITE-4248 [1].
I'll close it as duplicate. Please choose another issue. I'll added you to
contributor's list.

[1] https://issues.apache.org/jira/browse/IGNITE-4248

Vladimir.

On Mon, Dec 26, 2016 at 12:45 PM, Vyacheslav Daradur 
wrote:

> Hello everyone.
>
> I want to pick up https://issues.apache.org/jira/browse/IGNITE-4464 as my
> first issue.
>
> Please add me as contributor.

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

>Then, as I understand it, a lot of reconnection in the ring cannot create
even temporary but major problems for performance. And in general this
optimization will change practically nothing. Or am I missing some things?

I am afraid I did not understand this at all. Please elaborate.

I did not suggest any reconnections or ring rebuild. All I suggest is to
control over ring building process with arcs. And, yes, proper built ring
should reduce latency of ring messages IMO.

--Yakov

Re: Make async API great again

[Yakov Zhdanov]

Vladimir, I commented in https://issues.apache.org/jira/browse/IGNITE-4480
and https://issues.apache.org/jira/browse/IGNITE-4479 and
https://issues.apache.org/jira/browse/IGNITE-4476

I agree that current approach for async API is not good at all and needs to
be fixed.

As far as callback semantics, it does not seem to be a solution since user
may not pass executor parameter and will get the same starvation again. We
just need to add proper documentation and maybe implement smarter
starvation checker similar to one in striped pool.



--Yakov

2016-12-22 14:09 GMT+03:00 Vladimir Ozerov :

> And several more improvements to future API:
> 1) Remove startTime() and duration() methods:
> https://issues.apache.org/jira/browse/IGNITE-4479
> 2) Fix broken cancellation semantics:
> https://issues.apache.org/jira/browse/IGNITE-4480
>
> On Thu, Dec 22, 2016 at 1:40 PM, Vladimir Ozerov 
> wrote:
>
> > Gents,
> >
> > I created tickets for all proposed improvements:
> > 1) Nice async API: https://issues.apache.org/jira/browse/IGNITE-4475
> > 2) Do not process IO messages synchronously for local node:
> > https://issues.apache.org/jira/browse/IGNITE-4476
> > 3) Better IgniteFuture API and callback semantics: https://issues.
> > apache.org/jira/browse/IGNITE-4477
> >
> > Please review it and let me know if you have any comments.
> >
> > Vladimir.
> >
> > On Wed, Dec 21, 2016 at 4:32 AM, Dmitriy Setrakyan <
> dsetrak...@apache.org>
> > wrote:
> >
> >> Would be nice if someone would prototype a new cache API and post the
> >> generated javadoc here. I think we all will benefit from reviewing it.
> >>
> >> On Tue, Dec 20, 2016 at 12:17 PM, Vladimir Ozerov  >
> >> wrote:
> >>
> >> > Async API rework is mechanical addition of ~100 methods through
> >> copy-paste.
> >> > Should not take more than a day to implement and more than another day
> >> to
> >> > rework tests.
> >> >
> >> > On Tue, Dec 20, 2016 at 10:00 PM, Dmitriy Setrakyan <
> >> dsetrak...@apache.org
> >> > >
> >> > wrote:
> >> >
> >> > > How difficult is this change? Does not look like it can be done
> >> > overnight.
> >> > >
> >> > > On Tue, Dec 20, 2016 at 10:46 AM, Vladimir Ozerov <
> >> voze...@gridgain.com>
> >> > > wrote:
> >> > >
> >> > > > We already discussed this several months ago in other thread.
> >> > > >
> >> > > > "Async" methods is the most simple and straight API possible. .NET
> >> > world
> >> > > > goes this way all over their frameworks and nobody died. Hazelcast
> >> also
> >> > > > goes this way. Java goes this way (see CompletableFuture). This is
> >> > common
> >> > > > and well-known practice. The most impacted part of our API will be
> >> > cache,
> >> > > > +33 new methods. Though, I do not see how it can affect learning
> >> curve.
> >> > > >
> >> > > > Agree that we should deprecate AsyncSupport gradually and remove
> it
> >> no
> >> > > > earlier than in Apache Ignite 3.0.
> >> > > >
> >> > > > On Tue, Dec 20, 2016 at 9:31 PM, Dmitriy Setrakyan <
> >> > > dsetrak...@apache.org>
> >> > > > wrote:
> >> > > >
> >> > > > > On Tue, Dec 20, 2016 at 10:28 AM, Sergi Vladykin <
> >> > > > sergi.vlady...@gmail.com
> >> > > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > +1 For removing withAsync. It is a broken design.
> >> > > > > >
> >> > > > >
> >> > > > > Sergi, do you also want to add all the async methods to the main
> >> API
> >> > or
> >> > > > do
> >> > > > > you have some other design in mind?
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

IGNITE-4464 - JclLogger ignores IGNITE_QUIET system property

[Vyacheslav Daradur]

Hello everyone.

I want to pick up https://issues.apache.org/jira/browse/IGNITE-4464 as my
first issue.

Please add me as contributor.

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Александр Меньшиков]

Thank you, Denis, for your explanation. Then, as I understand it, a lot of
reconnection in the ring cannot create even temporary but major problems
for performance. And in general this optimization will change practically
nothing. Or am I missing some things?

2016-12-26 12:20 GMT+03:00 Yakov Zhdanov :

> >>
> For example, ordering on latency:
> - nodes on one host = 1
> - nodes in one rack-blade = 2
> - nodes in one server-rack = 3
> - nodes in one physical cluster = 4
> - nodes in one subnet = 5
> - etc.
>
> Maybe it'll be better to use some metrics from ClusterMetrics interface.
>
> The algorithm of ordering can be implemented in a class such as Comparator
> and use it when we build a cluster or we select a place for a new node.
> >>
>
> Vyacheslav, please elaborate on how we can determine whether we are on the
> same rack. I am not sure this is possible in general case. Please see my
> suggestions below.
>
> >>
> However, here is the concern I have. Currently when a new node joins,
> coordinator assigns order number to this node (e.g. if we already have
> nodes 1,2 and 3, new node will have order 4). This node will then be the
> last one on the ring, i.e. nodes are always ordered in the ring by this
> order number (1->2->3->4->1). If we change this, we will basically allow a
> node to be placed anywhere else (smth like 1->2->4->3->1). I'm not 100%
> sure if this is going to cause issues, but sounds dangerous.
>
> Yakov, can you please chime in and share your thoughts on this?
> >>
>
> I don't think this may cause issues. Nodes ordering and placement is
> implemented in TcpDiscoveryNodesRing and I think that we will just need to
> alter org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing#
> nextNode(java.util.Collection discovery.tcp.internal.TcpDiscoveryNode>)
> logic.
>
> As far as design of this, I would suggest the following.
>
> 1.  User should have an ability to define ARC_ID for the node. I suggest
> "arc" for this since we are using "ring" concept. This will be the most
> honored characteristic for nodes placement. By default arc_id is 0 and
> possible to set with system property IGNITE_DISCO_ARC_ID or env variable or
> via TcpDiscoverySpi.setArcId() - new method.
> So, if I have nodes A, D, G with arc_id set to 1 and B, Z with arc_id set
> to 5 then ring should be built as follows: A->D->G->B->Z->A. Here arcs can
> represent different racks or data centers.
>
> I am strongly against giving user an opportunity to point exact place in
> the ring with somewhat like this interface [int getIdex(Node newNode,
> List currentRing)]. This is very error prone and may require tricky
> consistency checks just to make sure that implementation of this interface
> is consistent along the topology.
> With "arcs" approach user can automatically assign proper ids basing on
> physical network topology and network routes.
>
> 2. Subnet - 2nd honored parameter. Nodes on the same subnet should be
> placed side by side in the same arc.
>
> 3. Physical host - 3rd honored parameter. Nodes on the same physical host
> should be placed together automatically in the same arc.
>
> 4. New mode involving points 1-3 should become default and we should also
> provide ability to switch to current mode which should become legacy.
>
> --Yakov
>

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Vyacheslav Daradur]

>>
Vyacheslav, please elaborate on how we can determine whether we are on the
same rack. I am not sure this is possible in general case. Please see my
suggestions below.
>>

I thought of latency values.

Latency between host nodes < Latency between same rack nodes < Latency
between subnet nodes < etc.


2016-12-26 12:20 GMT+03:00 Yakov Zhdanov :

> >>
> For example, ordering on latency:
> - nodes on one host = 1
> - nodes in one rack-blade = 2
> - nodes in one server-rack = 3
> - nodes in one physical cluster = 4
> - nodes in one subnet = 5
> - etc.
>
> Maybe it'll be better to use some metrics from ClusterMetrics interface.
>
> The algorithm of ordering can be implemented in a class such as Comparator
> and use it when we build a cluster or we select a place for a new node.
> >>
>
> Vyacheslav, please elaborate on how we can determine whether we are on the
> same rack. I am not sure this is possible in general case. Please see my
> suggestions below.
>
> >>
> However, here is the concern I have. Currently when a new node joins,
> coordinator assigns order number to this node (e.g. if we already have
> nodes 1,2 and 3, new node will have order 4). This node will then be the
> last one on the ring, i.e. nodes are always ordered in the ring by this
> order number (1->2->3->4->1). If we change this, we will basically allow a
> node to be placed anywhere else (smth like 1->2->4->3->1). I'm not 100%
> sure if this is going to cause issues, but sounds dangerous.
>
> Yakov, can you please chime in and share your thoughts on this?
> >>
>
> I don't think this may cause issues. Nodes ordering and placement is
> implemented in TcpDiscoveryNodesRing and I think that we will just need to
> alter org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing#
> nextNode(java.util.Collection discovery.tcp.internal.TcpDiscoveryNode>)
> logic.
>
> As far as design of this, I would suggest the following.
>
> 1.  User should have an ability to define ARC_ID for the node. I suggest
> "arc" for this since we are using "ring" concept. This will be the most
> honored characteristic for nodes placement. By default arc_id is 0 and
> possible to set with system property IGNITE_DISCO_ARC_ID or env variable or
> via TcpDiscoverySpi.setArcId() - new method.
> So, if I have nodes A, D, G with arc_id set to 1 and B, Z with arc_id set
> to 5 then ring should be built as follows: A->D->G->B->Z->A. Here arcs can
> represent different racks or data centers.
>
> I am strongly against giving user an opportunity to point exact place in
> the ring with somewhat like this interface [int getIdex(Node newNode,
> List currentRing)]. This is very error prone and may require tricky
> consistency checks just to make sure that implementation of this interface
> is consistent along the topology.
> With "arcs" approach user can automatically assign proper ids basing on
> physical network topology and network routes.
>
> 2. Subnet - 2nd honored parameter. Nodes on the same subnet should be
> placed side by side in the same arc.
>
> 3. Physical host - 3rd honored parameter. Nodes on the same physical host
> should be placed together automatically in the same arc.
>
> 4. New mode involving points 1-3 should become default and we should also
> provide ability to switch to current mode which should become legacy.
>
> --Yakov
>

[GitHub] ignite pull request #1382: Ignite 4147 SSL misconfiguration is not handled p...

[dkarachentsev]

GitHub user dkarachentsev opened a pull request:

https://github.com/apache/ignite/pull/1382

Ignite 4147 SSL misconfiguration is not handled properly



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-4147

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/1382.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1382


commit d643dcf2dd2caac4c3ff04cb72f31bbfbf97339a
Author: Pavel Tupitsyn 
Date:   2016-09-28T11:34:23Z

IGNITE-3970 .NET: Fix Cyrillic 'C' letters in code - add test

commit 474ade276c4ae3e8f93cce37473d37270b4e7ad9
Author: vozerov-gridgain 
Date:   2016-09-28T11:38:04Z

IGNITE-3988: Moved failing cloud tests to ignore module.

commit ccfaaf8d060ef984678d2376b16b5a17e7c17e9d
Author: vozerov-gridgain 
Date:   2016-09-28T11:38:17Z

Merge remote-tracking branch 'upstream/ignite-1.6.9' into ignite-1.6.9

commit c7fa918c10d771efa91cde1017662c26d0a61085
Author: Pavel Tupitsyn 
Date:   2016-09-28T11:47:17Z

Merge remote-tracking branch 'remotes/community/ignite-1.6.9' into 
ignite-1.7.3

commit b9105df77cc70b532f94899c754fba47e3e05e9a
Author: vozerov-gridgain 
Date:   2016-09-28T11:53:20Z

IGNITE-3989: Moved failing JTA tests to ignore module.

commit d595345765db2151ff432beecd478ce056393593
Author: vozerov-gridgain 
Date:   2016-09-28T12:08:38Z

IGNITE-3990: Moved failing Spring tests to "ignore" module.

commit e3f13455d4273e615727d0410783e3719db98f76
Author: sboikov 
Date:   2016-09-28T09:56:17Z

ignite-2833 Need call 'touch' for cache entry if it was obtained using 
'entryEx'.

(cherry picked from commit 17c2fc0)

commit b2faa339acb2eea24e6dd5e0c21fc3d3d0592ff6
Author: sboikov 
Date:   2016-09-28T10:47:51Z

ignite-3621 Fixed 'testEvictExpired'.

(cherry picked from commit 4ff19c2)

commit 74d2fc2416b8e6bc0598152552021f984a013061
Author: sboikov 
Date:   2016-09-28T11:31:24Z

ignite-3621 Fixed 'testEvictExpired'.

(cherry picked from commit bfe4458)

commit d2563dacceea61b19bb6e083e29ebacc28fdd323
Author: vozerov-gridgain 
Date:   2016-09-28T12:51:55Z

IGNITE-3993: Added failing client tests to "ignored" test suite.

commit 33d34941390c00e8d6a2488e8f2e11e6abba8a01
Author: sboikov 
Date:   2016-09-28T12:54:52Z

Merge remote-tracking branch 'community/ignite-1.6.9' into ignite-1.6.9

commit 78144c4c9d6200ceef8b666a186039685f053381
Author: vozerov-gridgain 
Date:   2016-09-28T13:52:13Z

Fixed incorrect test count calculation leading to afterTestsStopped() not 
being called.

commit e3dfdecc3607b5f3183bfcb1ce36c57543a8965f
Author: Alexander Paschenko 
Date:   2016-09-28T13:46:46Z

Validate hash code presence in BinaryObject. Fixes #928

commit 53229e290f7d6aab9b504693bd2b93155ecd2bad
Author: Pavel Tupitsyn 
Date:   2016-09-28T14:07:47Z

IGNITE-3995 .NET: Introduced default non-null ASP.NET Session-State Store 
Provider cache name. This closes #1128.

commit d1e3a78ae569fa5d5692816db44f2c677e1b8283
Author: Valentin Kulichenko 
Date:   2016-09-29T00:53:42Z

IGNITE-3191 - Fixed ordering of fields in binary objects

commit b280c3efa1eb84c6bc8abbe31ba669b0c24323d8
Author: vozerov-gridgain 
Date:   2016-09-29T07:23:54Z

Fixed missing Apache header.

commit 3e8a1c6b045c231dbb3c972463000f824386aee9
Author: vozerov-gridgain 
Date:   2016-09-29T08:08:23Z

Added node stop to IgniteCacheExpireAndUpdateConsistencyTest.

commit 062b9b69aa851d5253dfb7f44066b7d749f1ca12
Author: Alexey Goncharuk 
Date:   2016-09-29T08:26:50Z

IGNITE-3633 - Enforce key validation for tests.

commit a4d7aa343df23a5e00df11c5080cc1762b187161
Author: sboikov 
Date:   2016-09-29T09:04:44Z

ignite-3621 Fixed 'testEvictExpired'.

commit ad613afd7165ed2d8bf0a62805fc2506dfe6c59a
Author: sboikov 
Date:   2016-09-29T09:04:44Z

ignite-3621 Fixed 'testEvictExpired'.

(cherry picked from commit a4d7aa3)

commit 22dc2c9e05e5183f838865023c9cb7a8291ac67f
Author: Alexander Paschenko 
Date:   2016-09-29T12:43:20Z

GridNearCacheAdapter, GridDhtCacheAdapter - test fix

commit f447559ecba55149452e4e48e2bf44ef1fa1b4d7
Author: Alexey Goncharuk 
Date:   2016-09-29T14:10:14Z

ignite-1.6.9 - Fixing tests

commit a4c63ae113931615e940ada2411ebce32dba5db8
Author: 

Re: Sort nodes in the ring in order to minimize the number of reconnections

[Yakov Zhdanov]

>>
For example, ordering on latency:
- nodes on one host = 1
- nodes in one rack-blade = 2
- nodes in one server-rack = 3
- nodes in one physical cluster = 4
- nodes in one subnet = 5
- etc.

Maybe it'll be better to use some metrics from ClusterMetrics interface.

The algorithm of ordering can be implemented in a class such as Comparator
and use it when we build a cluster or we select a place for a new node.
>>

Vyacheslav, please elaborate on how we can determine whether we are on the
same rack. I am not sure this is possible in general case. Please see my
suggestions below.

>>
However, here is the concern I have. Currently when a new node joins,
coordinator assigns order number to this node (e.g. if we already have
nodes 1,2 and 3, new node will have order 4). This node will then be the
last one on the ring, i.e. nodes are always ordered in the ring by this
order number (1->2->3->4->1). If we change this, we will basically allow a
node to be placed anywhere else (smth like 1->2->4->3->1). I'm not 100%
sure if this is going to cause issues, but sounds dangerous.

Yakov, can you please chime in and share your thoughts on this?
>>

I don't think this may cause issues. Nodes ordering and placement is
implemented in TcpDiscoveryNodesRing and I think that we will just need to
alter 
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNodesRing#nextNode(java.util.Collection)
logic.

As far as design of this, I would suggest the following.

1.  User should have an ability to define ARC_ID for the node. I suggest
"arc" for this since we are using "ring" concept. This will be the most
honored characteristic for nodes placement. By default arc_id is 0 and
possible to set with system property IGNITE_DISCO_ARC_ID or env variable or
via TcpDiscoverySpi.setArcId() - new method.
So, if I have nodes A, D, G with arc_id set to 1 and B, Z with arc_id set
to 5 then ring should be built as follows: A->D->G->B->Z->A. Here arcs can
represent different racks or data centers.

I am strongly against giving user an opportunity to point exact place in
the ring with somewhat like this interface [int getIdex(Node newNode,
List currentRing)]. This is very error prone and may require tricky
consistency checks just to make sure that implementation of this interface
is consistent along the topology.
With "arcs" approach user can automatically assign proper ids basing on
physical network topology and network routes.

2. Subnet - 2nd honored parameter. Nodes on the same subnet should be
placed side by side in the same arc.

3. Physical host - 3rd honored parameter. Nodes on the same physical host
should be placed together automatically in the same arc.

4. New mode involving points 1-3 should become default and we should also
provide ability to switch to current mode which should become legacy.

--Yakov

Hello

[Alexander Fedotov]

Hello, Ignite Community!

My name is Alexandr. I want to contribute to Apache Ignite. Counting on
your help.

Thanks!

-- 
Kind regards,
Alexandr.