[jira] [Created] (IGNITE-7383) Failed to restore memory after cluster restart and activating from outdated node

[Alexandr Kuramshin (JIRA)]

Alexandr Kuramshin created IGNITE-7383:
--

 Summary: Failed to restore memory after cluster restart and 
activating from outdated node
 Key: IGNITE-7383
 URL: https://issues.apache.org/jira/browse/IGNITE-7383
 Project: Ignite
  Issue Type: Bug
  Components: persistence
Affects Versions: 2.3
Reporter: Alexandr Kuramshin


Do the following steps for reproducing the problem:

1) start nodes 0-1-2

2) stop node 2

3) create a new cache and put some data into it

4) stop remaining nodes 0-1

5) start nodes 0-1-2

6) activate the cluster from the node 2

Then 2 different results could be taken depending on which node is coordinator:

a) node 2 is a coordinator:

{noformat}
Failed to activate node components 
[nodeId=42d762c7-b1e0-4283-939b-aeeb3c70, client=false, 
topVer=AffinityTopologyVersion [topVer=3, minorTopVer=1]]
class org.apache.ignite.IgniteCheckedException: Failed to find cache group 
descriptor [grpId=3119]
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.getPageMemoryForCacheGroup(GridCacheDatabaseSharedManager.java:1602)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1544)
at 
org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:570)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:820)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:583)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2279)
at 
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:748)
{noformat}

and activation will be failed.

b) node 2 is NOT a coordinator:

we will get an error from the previous version, but the activation process will 
not be failed and then we will take "Failed to wait PME" after a number of 
assertions

{noformat}
Failed to process message [senderId=a940742f-bf17-41b4-bfc2-728bee72, 
messageType=class 
o.a.i.i.processors.cache.distributed.dht.preloader.GridDhtPartitionsSingleMessage]
java.lang.AssertionError: -2100569601
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.clientTopology(GridCachePartitionExchangeManager.java:733)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.updatePartitionSingleMap(GridDhtPartitionsExchangeFuture.java:2877)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.processSingleMessage(GridDhtPartitionsExchangeFuture.java:1935)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.access$100(GridDhtPartitionsExchangeFuture.java:116)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:1810)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture$2.apply(GridDhtPartitionsExchangeFuture.java:1798)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
at 
org.apache.ignite.internal.util.future.GridFutureAdapter.listen(GridFutureAdapter.java:353)
at 
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onReceiveSingleMessage(GridDhtPartitionsExchangeFuture.java:1798)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processSinglePartitionUpdate(GridCachePartitionExchangeManager.java:1484)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.access$1000(GridCachePartitionExchangeManager.java:131)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:327)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$2.onMessage(GridCachePartitionExchangeManager.java:307)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:2627)
at 
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:2606)
at 

[jira] [Created] (IGNITE-7382) Unknown Pair exception if work directory is removed after cluster shutdown

[Sunny Chan (JIRA)]

Sunny Chan created IGNITE-7382:
--

 Summary: Unknown Pair exception if work directory is removed after 
cluster shutdown
 Key: IGNITE-7382
 URL: https://issues.apache.org/jira/browse/IGNITE-7382
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.3
 Environment: Redhat Linux 7.4

Reporter: Sunny Chan


To reproduce, try this:

1) Start a server node
2) Start a node in client mode, connect to server node
3) shutdown the server node and _leave the client node running_
4) remove Ignite work directory for the shutdown server node
5) Client node reconnects automatically, send a service call request
6) Exception occurs on Server, unable to deserialize 
org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2

{noformat}
2018-01-05 00:00:26.027 processors.job.GridJobWorker [svc-#273%clog%] ERROR - 
Failed to initialize job 
[jobId=839fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7, ses=GridJobSessionImpl 
[ses=GridTaskSessionImpl [taskName=o.a.i.i.processors.
service.GridServiceProxy$ServiceProxyCallable, dep=LocalDeployment 
[super=GridDeployment [ts=1515076515280, depMode=SHARED, 
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, 
clsLdrId=3e4f891c061-5235b03b-87c3-4c29-a576-1b44936b0c11, user
Ver=0, loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, 
undeployed=false, usage=0]], 
taskClsName=o.a.i.i.processors.service.GridServiceProxy$ServiceProxyCallable, 
sesId=739fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7, st
artTime=1515078026015, endTime=1515078036021, 
taskNodeId=ad77f485-d677-4694-8d0c-e780f16be9b7, 
clsLdr=sun.misc.Launcher$AppClassLoader@764c12b6, closed=false, cpSpi=null, 
failSpi=null, loadSpi=null, usage=1, fullSup=false, internal=false
, subjId=ad77f485-d677-4694-8d0c-e780f16be9b7, mapFut=IgniteFuture 
[orig=GridFutureAdapter [ignoreInterrupts=false, state=INIT, res=null, 
hash=360535376]], execName=null], 
jobId=839fef0c061-ad77f485-d677-4694-8d0c-e780f16be9b7]]
org.apache.ignite.IgniteCheckedException: Failed to deserialize object 
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
at 
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9859) 
[logic.jar:1.1.0]
at 
org.apache.ignite.internal.processors.job.GridJobWorker.initialize(GridJobWorker.java:438)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1109)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1913)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1555)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1183)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126)
 [logic.jar:1.1.0]
at 
org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1090)
 [logic.jar:1.1.0]
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
[?:1.8.0_121]
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
[?:1.8.0_121]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121]
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to 
deserialize object 
[typeName=org.apache.ignite.internal.processors.closure.GridClosureProcessor$C2]
at 
org.apache.ignite.internal.binary.BinaryClassDescriptor.read(BinaryClassDescriptor.java:874)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1762)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize(BinaryReaderExImpl.java:1714)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.binary.GridBinaryMarshaller.deserialize(GridBinaryMarshaller.java:310)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.binary.BinaryMarshaller.unmarshal0(BinaryMarshaller.java:99)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:82)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9853) 
[logic.jar:1.1.0]
... 10 more
Caused by: org.apache.ignite.binary.BinaryObjectException: Failed to unmarshal 
object with optimized marshaller
at 
org.apache.ignite.internal.binary.BinaryUtils.doReadOptimized(BinaryUtils.java:1786)
 ~[logic.jar:1.1.0]
at 
org.apache.ignite.internal.binary.BinaryReaderExImpl.deserialize0(BinaryReaderExImpl.java:1962)
 ~[logic.jar:1.1.0]
at 

[jira] [Created] (IGNITE-7381) visorcmd: improve output of 'modify' command - show Value Class info

[Pavel Konstantinov (JIRA)]

Pavel Konstantinov created IGNITE-7381:
--

 Summary: visorcmd: improve output of 'modify' command - show Value 
Class info
 Key: IGNITE-7381
 URL: https://issues.apache.org/jira/browse/IGNITE-7381
 Project: Ignite
  Issue Type: Improvement
Reporter: Pavel Konstantinov
Priority: Trivial


In case if cache contains several keys with the same value but different type 
it is unclear what type of returned value in the result of command -get and 
-remove



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Los Angeles Area Apache Ignite/Kubernetes Meetup

[Denis Magda]

cross-posting to the user list

> On Jan 10, 2018, at 3:17 PM, Dani Traphagen  wrote:
> 
> Hi everyone,
> 
> 
> *January 31st *I will be speaking in Los Angeles at *Verizon Digital Media
> Services* on using Apache Ignite with Kubernetes!
> 
> 
> You can read all the meetup details here
>  along
> with the abstract description. Free Food and Drinks will be provided.
> 
> 
> Hope to see you there if you are in the area!
> 
> 
> Cheers,
> Dani
> 
> 
> -- 
> Dani Traphagen | d...@gridgain.com
> Solutions Architect
> *GridGain*

Los Angeles Area Apache Ignite/Kubernetes Meetup

[Dani Traphagen]

Hi everyone,


*January 31st *I will be speaking in Los Angeles at *Verizon Digital Media
Services* on using Apache Ignite with Kubernetes!


You can read all the meetup details here
 along
with the abstract description. Free Food and Drinks will be provided.


Hope to see you there if you are in the area!


Cheers,
Dani


-- 
Dani Traphagen | d...@gridgain.com
Solutions Architect
*GridGain*

[GitHub] ignite pull request #3356: IGNITE-7126: add new cache metrics parameters

[AlexeyRokhin]

GitHub user AlexeyRokhin opened a pull request:

https://github.com/apache/ignite/pull/3356

IGNITE-7126: add new cache metrics parameters

Required parameters were added.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/AlexeyRokhin/ignite ignite-7126

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3356.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3356


commit 6cde9dad36af56926e54304edb7189ae58c69b78
Author: Alexey Rokhin 
Date:   2018-01-10T22:10:26Z

IGNITE-7126: add new cache metrics parameters




---

Re: DiscoverySpi based on Apache ZooKeeper

[Andrey Kornev]

Vladimir,

I wish I could be more helpful, but as I mentioned, I can't remember the exact 
details. It had something to do with rather low level technical details of the 
watcher implementation in Zookeeper.

Also, I never said Zookeeper doesn't provide total ordering guarantees. In 
fact, its sole purpose of existence is to do just that: totally ordered atomic 
broadcast. My point was more about "impedance mismatch" between Zookeeper API 
and DiscoverySpi.

Regards
Andrey


From: Vladimir Ozerov 
Sent: Tuesday, January 9, 2018 11:38 PM
To: dev@ignite.apache.org
Cc: Semyon Boikov
Subject: Re: DiscoverySpi based on Apache ZooKeeper

Hi Andrey,

Could you please share detail of this API mismatch? AFAIK, the main
guarantee we need for disco SPI is total message ordering. Zookeeper
provides this guarantee. Moreover, Zookeeper is proven to be correct and
reliable coordinator service by many users and Jepsen tests, as opposed to
various in-house implementations (e.g. Zen of Elasticsearch).

вт, 9 янв. 2018 г. в 21:53, Andrey Kornev :

> Semyon,
>
> Not to discourage you or anything, just a interesting fact from recent
> history.
>
> I vaguely remember already trying to implement DiscoverySpi on top of
> Zookeeper back in 2012. After a few failed attempts and a lot of help from
> Zookeeper's original developers (Flavio Junqueira and Ben Reed) we (Dmitriy
> S. and I) concluded that its not possible to  implement DiscoverySpi on top
> of Zookeeper due to strict(er) semantics of DiscoverySpi. Unfortunately I
> do not remember details, but essentially, in some cases it was not possible
> to establish total ordering of watcher events and under certain
> circumstances loss of such events was possible.
>
> It's not to say that Zookeeper can't be used to implement the cluster
> membership tracking in general. The problem is rather with DiscoverySpi
> semantics that require a different set of APIs than what Zookeeper provides.
>
> Regards
> Andrey
>
> 
> From: Semyon Boikov 
> Sent: Tuesday, January 9, 2018 3:39 AM
> To: dev@ignite.apache.org
> Subject: DiscoverySpi based on Apache ZooKeeper
>
> Hi all,
>
> Currently I'm working on implementation of DiscoverySpi based on Apache
> ZooKeeper (ZookeeperDiscoverySpi) and want to share results of this work.
>
> In very large clusters (>1000 nodes) current default implementation of
> DiscoverySpi - TcpDiscoverySpi - has some significant drawbacks:
> - TcpDiscoverySpi organizes nodes in ring, and all messages are passed
> sequentially via ring. More nodes there are in ring, more time it takes to
> pass message. In very large clusters such architecture can cause slowdown
> of important operations (node join, node stop, cache start, etc).
> - in TcpDiscoverySpi's protocol each node in ring is able to fail next one
> in case of network issues, and it is possible that two nodes can 'kill'
> each other (it is possible in complex scenarios when network is broken and
> then restored back after some time, such problems were observed in real
> environments), and with current TcpDiscoverySpi protocol there is no easy
> way to completely fix such problems.
> - when some node in ring fails, then previous node tries to restore ring
> and sequentially tries to connect to next nodes. If large part of ring
> fails then it takes long time to sequentially detect failure of all nodes.
> - with TcpDiscoverySpi split brain is possible (one ring can split into two
> independent parts), separate mechanism is needed to protect from split
> brain when TcpDiscoverySpi is used
>
> Even though most probably some of these problems can be somehow fixed in
> TcpDiscoverySpi, it seems more robust and fast DiscoverySpi can be
> implemented on top of some existing coordination service.
>
> Apache ZooKeeper is known reliable service and it provides all mechanisms
> and consistency guarantees required for Ignite's DiscoverySpi. Some
> technical details of ZookeeperDiscoverySpi implementation can be found in
> description prepared by Sergey Puchnin:
>
> https://cwiki.apache.org/confluence/display/IGNITE/Discovery+SPI+by+ZooKeeper
> .
>
> In our preliminary tests we were able to successfully start 4000+ nodes
> with ZookeeperDiscoverySpi. New implementation works faster than
> TcpDiscoverySpi and does not have mentioned TcpDiscoverySpi's drawbacks:
> - nodes alive status is controlled by ZooKeeper, so nodes never can kill
> each other
> - ZooKeeper has protection from split brain
> - with ZooKeeper it is possible to detect nodes join/failures in batches so
> time to detect join/failure of 1 vs 100+ nodes is almost the same
>
> I'm going to finalize implementation of ZookeeperDiscoverySpi in next few
> days. But in Ignite there is one more issue caused by the fact that
> DiscoverySpi and CommunicationSpi are two independent component: it is
> possible that DiscoverySpi considers some 

Re: Thin Client examples for documentation

[Denis Magda]

Pavel, as a side note,

The methods/operations Prachi is struggling with look pretty standard to me.

Do you have tests for them in the code base? I mean *not* the tests you shared 
before where we use existing internal binary marshaller APIs but where we code 
every operation from scratch (what Prachi is doing for documentation code 
snippets).

Such tests would help to complete the doc quicker and would ensure that the 
protocol works as expected on the user side where people are not going to sit 
on the internal binary marshaller apis.

—
Denis

> On Jan 10, 2018, at 12:29 PM, Prachi Garg  wrote:
> 
> Pavel,
> 
> I am having trouble creating examples for some of the thin protocol
> operations. I have uploaded my project on github -
> 
> https://github.com/pgarg/ignite-examples/blob/master/src/main/java/ignite/myexamples/thinclient/ThinClientExample2.java
> 
> Please look into the following methods and provide a fix for them:
> 
>   - doSQLQuery()
>   - getQueryCursorPage()
>   - putBinaryType()
>   - doQueryScan()
>   - createCacheWithConfiguration()
> 
> Thanks,
> -Prachi

Thin Client examples for documentation

[Prachi Garg]

Pavel,

I am having trouble creating examples for some of the thin protocol
operations. I have uploaded my project on github -

https://github.com/pgarg/ignite-examples/blob/master/src/main/java/ignite/myexamples/thinclient/ThinClientExample2.java

Please look into the following methods and provide a fix for them:

   - doSQLQuery()
   - getQueryCursorPage()
   - putBinaryType()
   - doQueryScan()
   - createCacheWithConfiguration()

Thanks,
-Prachi

Re: Scala 2.10 support

[Petr Ivanov]

From the point of build configuration — it is required to delete 
modules/{scalar-2.10,spark-2.10,visor-console-2.10} directories and 
corresponding modules from pom.xml (they are currently disabled in ignite-6730).
From the point of project operability and consistency — I’ve started this 
discussion just to hear opinions on subject from Ignite experts will there be 
any pitfalls in simple modules removal from project.



> On 10 Jan 2018, at 22:02, Dmitriy Setrakyan  wrote:
> 
> I agree that we should drop it, I just want to understand how difficult it
> is. What is the amount of effort associated with this change?
> 
> On Wed, Jan 10, 2018 at 10:37 AM, Petr Ivanov  wrote:
> 
>> As much as I see, scala-2.10, spark-2.10 and visor-console-2.10 modules
>> require scala 2.10 by dependency, which is not going to work any way (or at
>> least without some hacky hack) with Java 9.
>> Considering Java 9 to be strictly 2.4+ functionality — I guess it is more
>> or less solid approach in dropping support of deprecated (for some decent
>> time) technologies in next releases in favour of the newest ones where
>> there is no other proper way of utilising them both.
>> 
>> 
>> 
>>> On 10 Jan 2018, at 18:55, Dmitriy Setrakyan 
>> wrote:
>>> 
>>> How much of an effort will be to support scala and spark 2.10?
>>> 
>>> On Wed, Jan 10, 2018 at 4:24 AM, vveider  wrote:
>>> 
 Hi, Igniters!
 
 
 While working on Java 9 support adaptation, came to a problem with scala
 2.10 and corresponding modules (scala-2.10, spark-2.10 and
 visor-console-2.10) which will not compile under Java 9. As I see here
>> [1]
 scala 2.10 is already deprecated in spark 2.1.0 (and lately spark
>> version
 updated to 2.2.0 in master).
 
 So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
 modules for deletion in 2.4 release.
 
 Support for scala 2.11.x (which also will not compile under Java 9)
>> will be
 maintained by maven profiles.
 
 
 [1] http://spark.apache.org/docs/latest/
 
 
 
 --
 Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
 
>> 
>> 

Re: Irrelevant data in discovery messages

[Dmitriy Setrakyan]

Absolutely agree. I thought we could already filter out system and
environment properties on startup via configuration. I think it was
implemented by Yakov a long time ago. Yakov Zhdanov, can you please chime
in? (

Denis, the information you mention is static and does not change. It would
be enough to include it only into Join requests and not regular heartbeats.
I hope that it is already happening this way. Can you please confirm?

D.

On Wed, Jan 10, 2018 at 12:30 AM, Denis Mekhanikov 
wrote:

> Igniters,
>
> Turns out, that we are sending a lot of irrelevant information in discovery
> messages. Some messages contain *TcpDiscoveryNode* objects, which in turn
> have such attributes like *PATH, java.class.path, sun.boot.class.path,
> java.library.path, org.apache.ignite.jvm.args, *etc.
> Some of these attributes may contain huge strings, that can sum up to
> megabytes of data.
>
> It was noticed by a user on our mailing list:
> http://apache-ignite-users.70518.x6.nabble.com/Connection-problem-between-
> client-and-server-td19243.html
> In his case these huge messages make discovery process really slow.
>
> I think, we should filter-out such attributes, because they are not used
> anywhere, but make messages grow enormous and slow down discovery. We could
> include only user-defined and internal attributes + a fixed set of
> environment variables.
>
> What do you think?
>
> Denis
>

Re: Scala 2.10 support

[Dmitriy Setrakyan]

I agree that we should drop it, I just want to understand how difficult it
is. What is the amount of effort associated with this change?

On Wed, Jan 10, 2018 at 10:37 AM, Petr Ivanov  wrote:

> As much as I see, scala-2.10, spark-2.10 and visor-console-2.10 modules
> require scala 2.10 by dependency, which is not going to work any way (or at
> least without some hacky hack) with Java 9.
> Considering Java 9 to be strictly 2.4+ functionality — I guess it is more
> or less solid approach in dropping support of deprecated (for some decent
> time) technologies in next releases in favour of the newest ones where
> there is no other proper way of utilising them both.
>
>
>
> > On 10 Jan 2018, at 18:55, Dmitriy Setrakyan 
> wrote:
> >
> > How much of an effort will be to support scala and spark 2.10?
> >
> > On Wed, Jan 10, 2018 at 4:24 AM, vveider  wrote:
> >
> >> Hi, Igniters!
> >>
> >>
> >> While working on Java 9 support adaptation, came to a problem with scala
> >> 2.10 and corresponding modules (scala-2.10, spark-2.10 and
> >> visor-console-2.10) which will not compile under Java 9. As I see here
> [1]
> >> scala 2.10 is already deprecated in spark 2.1.0 (and lately spark
> version
> >> updated to 2.2.0 in master).
> >>
> >> So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
> >> modules for deletion in 2.4 release.
> >>
> >> Support for scala 2.11.x (which also will not compile under Java 9)
> will be
> >> maintained by maven profiles.
> >>
> >>
> >> [1] http://spark.apache.org/docs/latest/
> >>
> >>
> >>
> >> --
> >> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >>
>
>

Re: Scala 2.10 support

[Petr Ivanov]

As much as I see, scala-2.10, spark-2.10 and visor-console-2.10 modules require 
scala 2.10 by dependency, which is not going to work any way (or at least 
without some hacky hack) with Java 9.
Considering Java 9 to be strictly 2.4+ functionality — I guess it is more or 
less solid approach in dropping support of deprecated (for some decent time) 
technologies in next releases in favour of the newest ones where there is no 
other proper way of utilising them both.



> On 10 Jan 2018, at 18:55, Dmitriy Setrakyan  wrote:
> 
> How much of an effort will be to support scala and spark 2.10?
> 
> On Wed, Jan 10, 2018 at 4:24 AM, vveider  wrote:
> 
>> Hi, Igniters!
>> 
>> 
>> While working on Java 9 support adaptation, came to a problem with scala
>> 2.10 and corresponding modules (scala-2.10, spark-2.10 and
>> visor-console-2.10) which will not compile under Java 9. As I see here [1]
>> scala 2.10 is already deprecated in spark 2.1.0 (and lately spark version
>> updated to 2.2.0 in master).
>> 
>> So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
>> modules for deletion in 2.4 release.
>> 
>> Support for scala 2.11.x (which also will not compile under Java 9) will be
>> maintained by maven profiles.
>> 
>> 
>> [1] http://spark.apache.org/docs/latest/
>> 
>> 
>> 
>> --
>> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>> 

[GitHub] ignite pull request #3355: Ignite 7195 hotfix

[abeliak]

GitHub user abeliak opened a pull request:

https://github.com/apache/ignite/pull/3355

Ignite 7195 hotfix

Fix printing complex objects with multiple recursive usages of 
GridToStringBuilder

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7195-hotfix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3355.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3355


commit f67ed19153e3ed7d01b8b3d654da81dd6222657b
Author: Alexander Belyak 
Date:   2018-01-10T16:54:16Z

ignite-7195: hotfix

commit 1fe477717ddf194f001875b74f4f1613ae09794b
Author: Alexander Belyak 
Date:   2018-01-10T17:02:09Z

ignite-7195: hotfix

commit 628d527dd16711230af2de5caa2b2ea080793627
Author: Alexander Belyak 
Date:   2018-01-10T17:37:00Z

ignite-7195: hotfix restore example debug




---

[GitHub] ignite pull request #3354: IGNITE-7380: Avoid updating PagePartitionCounters...

[dspavlov]

GitHub user dspavlov opened a pull request:

https://github.com/apache/ignite/pull/3354

IGNITE-7380: Avoid updating PagePartitionCounters in case all counters were 
not modified



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7380

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3354.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3354


commit e7ca9b65a68de7752195c8f4d2b5180f3c77d19f
Author: Dmitriy Govorukhin 
Date:   2017-11-13T18:52:47Z

ignite-blt-merge -> ignite-2.4.1

commit cc8168fc184bb7f5e3cc3bbb0743397097f78bfb
Author: Dmitriy Govorukhin 
Date:   2017-11-13T19:13:01Z

merge ignite-pitr-rc1 -> ignite-2.4.1

commit 87e6d74cf6a251c7984f9e68c391f790feccc281
Author: Dmitriy Govorukhin 
Date:   2017-11-14T12:49:33Z

ignite-gg-12877 Compact consistent ID in WAL

commit 9f5a22711baea05bd37ab07c8f928a4837dd83a4
Author: Ilya Lantukh 
Date:   2017-11-14T14:12:28Z

Fixed javadoc.

commit d5af2d78dd8eef8eca8ac5391d31d8c779649bb0
Author: Alexey Kuznetsov 
Date:   2017-11-15T08:09:00Z

IGNITE-6913 Baseline: Added new options to controls.sh for baseline 
manipulations.

commit 713924ce865752b6e99b03bd624136541cea5f9f
Author: Sergey Chugunov 
Date:   2017-11-15T09:03:12Z

IGNITE-5850 failover tests for cache operations during BaselineTopology 
changes

commit b65fd134e748d496f732ec2aa0953a0531f544b8
Author: Ilya Lantukh 
Date:   2017-11-15T12:54:35Z

TX read logging if PITR is enabled.

commit 9b2a567c0e04dc33116b51f88bee75f76e9107d1
Author: Ilya Lantukh 
Date:   2017-11-15T13:45:16Z

TX read logging if PITR is enabled.

commit 993058ccf0b2b8d9e80750c3e45a9ffa31d85dfa
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:51:54Z

ignite-2.4.1 optimization for store full set node more compacted

commit 1eba521f608d39967aec376b397b7fc800234e54
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:52:22Z

Merge remote-tracking branch 'professional/ignite-2.4.1' into ignite-2.4.1

commit 564b3fd51f8a7d1d81cb6874df66d0270623049c
Author: Sergey Chugunov 
Date:   2017-11-15T14:00:51Z

IGNITE-5850 fixed issue with initialization of data regions on node 
activation, fixed issue with auto-activation when random node joins inactive 
cluster with existing BLT

commit c6d1fa4da7adfadc80abdc7eaf6452b86a4f6aa4
Author: Sergey Chugunov 
Date:   2017-11-15T16:23:08Z

IGNITE-5850 transitionResult is set earlier when request for changing 
BaselineTopology is sent

commit d65674363163e38a4c5fdd73d1c8d8e1c7610797
Author: Sergey Chugunov 
Date:   2017-11-16T11:59:07Z

IGNITE-5850 new failover tests for changing BaselineTopology up (new node 
added to topology)

commit 20552f3851fe8825191b144179be032965e0b5c6
Author: Sergey Chugunov 
Date:   2017-11-16T12:53:43Z

IGNITE-5850 improved error message when online node is removed from baseline

commit 108bbcae4505ac904a6db774643ad600bfb42c21
Author: Sergey Chugunov 
Date:   2017-11-16T13:45:52Z

IGNITE-5850 BaselineTopology should not change on cluster deactivation

commit deb641ad3bdbf260fa60ad6bf607629652e324bd
Author: Dmitriy Govorukhin 
Date:   2017-11-17T09:45:44Z

ignite-2.4.1 truncate wal and checkpoint history on move/delete snapshot

commit 3c8b06f3659af30d1fd148ccc0f40e216a56c998
Author: Alexey Goncharuk 
Date:   2017-11-17T12:48:12Z

IGNITE-6947 Abandon remap after single map if future is done (fixes NPE)

commit ba2047e5ae7d271a677e0c418375d82d78c4023e
Author: devozerov 
Date:   2017-11-14T12:26:31Z

IGNITE-6901: Fixed assertion during 
IgniteH2Indexing.rebuildIndexesFromHash. This closes #3027.

commit abfc0466d6d61d87255d0fe38cbdf11ad46d4f89
Author: Sergey Chugunov 
Date:   2017-11-17T13:40:57Z

IGNITE-5850 tests for queries in presence of BaselineTopology

commit f4eabaf2a905abacc4c60c01d3ca04f6ca9ec188
Author: Sergey Chugunov 
Date:   2017-11-17T17:23:02Z

IGNITE-5850 implementation for setBaselineTopology(long topVer) migrated 
from wc-251

commit 4edeccd3e0b671aa277f58995df9ff9935baa95a
Author: EdShangGG 
Date:   2017-11-17T18:21:17Z

GG-13074 Multiple snapshot test failures after baseline topology is 
introduced
-adding baseline test to suite
-fixing issues with baseline

commit edae228c8f55990c15ef3044be987dcb00d6c81a
Author: EdShangGG 
Date:   2017-11-18T10:36:41Z

hack with sleep

[jira] [Created] (IGNITE-7380) Avoid updating CountersPageId in case all counters was not modifed

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-7380:
--

 Summary: Avoid updating CountersPageId in case all counters was 
not modifed
 Key: IGNITE-7380
 URL: https://issues.apache.org/jira/browse/IGNITE-7380
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov
Priority: Critical


PagePartitionCountersIO ignores actual values of counters and marks pages dirty 
even if partition has no counters update.

This leads to  PAGE_RECORD creation for pages from PagePartitionCountersIO

and as result with pages from TrackingPageIO.

These modificaiton may be skipped if counters are checked for equality before 
storing data.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3332: Ignite 7217 2.3

[slukyano]

Github user slukyano closed the pull request at:

https://github.com/apache/ignite/pull/3332


---

[GitHub] ignite pull request #3334: Ignite 7217 2.1

[slukyano]

Github user slukyano closed the pull request at:

https://github.com/apache/ignite/pull/3334


---

Re: Java 9 support

[Andrey Gura]

Andrey,

performance testing is in progress. The first, there are no
significant difference in performance on Java 8. But it would be great
to test performance on Java 9 also.

On Wed, Jan 10, 2018 at 1:25 PM, Andrey Kuznetsov  wrote:
> Thanks, Petr.
>
> I heard of some activity related to performance consequences of
> ReentrantLocks in IGNITE-6736 fix. So, I'd like to get a reviewer feedback
> first.
>
> Andrey G., Vladimir O., is it possible to merge the fix to master?
>
> 2018-01-10 9:56 GMT+03:00 Petr Ivanov :
>
>> Andrey — double checked your solution and it works now. Guess there was
>> some merge error first time.
>> Sorry for misleading.
>>
>> So, there is working solution for Java 9 build and I’d like to save this
>> configuration in ignite-6730 (making IGNITE-7144 and IGNITE-6736 to become
>> subtasks in the process).
>> What do you think?
>>
>>
>> > On 9 Jan 2018, at 20:49, Andrey Kuznetsov  wrote:
>> >
>> > Hi Petr!
>> >
>> > Could you please clarify what is wrong with fix proposed in IGNITE-6736,
>> > and what is supposed to be a replacement for monitorEnter/monitorExit
>> now?
>> >
>> > 2018-01-09 19:08 GMT+03:00 Petr Ivanov :
>> >
>> >> Hi all.
>> >>
>> >>
>> >> After some thorough research and with help of fellow igniters, I’ve
>> >> managed to prepare more or less stable Java 9 build configuration of
>> Apache
>> >> Ignite.
>> >>
>> >> Here are changes to make it work:
>> >> - Java 8 profiles and build process revision, made in IGNITE-7203;
>> >> - Java 9 maven profile prepared in IGNITE-7144 (will be moved to
>> >> IGNITE-6730 as subtask);
>> >> - specific maven-compiler-plugin configuration with JVM args for Java 9
>> >> profile (as was proposed by Vladimir Ozerov);
>> >> - maven-bundle-plugin version is updated to 3.5.0;
>> >> - maven-compiler-plugin version synchronised to 3.7.0 (in Cassandra
>> >> modules);
>> >> - scala version updated to 2.12.4;
>> >> - disabled scalar-2.10, spark-2.10 and visor-console-2.10 modules (due
>> to
>> >> dependency in scala 2.10 which is unsupported by Java 9);
>> >> - sun.misc.JavaNioAccess import changed to jdk.internal.misc.
>> JavaNioAccess
>> >> in GridUnsafe.java and PageMemoryImpl.java;
>> >> - sun.misc.SharedSecrets import changed to jdk.internal.misc.
>> SharedSecrets
>> >> in GridUnsafe.java and PageMemoryImpl.java;
>> >> - methods monitorEnter and monitorExit bodies commented out (fix from
>> >> IGNITE-6736 did not work).
>> >>
>> >> I’d like to put these changes into ignite-6730 to have working compiling
>> >> under Java 9 branch — so that we can continue work on improving Apache
>> >> Ignite’s Java 9 support.
>> >
>> >
>> >
>> >
>> > --
>> > Best regards,
>> >  Andrey Kuznetsov.
>>
>>
>
>
> --
> Best regards,
>   Andrey Kuznetsov.

[GitHub] ignite pull request #3353: IGNITE-7191 JDBC: set socket buffer to 64k by def...

[apopovgg]

GitHub user apopovgg opened a pull request:

https://github.com/apache/ignite/pull/3353

IGNITE-7191 JDBC: set socket buffer to 64k by default



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7191

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3353.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3353


commit 5f6bd3e2fa0567160031cdd067b3e2b1f20c9972
Author: apopov 
Date:   2018-01-10T16:08:05Z

IGNITE-7191 JDBC: set socket buffer to 64k by default




---

Re: Scala 2.10 support

[Dmitriy Setrakyan]

How much of an effort will be to support scala and spark 2.10?

On Wed, Jan 10, 2018 at 4:24 AM, vveider  wrote:

> Hi, Igniters!
>
>
> While working on Java 9 support adaptation, came to a problem with scala
> 2.10 and corresponding modules (scala-2.10, spark-2.10 and
> visor-console-2.10) which will not compile under Java 9. As I see here [1]
> scala 2.10 is already deprecated in spark 2.1.0 (and lately spark version
> updated to 2.2.0 in master).
>
> So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
> modules for deletion in 2.4 release.
>
> Support for scala 2.11.x (which also will not compile under Java 9) will be
> maintained by maven profiles.
>
>
> [1] http://spark.apache.org/docs/latest/
>
>
>
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>

[jira] [Created] (IGNITE-7379) .NET: CacheConfiguration.IsSqlOnheapCacheEnabled

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-7379:
--

 Summary: .NET: CacheConfiguration.IsSqlOnheapCacheEnabled
 Key: IGNITE-7379
 URL: https://issues.apache.org/jira/browse/IGNITE-7379
 Project: Ignite
  Issue Type: Improvement
  Components: platforms
Affects Versions: 2.4
Reporter: Pavel Tupitsyn
Priority: Trivial


Add {{CacheConfiguration.IsSqlOnheapCacheEnabled}}, see IGNITE-7173



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3352: IGNITE-7301 .NET: Baseline topology

[ptupitsyn]

GitHub user ptupitsyn opened a pull request:

https://github.com/apache/ignite/pull/3352

IGNITE-7301 .NET: Baseline topology



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ptupitsyn/ignite ignite-7301

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3352.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3352


commit e7ca9b65a68de7752195c8f4d2b5180f3c77d19f
Author: Dmitriy Govorukhin 
Date:   2017-11-13T18:52:47Z

ignite-blt-merge -> ignite-2.4.1

commit cc8168fc184bb7f5e3cc3bbb0743397097f78bfb
Author: Dmitriy Govorukhin 
Date:   2017-11-13T19:13:01Z

merge ignite-pitr-rc1 -> ignite-2.4.1

commit 87e6d74cf6a251c7984f9e68c391f790feccc281
Author: Dmitriy Govorukhin 
Date:   2017-11-14T12:49:33Z

ignite-gg-12877 Compact consistent ID in WAL

commit 9f5a22711baea05bd37ab07c8f928a4837dd83a4
Author: Ilya Lantukh 
Date:   2017-11-14T14:12:28Z

Fixed javadoc.

commit d5af2d78dd8eef8eca8ac5391d31d8c779649bb0
Author: Alexey Kuznetsov 
Date:   2017-11-15T08:09:00Z

IGNITE-6913 Baseline: Added new options to controls.sh for baseline 
manipulations.

commit 713924ce865752b6e99b03bd624136541cea5f9f
Author: Sergey Chugunov 
Date:   2017-11-15T09:03:12Z

IGNITE-5850 failover tests for cache operations during BaselineTopology 
changes

commit b65fd134e748d496f732ec2aa0953a0531f544b8
Author: Ilya Lantukh 
Date:   2017-11-15T12:54:35Z

TX read logging if PITR is enabled.

commit 9b2a567c0e04dc33116b51f88bee75f76e9107d1
Author: Ilya Lantukh 
Date:   2017-11-15T13:45:16Z

TX read logging if PITR is enabled.

commit 993058ccf0b2b8d9e80750c3e45a9ffa31d85dfa
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:51:54Z

ignite-2.4.1 optimization for store full set node more compacted

commit 1eba521f608d39967aec376b397b7fc800234e54
Author: Dmitriy Govorukhin 
Date:   2017-11-15T13:52:22Z

Merge remote-tracking branch 'professional/ignite-2.4.1' into ignite-2.4.1

commit 564b3fd51f8a7d1d81cb6874df66d0270623049c
Author: Sergey Chugunov 
Date:   2017-11-15T14:00:51Z

IGNITE-5850 fixed issue with initialization of data regions on node 
activation, fixed issue with auto-activation when random node joins inactive 
cluster with existing BLT

commit c6d1fa4da7adfadc80abdc7eaf6452b86a4f6aa4
Author: Sergey Chugunov 
Date:   2017-11-15T16:23:08Z

IGNITE-5850 transitionResult is set earlier when request for changing 
BaselineTopology is sent

commit d65674363163e38a4c5fdd73d1c8d8e1c7610797
Author: Sergey Chugunov 
Date:   2017-11-16T11:59:07Z

IGNITE-5850 new failover tests for changing BaselineTopology up (new node 
added to topology)

commit 20552f3851fe8825191b144179be032965e0b5c6
Author: Sergey Chugunov 
Date:   2017-11-16T12:53:43Z

IGNITE-5850 improved error message when online node is removed from baseline

commit 108bbcae4505ac904a6db774643ad600bfb42c21
Author: Sergey Chugunov 
Date:   2017-11-16T13:45:52Z

IGNITE-5850 BaselineTopology should not change on cluster deactivation

commit deb641ad3bdbf260fa60ad6bf607629652e324bd
Author: Dmitriy Govorukhin 
Date:   2017-11-17T09:45:44Z

ignite-2.4.1 truncate wal and checkpoint history on move/delete snapshot

commit 3c8b06f3659af30d1fd148ccc0f40e216a56c998
Author: Alexey Goncharuk 
Date:   2017-11-17T12:48:12Z

IGNITE-6947 Abandon remap after single map if future is done (fixes NPE)

commit ba2047e5ae7d271a677e0c418375d82d78c4023e
Author: devozerov 
Date:   2017-11-14T12:26:31Z

IGNITE-6901: Fixed assertion during 
IgniteH2Indexing.rebuildIndexesFromHash. This closes #3027.

commit abfc0466d6d61d87255d0fe38cbdf11ad46d4f89
Author: Sergey Chugunov 
Date:   2017-11-17T13:40:57Z

IGNITE-5850 tests for queries in presence of BaselineTopology

commit f4eabaf2a905abacc4c60c01d3ca04f6ca9ec188
Author: Sergey Chugunov 
Date:   2017-11-17T17:23:02Z

IGNITE-5850 implementation for setBaselineTopology(long topVer) migrated 
from wc-251

commit 4edeccd3e0b671aa277f58995df9ff9935baa95a
Author: EdShangGG 
Date:   2017-11-17T18:21:17Z

GG-13074 Multiple snapshot test failures after baseline topology is 
introduced
-adding baseline test to suite
-fixing issues with baseline

commit edae228c8f55990c15ef3044be987dcb00d6c81a
Author: EdShangGG 
Date:   2017-11-18T10:36:41Z

hack with sleep

commit b5bffc7580a4a8ffbcc06f60c282e73979179578
Author: Ilya 

[GitHub] ignite pull request #3351: ignite-7352 sun.misc.SharedSecrets and sun.misc.J...

[agura]

GitHub user agura opened a pull request:

https://github.com/apache/ignite/pull/3351

ignite-7352 sun.misc.SharedSecrets and sun.misc.JavaNioAccess direct usages 
are removed

…usages are removed

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/agura/incubator-ignite ignite-7352

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3351.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3351


commit c24b9031fb032800424d83b1ba713c0479bbd1a9
Author: Andrey Gura 
Date:   2018-01-10T14:51:51Z

ignite-7352 sun.misc.SharedSecrets and sun.misc.JavaNioAccess direct usages 
are removed




---

Re: Scala 2.10 support

[ALEKSEY KUZNETSOV]

+1

ср, 10 янв. 2018 г. в 17:28, Alexey Kuznetsov :

> +1
>
> On Wed, Jan 10, 2018 at 7:27 PM, Nikolay Izhikov 
> wrote:
>
> > +1 from me.
> >
> > В Ср, 10/01/2018 в 05:24 -0700, vveider пишет:
> > > Hi, Igniters!
> > >
> > >
> > > While working on Java 9 support adaptation, came to a problem with
> > > scala
> > > 2.10 and corresponding modules (scala-2.10, spark-2.10 and
> > > visor-console-2.10) which will not compile under Java 9. As I see
> > > here [1]
> > > scala 2.10 is already deprecated in spark 2.1.0 (and lately spark
> > > version
> > > updated to 2.2.0 in master).
> > >
> > > So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
> > > modules for deletion in 2.4 release.
> > >
> > > Support for scala 2.11.x (which also will not compile under Java 9)
> > > will be
> > > maintained by maven profiles.
> > >
> > >
> > > [1] http://spark.apache.org/docs/latest/
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
> >
>
>
>
> --
> Alexey Kuznetsov
>


-- 

*Best Regards,*

*Kuznetsov Aleksey*

Re: Scala 2.10 support

[Alexey Kuznetsov]

+1

On Wed, Jan 10, 2018 at 7:27 PM, Nikolay Izhikov 
wrote:

> +1 from me.
>
> В Ср, 10/01/2018 в 05:24 -0700, vveider пишет:
> > Hi, Igniters!
> >
> >
> > While working on Java 9 support adaptation, came to a problem with
> > scala
> > 2.10 and corresponding modules (scala-2.10, spark-2.10 and
> > visor-console-2.10) which will not compile under Java 9. As I see
> > here [1]
> > scala 2.10 is already deprecated in spark 2.1.0 (and lately spark
> > version
> > updated to 2.2.0 in master).
> >
> > So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
> > modules for deletion in 2.4 release.
> >
> > Support for scala 2.11.x (which also will not compile under Java 9)
> > will be
> > maintained by maven profiles.
> >
> >
> > [1] http://spark.apache.org/docs/latest/
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/
>



-- 
Alexey Kuznetsov

[jira] [Created] (IGNITE-7378) WAL converter: add records statistic to WAL reader dev-util

[Dmitriy Pavlov (JIRA)]

Dmitriy Pavlov created IGNITE-7378:
--

 Summary: WAL converter: add records statistic to WAL reader 
dev-util
 Key: IGNITE-7378
 URL: https://issues.apache.org/jira/browse/IGNITE-7378
 Project: Ignite
  Issue Type: Improvement
  Components: persistence
Reporter: Dmitriy Pavlov
Assignee: Dmitriy Pavlov


In Standalone WAL iterator implemented under 
https://issues.apache.org/jira/browse/IGNITE-5558 and  in 
https://issues.apache.org/jira/browse/IGNITE-6277

but this tool just prints record into log. 

It is usefull to see at least machine readable statistics of record types, 
caches involved into this log to find out most popular record types.

It is suggested to add statistic to this tool and print this at end of 
execution.
 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7377) Removed entry is not re-created in IgniteTxManager.lockMultiple

[Semen Boikov (JIRA)]

Semen Boikov created IGNITE-7377:


 Summary: Removed entry is not re-created in 
IgniteTxManager.lockMultiple
 Key: IGNITE-7377
 URL: https://issues.apache.org/jira/browse/IGNITE-7377
 Project: Ignite
  Issue Type: Bug
  Components: cache
Reporter: Semen Boikov
Assignee: Semen Boikov
Priority: Critical
 Fix For: 2.4


IgniteTxManager.lockMultiple: if GridCacheEntryRemovedException is thrown when 
entries are unlocked (txEntry2.cached().txUnlock(tx)) then entry is re-created 
for txEntry1, not for txEntry2. This can lead to infinite loop inside 
lockMultiple.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3350: IGNITE-7375: Right cleanup after training.

[artemmalykh]

GitHub user artemmalykh opened a pull request:

https://github.com/apache/ignite/pull/3350

IGNITE-7375: Right cleanup after training.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7375

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3350.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3350


commit 8700cae7c2727acbad53b782dea31f859545a685
Author: Artem Malykh 
Date:   2018-01-10T13:00:02Z

IGNITE-7375: Right cleanup after training.




---

[jira] [Created] (IGNITE-7376) Document optional on-heap row cache

[Taras Ledkov (JIRA)]

Taras Ledkov created IGNITE-7376:


 Summary: Document optional on-heap row cache
 Key: IGNITE-7376
 URL: https://issues.apache.org/jira/browse/IGNITE-7376
 Project: Ignite
  Issue Type: Task
  Components: documentation
Reporter: Taras Ledkov
Assignee: Taras Ledkov
 Fix For: 2.4


Add documentation for on-heap row cache implemented at IGNITE-7173



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3291: IGNITE-7173 SQL: implement optional row cache

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3291


---

[jira] [Created] (IGNITE-7375) Perform right cleanup for MLPGroupUpdateTrainer

[Artem Malykh (JIRA)]

Artem Malykh created IGNITE-7375:


 Summary: Perform right cleanup for MLPGroupUpdateTrainer
 Key: IGNITE-7375
 URL: https://issues.apache.org/jira/browse/IGNITE-7375
 Project: Ignite
  Issue Type: Bug
Affects Versions: 2.4
Reporter: Artem Malykh
Assignee: Artem Malykh
 Fix For: 2.4


We should clean training data from auxiliary caches after training.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] ignite pull request #3341: IGNITE-7277: override equals() & hashCode() metho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/3341


---

[jira] [Created] (IGNITE-7374) Add 'default' as a possible value for an SQL command parameter

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7374:
---

 Summary: Add 'default' as a possible value for an SQL command 
parameter
 Key: IGNITE-7374
 URL: https://issues.apache.org/jira/browse/IGNITE-7374
 Project: Ignite
  Issue Type: Improvement
  Components: sql
Affects Versions: 2.4
Reporter: Kirill Shirokov






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Scala 2.10 support

[Nikolay Izhikov]

+1 from me.

В Ср, 10/01/2018 в 05:24 -0700, vveider пишет:
> Hi, Igniters!
> 
> 
> While working on Java 9 support adaptation, came to a problem with
> scala
> 2.10 and corresponding modules (scala-2.10, spark-2.10 and
> visor-console-2.10) which will not compile under Java 9. As I see
> here [1]
> scala 2.10 is already deprecated in spark 2.1.0 (and lately spark
> version
> updated to 2.2.0 in master).
> 
> So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
> modules for deletion in 2.4 release.
> 
> Support for scala 2.11.x (which also will not compile under Java 9)
> will be
> maintained by maven profiles.
> 
> 
> [1] http://spark.apache.org/docs/latest/
> 
> 
> 
> --
> Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

Scala 2.10 support

[vveider]

Hi, Igniters!


While working on Java 9 support adaptation, came to a problem with scala
2.10 and corresponding modules (scala-2.10, spark-2.10 and
visor-console-2.10) which will not compile under Java 9. As I see here [1]
scala 2.10 is already deprecated in spark 2.1.0 (and lately spark version
updated to 2.2.0 in master).

So, I'd like to propose scala-2.10, spark-2.10 and visor-console-2.10
modules for deletion in 2.4 release.

Support for scala 2.11.x (which also will not compile under Java 9) will be
maintained by maven profiles.


[1] http://spark.apache.org/docs/latest/



--
Sent from: http://apache-ignite-developers.2346864.n4.nabble.com/

[GitHub] ignite pull request #3333: IGNITE-7318 .NET: Move SQL examples to Sql namesp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/ignite/pull/


---

[GitHub] ignite pull request #3349: IGNITE-7346: Enable Ignite cache events per cache

[alex-plekhanov]

GitHub user alex-plekhanov opened a pull request:

https://github.com/apache/ignite/pull/3349

IGNITE-7346: Enable Ignite cache events per cache



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/alex-plekhanov/ignite ignite-7346

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3349.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3349


commit 0ec834d8f463c9338cce7cfadcd63540f2321129
Author: Aleksey Plekhanov 
Date:   2018-01-09T08:56:24Z

IGNITE-7346 Enable Ignite cache events (EventType.EVTS_CACHE, 
EventType.EVTS_CACHE_REBALANCE) per cache

commit 636bb2988f7711a742f1d1c53cbdbbe411b696f9
Author: Aleksey Plekhanov 
Date:   2018-01-09T10:07:09Z

IGNITE-7346 Enable Ignite cache events (EventType.EVTS_CACHE_REBALANCE) per 
cache

commit 8d3bdec1ad82179b912fdbb14cc08a908cfb46c7
Author: Aleksey Plekhanov 
Date:   2018-01-09T13:30:04Z

IGNITE-7346 Enable Ignite cache events (EventType.EVTS_CACHE_QUERY) per 
cache




---

[GitHub] ignite pull request #3348: IGNITE-7203 Java 8 by default

[vveider]

GitHub user vveider opened a pull request:

https://github.com/apache/ignite/pull/3348

IGNITE-7203 Java 8 by default

 * further improved AOP test behaviour by moving javaagent call into pom.xml
.NET: Fix WindowsServiceTest - pass JVM path to the service
.NET: Another attempt to fix WindowsServiceTest, adding logger, no idea 
what the problem is
.NET: Another attempt to fix WindowsServiceTest
.NET: Another attempt to fix WindowsServiceTest
.NET: Another attempt to fix WindowsServiceTest
IGNITE-7203 Java 8 by default
 * replaced aspectj dependency with newest one for attempt to fix AOP tests
Attempty to fix AOP failure.
.NET: Fix TestStopFromJava
.NET: Fix ExecutableTest
IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * fixed notes from review
IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * refactored examples module and ml part of other modules: merged java8 
and java packages
IGNITE-7203 Java 8 by default
 * got rid of java-1.8 related profiles
 * remove ml profile and included ml build into main reactor
 * added master properties to maven-compiler-plugin for whole project to be 
compiled only by java-1.8
 * refactored met properties + gathered them in parent module

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gridgain/apache-ignite ignite-7203-review

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/ignite/pull/3348.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3348


commit 5d40edbdb6b4b8a5d84485b474e88a03660a21cb
Author: Ivanov Petr 
Date:   2018-01-10T11:27:15Z

IGNITE-7203 Java 8 by default
 * further improved AOP test behaviour by moving javaagent call into pom.xml
.NET: Fix WindowsServiceTest - pass JVM path to the service
.NET: Another attempt to fix WindowsServiceTest, adding logger, no idea 
what the problem is
.NET: Another attempt to fix WindowsServiceTest
.NET: Another attempt to fix WindowsServiceTest
.NET: Another attempt to fix WindowsServiceTest
IGNITE-7203 Java 8 by default
 * replaced aspectj dependency with newest one for attempt to fix AOP tests
Attempty to fix AOP failure.
.NET: Fix TestStopFromJava
.NET: Fix ExecutableTest
IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * fixed notes from review
IGNITE-7203 Java 8 by default
IGNITE-7216 Refactor examples project
 * refactored examples module and ml part of other modules: merged java8 
and java packages
IGNITE-7203 Java 8 by default
 * got rid of java-1.8 related profiles
 * remove ml profile and included ml build into main reactor
 * added master properties to maven-compiler-plugin for whole project to be 
compiled only by java-1.8
 * refactored met properties + gathered them in parent module




---

Re: Java 9 support

[Andrey Kuznetsov]

No. We got rid of monitorEnter/monitorExit there. And ReentrantLock is
supposed to be a replacement.

2018-01-10 13:49 GMT+03:00 Антон Чураев :

> Andrey, do You mean https://issues.apache.org/jira/browse/IGNITE-4908?
>
>
> --
Best regards,
  Andrey Kuznetsov.

Re: Java 9 support

[Антон Чураев]

Andrey, do You mean https://issues.apache.org/jira/browse/IGNITE-4908?

2018-01-10 13:25 GMT+03:00 Andrey Kuznetsov :

> Thanks, Petr.
>
> I heard of some activity related to performance consequences of
> ReentrantLocks in IGNITE-6736 fix. So, I'd like to get a reviewer feedback
> first.
>
> Andrey G., Vladimir O., is it possible to merge the fix to master?
>
> 2018-01-10 9:56 GMT+03:00 Petr Ivanov :
>
> > Andrey — double checked your solution and it works now. Guess there was
> > some merge error first time.
> > Sorry for misleading.
> >
> > So, there is working solution for Java 9 build and I’d like to save this
> > configuration in ignite-6730 (making IGNITE-7144 and IGNITE-6736 to
> become
> > subtasks in the process).
> > What do you think?
> >
> >
> > > On 9 Jan 2018, at 20:49, Andrey Kuznetsov  wrote:
> > >
> > > Hi Petr!
> > >
> > > Could you please clarify what is wrong with fix proposed in
> IGNITE-6736,
> > > and what is supposed to be a replacement for monitorEnter/monitorExit
> > now?
> > >
> > > 2018-01-09 19:08 GMT+03:00 Petr Ivanov :
> > >
> > >> Hi all.
> > >>
> > >>
> > >> After some thorough research and with help of fellow igniters, I’ve
> > >> managed to prepare more or less stable Java 9 build configuration of
> > Apache
> > >> Ignite.
> > >>
> > >> Here are changes to make it work:
> > >> - Java 8 profiles and build process revision, made in IGNITE-7203;
> > >> - Java 9 maven profile prepared in IGNITE-7144 (will be moved to
> > >> IGNITE-6730 as subtask);
> > >> - specific maven-compiler-plugin configuration with JVM args for Java
> 9
> > >> profile (as was proposed by Vladimir Ozerov);
> > >> - maven-bundle-plugin version is updated to 3.5.0;
> > >> - maven-compiler-plugin version synchronised to 3.7.0 (in Cassandra
> > >> modules);
> > >> - scala version updated to 2.12.4;
> > >> - disabled scalar-2.10, spark-2.10 and visor-console-2.10 modules (due
> > to
> > >> dependency in scala 2.10 which is unsupported by Java 9);
> > >> - sun.misc.JavaNioAccess import changed to jdk.internal.misc.
> > JavaNioAccess
> > >> in GridUnsafe.java and PageMemoryImpl.java;
> > >> - sun.misc.SharedSecrets import changed to jdk.internal.misc.
> > SharedSecrets
> > >> in GridUnsafe.java and PageMemoryImpl.java;
> > >> - methods monitorEnter and monitorExit bodies commented out (fix from
> > >> IGNITE-6736 did not work).
> > >>
> > >> I’d like to put these changes into ignite-6730 to have working
> compiling
> > >> under Java 9 branch — so that we can continue work on improving Apache
> > >> Ignite’s Java 9 support.
> > >
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >  Andrey Kuznetsov.
> >
> >
>
>
> --
> Best regards,
>   Andrey Kuznetsov.
>



-- 

Best Regards, Anton Churaev

Re: Java 9 support

[Andrey Kuznetsov]

Thanks, Petr.

I heard of some activity related to performance consequences of
ReentrantLocks in IGNITE-6736 fix. So, I'd like to get a reviewer feedback
first.

Andrey G., Vladimir O., is it possible to merge the fix to master?

2018-01-10 9:56 GMT+03:00 Petr Ivanov :

> Andrey — double checked your solution and it works now. Guess there was
> some merge error first time.
> Sorry for misleading.
>
> So, there is working solution for Java 9 build and I’d like to save this
> configuration in ignite-6730 (making IGNITE-7144 and IGNITE-6736 to become
> subtasks in the process).
> What do you think?
>
>
> > On 9 Jan 2018, at 20:49, Andrey Kuznetsov  wrote:
> >
> > Hi Petr!
> >
> > Could you please clarify what is wrong with fix proposed in IGNITE-6736,
> > and what is supposed to be a replacement for monitorEnter/monitorExit
> now?
> >
> > 2018-01-09 19:08 GMT+03:00 Petr Ivanov :
> >
> >> Hi all.
> >>
> >>
> >> After some thorough research and with help of fellow igniters, I’ve
> >> managed to prepare more or less stable Java 9 build configuration of
> Apache
> >> Ignite.
> >>
> >> Here are changes to make it work:
> >> - Java 8 profiles and build process revision, made in IGNITE-7203;
> >> - Java 9 maven profile prepared in IGNITE-7144 (will be moved to
> >> IGNITE-6730 as subtask);
> >> - specific maven-compiler-plugin configuration with JVM args for Java 9
> >> profile (as was proposed by Vladimir Ozerov);
> >> - maven-bundle-plugin version is updated to 3.5.0;
> >> - maven-compiler-plugin version synchronised to 3.7.0 (in Cassandra
> >> modules);
> >> - scala version updated to 2.12.4;
> >> - disabled scalar-2.10, spark-2.10 and visor-console-2.10 modules (due
> to
> >> dependency in scala 2.10 which is unsupported by Java 9);
> >> - sun.misc.JavaNioAccess import changed to jdk.internal.misc.
> JavaNioAccess
> >> in GridUnsafe.java and PageMemoryImpl.java;
> >> - sun.misc.SharedSecrets import changed to jdk.internal.misc.
> SharedSecrets
> >> in GridUnsafe.java and PageMemoryImpl.java;
> >> - methods monitorEnter and monitorExit bodies commented out (fix from
> >> IGNITE-6736 did not work).
> >>
> >> I’d like to put these changes into ignite-6730 to have working compiling
> >> under Java 9 branch — so that we can continue work on improving Apache
> >> Ignite’s Java 9 support.
> >
> >
> >
> >
> > --
> > Best regards,
> >  Andrey Kuznetsov.
>
>


-- 
Best regards,
  Andrey Kuznetsov.

[jira] [Created] (IGNITE-7373) Fix style guide violations and imprecise naming in SqlParser-related code

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7373:
---

 Summary: Fix style guide violations and imprecise naming in 
SqlParser-related code
 Key: IGNITE-7373
 URL: https://issues.apache.org/jira/browse/IGNITE-7373
 Project: Ignite
  Issue Type: Bug
  Components: sql
Affects Versions: 2.4
Reporter: Kirill Shirokov
Priority: Minor


SQL parser implementation contains many abbreviation rule violations (for 
instance, token instead of tok) and imprecise names (e.g.: 
SqlParserUtils.skipIfMatches throws an exception if the token doesn't match, 
although the name doesn't reflect this. 'accept' or 'skip' without 'if' would 
be more precise)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7372) Implement escape sequences handling in internal SQL parser

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7372:
---

 Summary: Implement escape sequences handling in internal SQL parser
 Key: IGNITE-7372
 URL: https://issues.apache.org/jira/browse/IGNITE-7372
 Project: Ignite
  Issue Type: New Feature
  Components: sql
Reporter: Kirill Shirokov
Priority: Minor


Implement escape sequences handling inside quoted literals. 

The initial implementation is pruned from IGNITE-6861 and put into this feature 
branch.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7371) MVCC TX Repeatable read semantic

[Igor Seliverstov (JIRA)]

Igor Seliverstov created IGNITE-7371:


 Summary: MVCC TX Repeatable read semantic
 Key: IGNITE-7371
 URL: https://issues.apache.org/jira/browse/IGNITE-7371
 Project: Ignite
  Issue Type: New Feature
  Components: cache
Reporter: Igor Seliverstov
 Fix For: 2.5


Repeatable read isolation implies that each GET operation whithin tx gets a 
last commited version of entry *at the time the tx was started*. Curentrly we 
get a last commited version of entry *at the time the first read operation 
invokes on a particular key whithin tx.* We need to fix this unconsistence.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (IGNITE-7370) Drop test tables in each test case in H2DynamicTableSelfTest

[Kirill Shirokov (JIRA)]

Kirill Shirokov created IGNITE-7370:
---

 Summary: Drop test tables in each test case in 
H2DynamicTableSelfTest
 Key: IGNITE-7370
 URL: https://issues.apache.org/jira/browse/IGNITE-7370
 Project: Ignite
  Issue Type: Task
Reporter: Kirill Shirokov
Priority: Minor


Failing test cases are leaving tables they've created which in turn causes 
subsequent test cases to report false failures, e.g. when latter test tries to 
create a table with the same name.

It would be great for such tables tables to run down curtain and join the choir 
invisible in finally clause; an example:

{noformat}
public void testInternalCreateExistingTable() throws Exception {
+   try {
execute("CREATE TABLE \"Person\" (\"id\" int, \"city\" 
varchar," +
" \"name\" varchar, \"surname\" varchar, \"age\" int, 
PRIMARY KEY (\"id\", \"city\")) " +
"template=\"cache\"");

GridTestUtils.assertThrows(null, new Callable() {
@Override public Object call() throws Exception {
execute("CREATE TABLE \"Person\" (\"id\" int, 
\"city\" varchar" +
", \"name\" varchar, \"surname\" varchar, 
\"age\" int, PRIMARY KEY (\"id\", \"city\")) " +
"template=\"cache\"");

return null;
}
}, IgniteSQLException.class, "Table already exists: 
Person");
+   }
+   finally {
+   execute("DROP TABLE \"Person\"");
+   }
...
{noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Transaction operations using the Ignite Thin Client Protocol

[Pavel Tupitsyn]

Hi Denis,

Yes, transactions in thin client protocol are surely very important, I
think we should add them soon (in 2.5).
Ticket: https://issues.apache.org/jira/browse/IGNITE-7369

And yes, server node will handle everything, client just performs TX_START,
TX_COMMIT, TX_ROLLBACK operations.

Pavel

On Wed, Jan 10, 2018 at 1:08 AM, Denis Magda  wrote:

> + dev list
>
> Igniters, Pavel,
>
> I think we need to bring support for key-value transactions to one of the
> future versions. As far as I understand, a server node, a thin client will
> be connected to, will be the transaction coordinator and the client will
> simply offloading everything to it. What do you think?
>
> —
> Denis
>
> > On Jan 8, 2018, at 1:12 AM, kotamrajuyashasvi <
> kotamrajuyasha...@gmail.com> wrote:
> >
> > Hi
> >
> > I would like to perform Ignite Transaction operations from a C++ program
> > using the Ignite Thin Client Protocol. Is it possible to do so ? If this
> > feature is not available now, will it be added in future ?
> >
> >
> >
> > --
> > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>
>

[jira] [Created] (IGNITE-7369) .NET: Thin client: Transactions

[Pavel Tupitsyn (JIRA)]

Pavel Tupitsyn created IGNITE-7369:
--

 Summary: .NET: Thin client: Transactions
 Key: IGNITE-7369
 URL: https://issues.apache.org/jira/browse/IGNITE-7369
 Project: Ignite
  Issue Type: Improvement
  Components: platforms, thin client
Affects Versions: 2.4
Reporter: Pavel Tupitsyn


Implement transactions in thin client protocol and .NET thin client.

Main issue: Ignite transactions are tied to a specific thread.
See how JDBC works around this by starting a dedicated thread.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Irrelevant data in discovery messages

[Denis Mekhanikov]

Igniters,

Turns out, that we are sending a lot of irrelevant information in discovery
messages. Some messages contain *TcpDiscoveryNode* objects, which in turn
have such attributes like *PATH, java.class.path, sun.boot.class.path,
java.library.path, org.apache.ignite.jvm.args, *etc.
Some of these attributes may contain huge strings, that can sum up to
megabytes of data.

It was noticed by a user on our mailing list:
http://apache-ignite-users.70518.x6.nabble.com/Connection-problem-between-client-and-server-td19243.html
In his case these huge messages make discovery process really slow.

I think, we should filter-out such attributes, because they are not used
anywhere, but make messages grow enormous and slow down discovery. We could
include only user-defined and internal attributes + a fixed set of
environment variables.

What do you think?

Denis

Re: DiscoverySpi based on Apache ZooKeeper

[Semyon Boikov]

Andrey, Vladimir,

Zookeeper does provide required ordering guarantees, but Zookeeper watcher
API really does not provide functionality required for DiscoverySpi out of
the box. To detect nodes join/failure we need to watch for
creation/deletion of znodes, but in case if some client disconnects from
Zookeeper and reconnects after some time, then it can miss some change
events and after reconnect it will get only current state. For example:
there are 2 znodes (related to Ignite cluster nodes): A, B. One of
zookeeper clients disconnects from Zookeeper and tries to reconnect, at
this moment new cluster node joins and creates znode C, then it immediately
fails and znode C is removed. When disconnected client restores connection
it still sees znodes A and B and is not aware about node C. This means that
different clients can see different Zookeeper events, to overcome this
issue ZookeeperDiscoverySpi has single coordinator node which listens for
Zookeeper notifications and transforms then into DiscoverySpi events
(scenario when znode C is created and removed while coordinator is
disconnected is still possible, but it is not an issue, this means cluster
node C failed before it finished join). With such approach with single
coordinator I don't see scenario when different nodes can see different
events or some event can be missed.

Semyon

On Wed, Jan 10, 2018 at 10:38 AM, Vladimir Ozerov 
wrote:

> Hi Andrey,
>
> Could you please share detail of this API mismatch? AFAIK, the main
> guarantee we need for disco SPI is total message ordering. Zookeeper
> provides this guarantee. Moreover, Zookeeper is proven to be correct and
> reliable coordinator service by many users and Jepsen tests, as opposed to
> various in-house implementations (e.g. Zen of Elasticsearch).
>
> вт, 9 янв. 2018 г. в 21:53, Andrey Kornev :
>
>> Semyon,
>>
>> Not to discourage you or anything, just a interesting fact from recent
>> history.
>>
>> I vaguely remember already trying to implement DiscoverySpi on top of
>> Zookeeper back in 2012. After a few failed attempts and a lot of help from
>> Zookeeper's original developers (Flavio Junqueira and Ben Reed) we (Dmitriy
>> S. and I) concluded that its not possible to  implement DiscoverySpi on top
>> of Zookeeper due to strict(er) semantics of DiscoverySpi. Unfortunately I
>> do not remember details, but essentially, in some cases it was not possible
>> to establish total ordering of watcher events and under certain
>> circumstances loss of such events was possible.
>>
>> It's not to say that Zookeeper can't be used to implement the cluster
>> membership tracking in general. The problem is rather with DiscoverySpi
>> semantics that require a different set of APIs than what Zookeeper provides.
>>
>> Regards
>> Andrey
>>
>> 
>> From: Semyon Boikov 
>> Sent: Tuesday, January 9, 2018 3:39 AM
>> To: dev@ignite.apache.org
>> Subject: DiscoverySpi based on Apache ZooKeeper
>>
>> Hi all,
>>
>> Currently I'm working on implementation of DiscoverySpi based on Apache
>> ZooKeeper (ZookeeperDiscoverySpi) and want to share results of this work.
>>
>> In very large clusters (>1000 nodes) current default implementation of
>> DiscoverySpi - TcpDiscoverySpi - has some significant drawbacks:
>> - TcpDiscoverySpi organizes nodes in ring, and all messages are passed
>> sequentially via ring. More nodes there are in ring, more time it takes to
>> pass message. In very large clusters such architecture can cause slowdown
>> of important operations (node join, node stop, cache start, etc).
>> - in TcpDiscoverySpi's protocol each node in ring is able to fail next one
>> in case of network issues, and it is possible that two nodes can 'kill'
>> each other (it is possible in complex scenarios when network is broken and
>> then restored back after some time, such problems were observed in real
>> environments), and with current TcpDiscoverySpi protocol there is no easy
>> way to completely fix such problems.
>> - when some node in ring fails, then previous node tries to restore ring
>> and sequentially tries to connect to next nodes. If large part of ring
>> fails then it takes long time to sequentially detect failure of all nodes.
>> - with TcpDiscoverySpi split brain is possible (one ring can split into
>> two
>> independent parts), separate mechanism is needed to protect from split
>> brain when TcpDiscoverySpi is used
>>
>> Even though most probably some of these problems can be somehow fixed in
>> TcpDiscoverySpi, it seems more robust and fast DiscoverySpi can be
>> implemented on top of some existing coordination service.
>>
>> Apache ZooKeeper is known reliable service and it provides all mechanisms
>> and consistency guarantees required for Ignite's DiscoverySpi. Some
>> technical details of ZookeeperDiscoverySpi implementation can be found in
>> description prepared by Sergey Puchnin:
>>